perl lwp

perl_sakthe has asked for the wisdom of the Perl Monks concerning the following question:

hi all, From perl I need to hit the url and retrieve the contents. I am using the LWP module to hit the url. But the contents are empty and also, i am getting the header as HTTP/0.9 200 (OK) EOF. Can you please tell why i am getting this issue

 #!/usr/bin/perl

use strict;
use warnings;
use LWP::UserAgent;
use HTTP::Request;
use LWP 5.00;
use URI::URL;


my $url='http://www.google1.com/';

my $ua = LWP::UserAgent->new(env_proxy => 1,keep_alive =>1, timeout =>
+30);
my $header = HTTP::Request->new( GET => $url);
my $req = HTTP::Request->new('GET', $url,$header);

#PASS REQUEST
my $res = $ua->request($req);


#GET RESULTS
if($res->is_success)
{
   print "New url :",$url,"\n";
   print "Response Headers:\n";
   print $res->headers_as_string;
   print '='x70,"\n",$res->as_string(), '='x70,"\n";
   print "Content_type:".$res->content_type . "\n";

}
else
{
   print "No response\n";
   print $res->status_line . "\n";

}
[download]

Result is New url :http://www.google.com/ Response Headers: Client-Date: Tue, 07 Feb 2012 10:17:21 GMT Client-Peer: 74.125.113.106:80 Client-Response-Num: 1 ====================================================================== HTTP/0.9 200 (OK) EOF Client-Date: Tue, 07 Feb 2012 10:17:21 GMT Client-Peer: 74.125.113.106:80 Client-Response-Num: 1 ====================================================================== Content_type: </code>

Comment on perl lwp Download Code

Replies are listed 'Best First'.
Re: perl lwp by kcott (Archbishop) on Feb 07, 2012 at 11:45 UTC
Your code is fine; the problem is with the URL. In general, when testing this sort of code, check the validity of the URL before assuming a problem with your Perl code. What do you get when you plug that URL into a browser? Removing the "1" (i.e. `my $url='http://www.google.com/';`) gives: `New url :http://www.google.com/ Response Headers: Cache-Control: private, max-age=0 Date: Tue, 07 Feb 2012 11:20:59 GMT Server: gws Content-Type: text/html; charset=ISO-8859-1 ...` [download] If you think the URL you provided should be valid, contact Google for a reason why it's not available. -- Ken	[reply] [d/l] [select]
Re^2: perl lwp by perl_sakthe (Initiate) on Feb 08, 2012 at 05:21 UTC
Hi, thanks for the reply.. I am able to get the contents of our application url.. For the current requirement I need to get the xml content from an external url.. from my perl script, I am not able to get any other url's content...should i check any firewall or proxy.. when i run the perl script for any other url , i am getting the result as, HTTP/0.9 200 (OK) EOF Client-Date: Wed, 08 Feb 2012 05:13:58 GMT Client-Peer: 66.39.54.27:80 Client-Response-Num: 1 Can you please suggest..	[reply]
Re^3: perl lwp by kcott (Archbishop) on Feb 08, 2012 at 09:05 UTC
Your code works when I run it and I can access external content so I can't reproduce your problem. I suggest you check with your local system/network administrators or whoever manages such tasks. -- Ken	[reply]
Re: perl lwp by Anonymous Monk on Feb 07, 2012 at 11:06 UTC
You're programming much too complicated. Do read the documentation of the software you are working with: LWP::UserAgent, chapter REQUEST METHODS. There is a convenience method `get`. `#!/usr/bin/perl use strict; use warnings; use LWP::UserAgent 6; my $url = 'http://www.perlmonks.org/'; my $ua = LWP::UserAgent->new(env_proxy => 1, keep_alive => 1, timeout +=> 30); my $res = $ua->get($url); if ($res->is_success) { print "New URL: $url\n"; print "Response Headers:\n"; print $res->headers_as_string; print '='x70, "\n", $res->as_string, '='x70, "\n"; print "Content_type:" . $res->content_type . "\n"; } else { print "No success\n"; print $res->status_line . "\n"; }` [download]	[reply] [d/l]
Re^2: perl lwp by perl_sakthe (Initiate) on Feb 08, 2012 at 05:16 UTC
Hi, Even with this code, i am getting the result as, HTTP/0.9 200 (OK) EOF Client-Date: Wed, 08 Feb 2012 05:13:58 GMT Client-Peer: 66.39.54.27:80 Client-Response-Num: 1 There is not content. Should i check for any firewall issues or any proxy setting.. i am running from my unix sevrer.. should any permissions be granted.. Because I am able to get the contents of only our application url.. other than that I am not able to hit any external url.. Can you please suggest	[reply]