collecting header data with LWP::Simple

coldfingertips has asked for the wisdom of the Perl Monks concerning the following question:

This is sort of a weird question and it boggles my mind why it's acting like this. There is a site that displays header data like this

Connection: close
Date: Tue, 21 Mar 2006 21:10:49 GMT
Accept-Ranges: bytes
ETag: "8d7ba-16ab-441f261f"
Server: Apache/1.3.29
Content-Length: 5803
Content-Type: text/html
Content-Type: text/html; charset=iso-8859-1
Last-Modified: Mon, 20 Mar 2006 22:01:03 GMT
Client-Date: Tue, 21 Mar 2006 20:54:08 GMT
Client-Peer: 66.39.76.93:80
Client-Response-Num: 1
Link: <mailto:cpan@perl.org>; rev="made"
Title: CPAN
[download]

I want to mimic this the best I can with Perl. LWP::Simple has a head($url) function which I finally got to work (seems as though another module may have been sharing the same head() and it was messing everything up).

Anyway, this is where it's weird.

use strict;
use warnings;
use LWP::Simple;

my $url = "http://www.cpan.org";
my $src = get($url);

my @stuff = head($url);

print @stuff;
[download]

This results in the following

C:\Documents and Settings\admin\Desktop>perl test.pl
Use of uninitialized value in print at test.pl line 12.
text/html58031142892063Apache/1.3.29
[download]

Now when I save it as a scalar, $head = get($url) and do a data dump on it, I get the following.

$VAR1 = bless( {
                 '_protocol' => 'HTTP/1.1',
                 '_content' => '',
                 '_rc' => 200,
                 '_headers' => bless( {
                                        'connection' => 'close',
                                        'client-response-num' => 1,
                                        'last-modified' => 'Mon, 20 Ma
+r 2006 22:
01:03 GMT',
                                        'accept-ranges' => 'bytes',
                                        'date' => 'Tue, 21 Mar 2006 21
+:17:28 GMT
',
                                        'client-peer' => '66.39.76.93:
+80',
                                        'content-length' => '5803',
                                        'client-date' => 'Tue, 21 Mar 
+2006 21:17
:43 GMT',
                                        'etag' => '"8d7ba-16ab-441f261
+f"',
                                        'content-type' => 'text/html',
                                        'server' => 'Apache/1.3.29'
                                      }, 'HTTP::Headers' ),
                 '_msg' => 'OK',
                 '_request' => bless( {
                                        '_content' => '',
                                        '_uri' => bless( do{\(my $o = 
+'http://ww
w.cpan.org')}, 'URI::http' ),
                                        '_headers' => bless( {
                                                               'user-a
+gent' => '
LWP::Simple/5.805'
                                                             }, 'HTTP:
+:Headers'
),
                                        '_method' => 'HEAD'
                                      }, 'HTTP::Request' )
               }, 'HTTP::Response' );
[download]

It seems to me, there is a TON more information than what I am getting back in my array. My question is, how do I get back all this information? It has things like the IP address and time in here which my @head didn't have. I was told to use UserAgent this afternoon in the CB, but it looks to me that all the information I am looking for is available to me already.

Comment on collecting header data with LWP::Simple Select or Download Code

Replies are listed 'Best First'.
Re: collecting header data with LWP::Simple by ikegami (Patriarch) on Mar 21, 2006 at 21:35 UTC
For starters, `head` does a HEAD request (as opposed to a GET or a POST request), so it's useless to call both `get` and `head`. Then, if you read the documentation, you'll see that `head` returns five values in array context. That doesn't help you. However, in scalar context, `head` returns an HTTP::Response object, and that contains the headers of the response. HTTP::Response inherits `headers` from HTTP::Message, which returns an HTTP::Headers object. HTTP::Headers has a `as_string` method that returns "the header fields as a formatted MIME header. Since it internally uses the scan method to build the string, the result will use case as suggested by HTTP spec, and it will follow recommended 'Good Practice' of ordering the header fields." So, `use strict; use warnings; use LWP::Simple; my $url = "http://www.cpan.org/"; my $response = head($url) or die("Error fetching URL $url\n"); print($response->protocol(), ' '); print($response->status_line(), "\n"); print($response->headers_as_string()); # $response->headers->as_string` [download] Read more... Variations with extra functionality (3 kB)	[reply] [d/l] [select]

Replies are listed 'Best First'.

Re: collecting header data with LWP::Simple
by ikegami (Patriarch) on Mar 21, 2006 at 21:35 UTC

For starters, head does a HEAD request (as opposed to a GET or a POST request), so it's useless to call both get and head.

Then, if you read the documentation, you'll see that head returns five values in array context. That doesn't help you. However, in scalar context, head returns an HTTP::Response object, and that contains the headers of the response.

HTTP::Response inherits headers from HTTP::Message, which returns an HTTP::Headers object. HTTP::Headers has a as_string method that returns "the header fields as a formatted MIME header. Since it internally uses the scan method to build the string, the result will use case as suggested by HTTP spec, and it will follow recommended 'Good Practice' of ordering the header fields."

So,

use strict;
use warnings;

use LWP::Simple;

my $url = "http://www.cpan.org/";

my $response = head($url)
   or die("Error fetching URL $url\n");

print($response->protocol(), ' ');
print($response->status_line(), "\n");
print($response->headers_as_string()); # $response->headers->as_string
[download]

Read more... Variations with extra functionality (3 kB)

[reply]
[d/l]
[select]