Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

Hi,

I am struggling with trying to understand how to loop through (or even access) headers returned from an HTTP request.

My code is like this:

use strict;
use LWP::Simple;
require LWP::UserAgent;

my $ua = LWP::UserAgent->new;
$ua->timeout(10);
$ua->env_proxy;
$ua->max_redirect(0);

my $response = $ua->get('http://www.msn.co.uk');

if ($response->is_success)
{
    while (my ($x,$y) = each (%$response))
    {
    print "$x - $y\n";
    }
}
else
{
die $response->status_line;
}

[download]

Since the response is a 301, and I have redirects set to zero, then it's easy enough to use Data Dumper to see what the response is, per below:

$VAR1 = bless( {
    '_protocol' => 'HTTP/1.1',
    '_content' => '',
    '_rc' => '301',
    '_headers' => bless( {
          'x-powered-by' => 'ASP.NET',
          'client-response-num' => 1,
          'cache-control' => 'no-cache',
          'set-cookie' => 'MC1=V=3&GUID=e03d ....',
          'location' => 'http://uk.msn.com/',
          'date' => 'Mon, 27 Sep 2010 15:31:10 GMT',
          'client-peer' => '213.199.164.24:80',
          'content-length' => '0',
          'x-aspnet-version' => '2.0.50727',
          'p3p' => 'CP="BUS CUR CONo FIN IVDo ONL ..."',
          'client-warning' => 'Redirect loop detected (max_redirect = 
+0)',
           'client-date' => 'Mon, 27 Sep 2010 15:31:54 GMT',
           'pragma' => 'no-cache',
           'server' => 'Microsoft-IIS/6.0',
           's:' => '04'
           }, 'HTTP::Headers' ),
    '_msg' => 'Moved Permanently',
    '_request' => bless( {
           '_content' => '',
           '_uri' => bless( do{\(my $o = 'http://www.msn.co.uk')}, 'UR
+I::http' ),
    '_headers' => bless( {
           'user-agent' => 'libwww-perl/5.803'
           }, 'HTTP::Headers' ),
    '_method' => 'GET'
              }, 'HTTP::Request' )
}, 'HTTP::Response' );

[download]

However, I still can't figure out how to access it.

If I set the redirect to 1, then I'll end up with a bunch of content returned, and a response that will include: 

_protocol - HTTP/1.1
_content - bunch of content
_rc - 200
_headers - HTTP::Headers=HASH(0x1cde8fc)
_previous - HTTP::Response=HASH(0x1cc618c)
_msg - OK
_request - HTTP::Request=HASH(0x1c7e310)

[download]

However, I simply do not understand how to access any of the data, whichever way it gets returned. I want to be able to process several different URLs, and whether it's a 301 response or 200, or whatever, I would like to be able to loop through the data in the headers and also grab the response code, protocol, and msg. I don't need to know what the content is. I have spent hours trying to figure this out but really seem to have no clue. Any assistance would be much appreciated.

Replies are listed 'Best First'.
Re: Looping through HTTP headers
by jethro (Monsignor) on Sep 27, 2010 at 16:26 UTC

    $response is an object of class HTTP::Response. If you read that documentation, you will find the following info:

    $r->header( $field )
    $r->header( $field => $value )

    This is used to get/set header values and it is inherited from HTTP::Headers via HTTP::Message. See HTTP::Headers for details and other similar methods that can be used to access the headers.

    If you check out HTTP::Headers, you will find more methods that are implied by the above to work too, for example $h->header_field_names, which returns a list of all header names in the response.

[reply]
      Thanks for that; I managed to figure out how to do what I need to do from your comments.
[reply]
Re: Looping through HTTP headers
by Khen1950fx (Canon) on Sep 27, 2010 at 18:00 UTC
    If you use LWP::Simple, then you don't need to use $ua->env_proxy. It does it automatically. Also, I would use LWP::Simple for simple uses such as url validation. For example, here I have a bunch of urls that I want to check for validity: 
    #!/usr/bin/perl

use strict;
use warnings;
use LWP::Simple;

my @pages = (
        'http://www.perlmonks.org',
        'http://www.perlmonks.com',
        'http://www.perlmonks.net',
        'http://www.perlmonks.edu',
        'http://www.perlmonks.co.uk',
        'http://perlmonks.perl.org'
);

foreach my $page(@pages) {
        if (head($page)) {
            print "Exists\n";
        }
        else {
            print "It doesn't exist\n";
       }
}

    [download]
[reply]
[d/l]
[select]
      Thanks for those additional comments too.
[reply]

Back to Seekers of Perl Wisdom