flipper has asked for the wisdom of the Perl Monks concerning the following question:

Hi Monks,

I'm using LWP::UserAgent to download some files from behind a slightly broken proxy server, which generates lots of keepalive HTTP response headers, causing LWP to keel over saying "Too many header lines (limit is 128)".

I've tracked this message down to Net::HTTP::Methods, and found you can override the default MaxHeaderLines from Net::HTTP::configure() and I've discovered that LWP::Protocol::http::Socket ISA Net::HTTP, but I don't understand how I can get to the Net::HTTP object.

I was hoping for something like 

my $ua=LWP::UserAgent->new();
$ua->configure({MaxHeaderLines => 1000});

[download]

....or...

my $ua=LWP::UserAgent->new(MaxHeaderLines => 1000);
but it's the LWP::Protocol::http::Socket which ISA Net::HTTP, not the LWP::UserAgent, and the constructor complains that it doesn't understand the arguments I've passed it - can anyone point me in the right direction??


Thanks in advance!

Replies are listed 'Best First'.
Re: Net::HTTP::Methods and LWP::UserAgent
by Fletch (Bishop) on Sep 26, 2008 at 17:47 UTC

    LWP::UA will call LWP::Protocol::create in order to get the implementor for a given scheme. You can use LWP::Protocol::implementor to set a custom subclass as the handler for the http scheme, and in your custom subclass (which should more or less inherit directly from LWP::Protocol::http) you should be able to override the default header lines value.

    I used to have an example which showed how to do this for something similar (overriding the IO::Socket to use a specific source address) but I can't find it for the life of me . . . aaaah, thank you Wayback Machine: lwplocal.plx.

    The cake is a lie.
    The cake is a lie.
    The cake is a lie.

[reply]
[d/l]
[select]
      Perfect, I ended up with:
      #!/usr/bin/perl -w
package MyHTTP;
use base qw(LWP::Protocol::http);

sub _new_socket {
        my $self=shift;
        my $ret =$self->SUPER::_new_socket(@_);
        $ret->max_header_lines(1000);
        return $ret;
}
package MyHTTP::Socket;
use vars qw(@ISA);

@ISA = qw(LWP::Protocol::http::SocketMethods Net::HTTP);


package main;
LWP::Protocol::implementor( 'http', 'MyHTTP' );
use LWP::Simple qw( getprint );

getprint( shift || 'http://localhost/' );

exit 0;

__END__

      [download]

      Hopefully that's unobtrusive enough... Many thanks!
[reply]
[d/l]
        or even 
        package MyHTTP::Socket;
use vars qw(@ISA);

@ISA = qw(LWP::Protocol::http::Socket);

        [download]
[reply]
[d/l]

Back to Seekers of Perl Wisdom