theloanarranger has asked for the wisdom of the Perl Monks concerning the following question:

Dear PerlMonks,

I am using LWP::UserAgent for some spidering work. After growing sick and tired of unreliable free proxy servers I went and purchased access to private proxy servers that require password authentication. I looked through all of the documentation for LWP::UserAgent and couldn't find anything.
Does any one have any insight as to how to do this?

Matt Schick, Chicago, IL
  • Comment on LWP::UserAgent and private proxy servers

Replies are listed 'Best First'.
Re: LWP::UserAgent and private proxy servers
by tachyon (Chancellor) on Dec 02, 2004 at 08:40 UTC

    It may be as simple as doing this: 

    $ua->proxy(["http"], "http://username:password@proxy.com");

    [download]

    Or you may find that this code (which does 401 authentication for you) does what you want. 

    my $ua = LWP::Custom->new();
$ua->proxy(["http"], "http://proxy.com");
$ua->set_basic_credentials( $user, $pass );

package LWP::Custom;

use base 'LWP::UserAgent';

# add a set_basic_credentials method, using a closure to remember
{
    my ( $username, $password );
    sub set_basic_credentials{ ( $username, $password ) = @_[1..2] }
    sub get_basic_credentials{ $username, $password };
}

    [download]

    cheers

    tachyon

[reply]
[d/l]
[select]
      Tachyon,

      Thank you so much for your help. The first idea didn't work because LWP::UserAgent couldn't handle the username and password within the url, but the second idea worked wonderfully and you showed me how to extend classes.
      Thanks again,

      Matt Schick
[reply]
      is that documented?
[reply]
        It sure is. Read the proxy attributes section of the documentation.

        davidj
[reply]

Back to Seekers of Perl Wisdom