in reply to Accessing webpages with proxy url requiring user/pass

You're a bit fuzzy (at least it's not clear to me) on whether it's the proxy which is requiring authentication, the remote server which is using browser-based (i.e. HTTP Basic authentication), or the target site which has a login page which you need to fill in.

In the first case, read perldoc lwpcook for its section on proxies which gives an example of the syntax to use to connect through an authenticating HTTP proxy. In the second case, you want to read "Access to protected documents" in the same place.

As for the final case, you probably want WWW::Mechanize as has been already mentioned.

Replies are listed 'Best First'.
Re^2: Accessing webpages with proxy url requiring user/pass
by cdherold (Monk) on Sep 08, 2007 at 04:59 UTC
    Mechanize doesn't meet my needs in this case (as far as I can tell). LWPcook was exactly what I needed -- see PROXIES.

    The following got me logged in to the website programatically, which then allowed me to open the links in the site ($links) to view their content($body). 

    $ua = LWP::UserAgent->new;
$ua->proxy(['http', 'ftp'] => 'http://user:pass@proxy.site');
            
$req = HTTP::Request->new('GET',"$link");
            
$res = $ua->request($req);
            
$body = $res->content;

    [download]

    It is important to make sure that proxy.site is not the url that you entered in the tools options of the web browser as the proxy server, but is the one that comes up on the user/pass request box when you are entering those values manually when requested (that took me a while to figure out).

    Thanks for pointing LWPcook out. There are a number of copies out there, but the only one complete one is on CPAN at http://search.cpan.org/~gaas/libwww-perl-5.808/lwpcook.pod

    Chris Herold
[reply]
[d/l]

In Section Seekers of Perl Wisdom