Jonathan has asked for the wisdom of the Perl Monks concerning the following question:

Like many others I need to download a page from the web. The problem (I think I have) is that I am at work behind our firewall and to connect to the net I have to enter my username/password. I think this is why all the example scripts are failing.

Is there any way I can automate this? Is there a module that will pass these as parameters?
I've looked at the pm files and nothing suggested itself.

help!

Replies are listed 'Best First'.
Re: Yet another LWP question
by flyfishin (Monk) on Jun 28, 2000 at 20:51 UTC
    With our firewall we essentially had to fake out the firewall and tell it that we are using a Mozilla agent to get out, and also gave it the proxy. To get outside our file wall this was added to our code

    $ua->agent("Mozilla/4.51 [en] (X11; I; Linux 2.2.14-15mdk i686)");
$ua->proxy('http', 'http://ourproxy:8080');

    [download]

[reply]
[d/l]
Re: Yet another LWP question
by lhoward (Vicar) on Jun 28, 2000 at 20:47 UTC
    The crux of your problem is understanding how your firewall/proxy works. You need to come to an understanding of how your firewall authentication works, then you can address how to deal with it in perl. Exactly how does the login you have to do to browse the web work? Does your web browser have to have any proxy settings?

    There are proxy methods in LWP::UserAgent to support proxying and other methods for HTTP authentication; either of those might fit the bill depending on how your firewall authentication works.

[reply]
RE: Yet another LWP question
by gronkulator (Sexton) on Jun 28, 2000 at 21:22 UTC
    it depends - if you are going through an http proxy, you may have to send the proxy-authorization header as part of the http request object. You'll probably have to use the "basic" auth type, and use MIME::Base64 to encode the string "user:password" for your credentials. A quick way to cheat past this is to sniff a session from your browser and cut & paste the proxy auth header into your perl script as part of the http request headers.
[reply]
Re: Yet another LWP question
by barndoor (Pilgrim) on Jun 28, 2000 at 20:45 UTC
    You could try using the credentials() method of UserAgent to set a username and password for specific URL ranges. I've not tried it myself but this defines HTTP::Headers which are used by the HTTP::Request method. I've only really just looked this up so if anyone can give an example they've done better than me. You can find out more on this in 'Nutshell'
[reply]
Re: Yet another LWP question (wget)
by ybiC (Prior) on Jun 28, 2000 at 21:10 UTC

    If it doesn't have to be done with Perl, GNU wget supports authenticated proxy and SOCKS. Unix, Win95, and DOS ports.

    I've had good luck with it on Debian, anyway.

        cheers,
        ybiC

[reply]
Re: Yet another LWP question
by chromatic (Archbishop) on Jun 29, 2000 at 04:24 UTC
    A quick glance at `perldoc lwpcook` gives the following: 
    use LWP::UserAgent;

$ua = new LWP::UserAgent;
$ua->proxy(['http', 'ftp'] => 'http://proxy.myorg.com');

$req = new HTTP::Request 'GET',"http://www.perl.com";
$req->proxy_authorization_basic("proxy_user", "proxy_password");

$res = $ua->request($req);
print $res->content if $res->is_success;

    [download]
    If your OS supports it, you might also be able to set HTTP_PROXY and FTP_PROXY environment variables. But I'd do it the LWP Cookbook way.
[reply]
[d/l]

Back to Seekers of Perl Wisdom