Cookie protected web page and file downloading

reTard has asked for the wisdom of the Perl Monks concerning the following question:

Replies are listed 'Best First'.
Re: Cookie protected web page and file downloading by tlm (Prior) on Jun 02, 2005 at 03:13 UTC
I find that WWW::Mechanize is pretty good with cookie-ness. Have you tried it? the lowliest monk	[reply]
Re^2: Cookie protected web page and file downloading by reTard (Sexton) on Jun 02, 2005 at 03:42 UTC
I've had a look at Mech but I can't see where to specify the HTTP proxy details? thanks	[reply]
Re^3: Cookie protected web page and file downloading by tlm (Prior) on Jun 02, 2005 at 03:49 UTC
WWW::Mechanize is a subclass of LWP::UserAgent, which has methods for setting proxy info; see the docs for the latter. the lowliest monk	[reply]
Re^4: Cookie protected web page and file downloading by reTard (Sexton) on Jun 02, 2005 at 05:41 UTC
Re^5: Cookie protected web page and file downloading by tlm (Prior) on Jun 02, 2005 at 05:58 UTC
Some notes below your chosen depth have not been shown here
Re: Cookie protected web page and file downloading by reTard (Sexton) on Jun 03, 2005 at 00:44 UTC
Hi again I've made many of the suggested changes and the script now looks like: #!/usr/bin/perl use Data::Dumper; use WWW::Mechanize; my $mech = WWW::Mechanize->new( cookie_jar => {}, agent => "WWW-Mechanize/0.01", protocols_allowed => ['http'], autocheck => 1); $url = 'http://www.ibm.com/servers/eserver/support/pseries/aixfixes.ht +ml'; $mech->proxy('http','172.17.1.248'); $mech->get( $url ); print Dumper $mech; $a=<STDIN>; `clear`; $mech->follow_link( text_regex => qr/More fix services/) or die; print Dumper $mech; $a=<STDIN>; `clear`; $mech->follow_link( text_regex => qr/AIX 5.3/) or die; print Dumper $mech; $a=<STDIN>; `clear`; $mech->follow_link( text_regex => qr/Data file for AIX 5.3/) or die; print Dumper $mech; [download] Now I'm getting the following error: `Error GETing http://www.ibm.com/servers/eserver/support/pseries/aixfixes.html: Access to 'http' URIs has been disabled at aixfixes.pl line 15` It's dying on the `$mech->get( $url );` line. If I remove the `protocols_allowed => ['http'],` bit I get a different error: `Error GETing http://www.ibm.com/servers/eserver/support/pseries/aixfixes.html: Protocol scheme '' is not supported at lwp.pl line 15`. I can manually access this page with lynx so it must be something in the script I'm doing wrong. Any more ideas? Thanks	[reply] [d/l] [select]
Re^2: Cookie protected web page and file downloading by dave0 (Friar) on Jun 03, 2005 at 01:51 UTC
It looks like you're specifying your proxy incorrectly. `$mech->proxy()` takes a URL as its second argument, not an IP address. Try `$mech->proxy('http', 'http://172.17.1.248/')` and see if that works.	[reply] [d/l] [select]
Re^3: Cookie protected web page and file downloading by reTard (Sexton) on Jun 03, 2005 at 03:00 UTC
Yay! Thank you!! This almost works now! I can see the link to the file I want to download now but I'm now sure how to reference it. `'last_uri' => 'http://www-912.ibm.com/eserver/support +/fixinfo/download?file=LatestFixData53', 'uri' => 'http://www-912.ibm.com/eserver/support/fixi +nfo/download?file=LatestFixData53',` [download] How do I save this? I've looked at lwp-download but that only confused me more. Thanks	[reply] [d/l]
Re^4: Cookie protected web page and file downloading by reTard (Sexton) on Jun 03, 2005 at 03:20 UTC