WWW::Mechanize HTTPS

Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

I am wanting to automate a login to a SSL website. I was advised to use WWW::Mechanize for this task. I have successfully created test scripts that login to basic authentication websites. I live in New Zealand and am wanting to access Telecom Jetstream usage information via a script rather than manually keying in information. I followed the following link (http://johnbokma.com/perl/https.html) to install Crypt::SSLeay which worked perfectly. However I am unable to be redirected to this page.

my $url = "https://www.telecom.co.nz/xtralogin.fcc?TYPE=33554433&REALM
+OID=06-0005cc45-3c3e-1f6c-93b5-80edbce40000&GUID=&SMAUTHREASON=0&METH
+OD=GET&SMAGENTNAME=$SM$bPLaid7UiFr%2fcTQchTKdU5ckPa0CnhjCk2Zds2Wopak%
+3d&TARGET=$SM$https%3a%2f%2fwww%2etelecom%2eco%2enz%2fjetstreamum%2fx
+tra";
my $mechanize = WWW::Mechanize->new(autocheck => 1);
$mechanize->cookie_jar(HTTP::Cookies->new);
$mechanize->get($url);
$mechanize->form_name('Login');
$mechanize->field('USER','digger');
$mechanize->field('PASSWORD','testpass');
$mechanize->click();
my $page = $mechanize->content;
open (FH, ">usage.html");
print FH $page;
close(FH);
[download]

Am i on the right track? Are there some websites that just won't allow this automation? Thank you very much for your time

Janitored by holli - added code tags

Comment on WWW::Mechanize HTTPS Download Code

Replies are listed 'Best First'.
Re: WWW::Mechanize HTTPS by monarch (Priest) on May 26, 2005 at 08:13 UTC
Just a disclaimer - I may well be miles off track with this comment. Your URL is hideously long, which suggests it is carrying state information. What does this mean? Well, odds are that the URL is keeping a key to an entry in Telecom's website database.. that entry is created when you log into the website. The beauty of WWW::Mechanize is that it allows you to simulate a browser session, the WWW::Mechanize module keeping track of state information from page to page. Thus I would suggest that you go to a simpler site (www.telecom.co.nz) and follow the link to the login page, and then submit the information. The other thing to do is dump error information after $mechanize->click() e.g. `if ( ! $mech->success() ) { print( STDERR $mech->status() . "\n" ); } else { print( STDERR "-At->" . $mech->uri() . "\n" ); }` [download]	[reply] [d/l]
Re: WWW::Mechanize HTTPS by monarch (Priest) on May 26, 2005 at 08:42 UTC
Oh ok I missed the obvious. You're doing everything right except for one thing - and let this be a lesson - what exactly are you clicking on? Have you had a look at the source code of the webpage..? Evidently you have or you wouldn't have filled in the form name and fields correctly.. Need help? OK the answer is that you are not clicking on anything, for one, because you're blindly calling `$mechanize->click()`. Secondly, the website in question uses javascript to submit the form. THAT is the part you shouldn't have missed. So to fix, consider merely calling `$mechanize->submit()` instead of ->click(). Then print out the result of `$mechanize->success()` and `$mechanize->uri()` to confirm this worked as expected. This particular form merely calls itself so the uri may not give any clues as to the success, but the boolean `$mechanize->success()` will tell you if you've succeeded.	[reply] [d/l] [select]
Re^2: WWW::Mechanize HTTPS by BigRare (Pilgrim) on May 26, 2005 at 13:38 UTC
I have found that sometimes using $mech->click() works in situations where $mech->submit() does not. For example: If I have a page that redirects to another page, based on a click input rather than a submit to the same URL. This becomes especially true in the world of dotNet where many submit buttons fire a JavaScript event to verify the form before allowing the page to be submitted. Usually, however, these same buttons have names and IDs, because that would be good website design. (Example: $mech->click('btnSubmit');) Though sometimes, just occasionally, buttons have neither a name nor and ID. This is where $mech->click() comes in handy. From the Documentation: If there is only one button on the form, $mech->click() with no arguments simply clicks that one button.	[reply]
Re: WWW::Mechanize HTTPS by displeaser (Hermit) on May 26, 2005 at 07:11 UTC
Hi, I think Crypt::SSLeay is only the glue to allow LWP to access https pages, I don't know for sure about mechanize. I've sucessfully used LWP for logging into https sites. Maybe have a look at the LWP modules, they are as simple as mechanize to use. D	[reply]
Re^2: WWW::Mechanize HTTPS by terce (Friar) on May 26, 2005 at 08:45 UTC
Crypt::SSLeay definitely works with WWW::Mechanize (which, btw, extends LWP). monarch's replies below definitely suggest the cause of the issue.	[reply]