mujiburger has asked for the wisdom of the Perl Monks concerning the following question:

Hi,

I've run into a strange problem. I've tested this code on 3 boxes:

Linux Testing Server:
debian sarge
PERL 5.8.4-8sarge3
libwww-perl 5.803-4
www-mechanize 1.12-2
crypt-ssleay 0.51-3


W32 Testing Server:
XP PRO 2002 SP2
ActiveState PERL v5.8.6 built for MSWin32-x86-multi-thread
libwww-perl 5.801
www-mechanize 0.72
crypt-ssleay 0.51


Client's Server:
XP PRO 2002 SP2
ActiveState PERL 5.8.0 built for MSWin32-x86-multi-thread
libwww-perl 5.68
www-mechanize 0.72
crypt-ssleay 0.51


It works everywhere except of course the client's server... While debugging I've found that everything works up until this request:

$mech->submit_form( form_number => 1, fields => { 'password' => $self->{password} }, );

After that request $mech->uri returns the same url as before the request, but $mech->content(...) is empty. The strange part is that this works fine and they are both very similar https posts:

$mech->submit_form( form_name => 'login', fields => { 'input-login-email' => $self->{account}, 'password' + => $self->{password} }, );


I've tried everything I can think of to get it working... Using:

**************
$mech->field(...info...);
$mech->click();
**************
$mech->submit(...info...);
**************
$mech->request( $request); # with a proper post request...
**************
And 100 other variations.
**************


They all work on the testing servers, but not the production... I tried using regular LWP requests to get through the security before I even started using WWW:Mechanize, I couldn't even get that working on the testing servers. This is not my first rodeo, but I'm stumped. All I can think of is that the LWP version is breaking the code. Anybody else have any other ideas? Upgrading PERL/LWP can only happen as a last resort.



Here is a snippet of the relevant parts of the offending code:

if ($self->{location} eq 'us'){ login_us($self) unless $self->{logged_in}; $mech->get('https://s1.amazon.com/exec/varzea/subst/your-account/m +anage-your-seller-account.html/ref=ya_hp_az_2/'); } elsif ($self->{location} eq 'uk'){ login_uk($self) unless $self->{logged_in}; $mech->get('http://s1.amazon.co.uk/exec/varzea/subst/your-account/ +manage-your-seller-account.html/ref=ya_hp_az_2/'); } $mech->follow_link(text_regex => qr/View your Amazon Payments account +and billing history/i); $mech->follow_link(text_regex => qr/View your Amazon Payments account +summary/i); $mech->submit_form( form_number => 1, fields => { 'password' => $self->{password} }, ); $mech->follow_link(url_regex => qr/vg=1&ve=6&vf=4/i); sub login_us{ my $self = shift; print "Performing US LOGIN...\n"; $mech->get('http://www.amazon.com/'); $mech->follow_link(text_regex => qr/your account/i); $mech->follow_link(text_regex => qr/Your seller account/i); $mech->submit_form( form_name => 'login', fields => { 'input-login-email' => $self->{account}, 'p +assword' => $self->{password} }, ); $self->{logged_in} = 1; return $mech->content(base_href => [undef]); }

Replies are listed 'Best First'.
Re: www::mechanize submit _form silent fail
by PodMaster (Abbot) on Feb 18, 2006 at 07:41 UTC
    And 100 other variations.
    Why don't you check the response? status? Turn on debugging (see LWP::Debug)?

    Just like open('perldoc -f open'), Mechanize's submit can fail, get can fail .... you have to check the response.

    MJD says "you can't just make shit up and expect the computer to know what you mean, retardo!"
    I run a Win32 PPM repository for perl 5.6.x and 5.8.x -- I take requests (README).
    ** The third rule of perl club is a statement of fact: pod is sexy.

Re: www::mechanize submit _form silent fail
by adamk (Chaplain) on Feb 18, 2006 at 08:53 UTC
    You can analyze and analyze and guess, and add print statements and tweak and repeat this over and over and over again.

    But at a certain point, we really really need to break out the debugger.

    Get into the debugger, and trace it ALL the way down. Down into LWP, into the HTTPS networking code. You really need to just burrow down into the guts, looking at the contents of the variables as you go.

    And go as deep as it takes to find the _exact_ place at which it fails.

    Speculation and guessing can only go so far, before you need to break out the debugger. And it sounds like you are there.
Re: www::mechanize submit _form silent fail
by mujiburger (Acolyte) on Feb 18, 2006 at 09:09 UTC
    Well....

    Thanks for the response. I ended up getting access to another box on the client's LAN (with perl 5.8.6 and LWP 5.8) and opened a network connection to mySQL for the parts of the code that needed it but weren't included in the initial post. It runs just fine that way and the client is happy. I'm pretty convinced that the problem was an incompatibility between www:mech's handling of https authentication and ActiveState 5.8.0 with LWP 5.68. I'll run it with LWP::Debug and post a more educated opinion once I figure out exactly what was wrong. I had been working with LWP::Debug earlier, when I was trying to jump through amazon's hoops with LWP, but stopped when I switched to www::mech and things seemed to be working. God knows, I looked all over the net for an explanation or someone else who had this problem and never found it. But for now, I'm just happy that I can send this invoice!
Re: www::mechanize submit _form silent fail
by monarch (Priest) on Feb 18, 2006 at 11:02 UTC
      I tried both versions of www::mechanize that were available on the client's server, I think 1.14 was the other one. I'm pretty sure that the incompatibilty was with LWP 5.68... mostly because the code worked on my W32 testing server which is running www:mech 0.72.