Skeeve has asked for the wisdom of the Perl Monks concerning the following question:

Greetings fellow monks!

Usually I code as much as possible on my own. So when I want to do something automatic on a webpage I usually loaded that page with curl and fetched the data I needed with a bunch of REs.

But the lessons in th monastry sunk in and so I started the adventure of WWW::Mechanize, when I felt the need to automatically retrieve some data from my account at Nintendo.

But I failed miserably! Analyzing the web traffic revealed that a login using Mozilla sends different data than a login tried with my script.

Maybe a WWW::Mechanize expert can find out what I'm doing wrong or what I'm missing.

Here is my script with personal data X-ed

#!/usr/bin/perl use strict; use warnings; use WWW::Mechanize; my $user= 'XXXX@spamgourmet.com'; my $password= 'XXXX'; my $n=WWW::Mechanize->new(); $n->get( q(http://www.nintendo-europe.com/NOE/de/DE/logout/index.jsp?D +PSLogout=true) ); $n->form_name('login'); $n->set_visible( $user, $password ); $n->submit_form( x => 3, y => 7); print $n->content()


s$$([},&%#}/&/]+}%&{})*;#$&&s&&$^X.($'^"%]=\&(|?*{%
+.+=%;.#_}\&"^"-+%*).}%:##%}={~=~:.")&e&&s""`$''`"e

Replies are listed 'Best First'.
Re: WWW::Mechanize can't log in using it
by svenXY (Deacon) on Oct 26, 2005 at 14:18 UTC
    Hi,
    just out of curiosity:
    Did you try to use WWW::Mechanize::Shell, run an interactive session with it and then use the script function to dump the session as executable perl code to a file which you can then proceed with?
    I've found this much easier and faster than to start from scratch.
    Regards,
    svenXY
Re: WWW::Mechanize can't log in using it
by friedo (Prior) on Oct 26, 2005 at 14:33 UTC
    $n->set_visible( $user, $password );

    I've found that set_visible isn't always reliable; sometimes there are other fields or weird javascripty widgets that mess it up. I find it's almost always better to specify the fields by name. So assuming the fields are called 'username' and 'password', try something like:

    $n->set_fields( username => $user, password => $password );

      Unfortunately the names are not that easy. Example:

      and I don't want to have those complicated names in my script. Especially not because they might choose to change them!

      Fortunately set_visible is not the problem as the data really goes to the expected fields as the network analysis reveald.


      s$$([},&%#}/&/]+}%&{})*;#$&&s&&$^X.($'^"%]=\&(|?*{%
      +.+=%;.#_}\&"^"-+%*).}%:##%}={~=~:.")&e&&s""`$''`"e
Re: WWW::Mechanize can't log in using it
by ickyb0d (Monk) on Oct 26, 2005 at 14:56 UTC

    I've recently gotten into WWW::Mechanize... i would try this instead...

    #!/usr/bin/perl use strict; use warnings; use WWW::Mechanize; my $user= 'XXXX@spamgourmet.com'; my $password= 'XXXX'; #maybe need cookies? my $n=WWW::Mechanize->new(cookie_jar => {}); $n->get( q(http://www.nintendo-europe.com/NOE/de/DE/logout/index.jsp?D +PSLogout=true) ); $n->form_name('login'); $n->set_visible( $user, $password ); #specify form to submit $n->submit_form(form_name => 'login'); print $n->content()

    just by quickly looking at it... set_visible might be your best bet since those fields don't have specific names. And as far as I can tell... those are the first two (non-hidden) fields in that form.

    If it still doesn't work, try debugging by taking a look at $n->forms(). That should return an object of all of the form elements that mech sees and give you a better idea what to use for the fields. hope this helps.

Re: WWW::Mechanize can't log in using it
by Corion (Patriarch) on Oct 26, 2005 at 14:15 UTC

    Maybe it would help to post the data your script is sending as well as the data your browser is sending, or even better, to compare the data of both yourself, and then to determine what the difference is, and where it might stem from. Maybe the website uses Javascript, for example.

      I already tried to analyze the data by myself and found differences. But I have absolutely no clue why the difference is there.

      JavaScript yes-but...

      Yes, they use JavaScript and there is an onclick.handler for the login.

      But, you can log in without JavaScript and I did that using the browser.

      I didn't want to dump the data here unless it's necessary. I think I will first try the SHell-approach...


      s$$([},&%#}/&/]+}%&{})*;#$&&s&&$^X.($'^"%]=\&(|?*{%
      +.+=%;.#_}\&"^"-+%*).}%:##%}={~=~:.")&e&&s""`$''`"e
Re: WWW::Mechanize can't log in using it
by planetscape (Chancellor) on Oct 27, 2005 at 06:33 UTC

      Thanks, planetscape! I'm using ethereal since the last millenium ;-) and used it to find out that there is something different as stated above.

      But HTTP::Recorder is new to me! Very helpful it seems. I will test it.

      Currently I have problems installing WWW::Mechanize::Shell and so I'm not gone any further in solving my "problem".


      s$$([},&%#}/&/]+}%&{})*;#$&&s&&$^X.($'^"%]=\&(|?*{%
      +.+=%;.#_}\&"^"-+%*).}%:##%}={~=~:.")&e&&s""`$''`"e

        Don't forget that HTTP::Recorder's author is leira, a PerlMonk (and an extremely helpful one, too!).

        I had a few issues getting things up and running using WinXP SP2, Cygwin, and HTTP::Proxy, so if you need anything else, run it by me and I'll see if I can help further. :-)

        planetscape