ragas has asked for the wisdom of the Perl Monks concerning the following question:

I am trying to login to a page programmatically using Perl. I am using the WWW::Mechanize module to get the content of a page, set the fields and submit the values. But since WWW::Mechanize does not support Javascript, the login process has been failing. I would like to know if there is any other way/workaround I could login to a page which does javascript validation of the field before POSTing.

Here is my program:
use strict; use LWP::UserAgent; use WWW::Mechanize; use HTTP::Cookies; use LWP::Debug qw(+); my $user_name = '...'; my $password = '...'; my $base_url = 'https://courses.northwestern.edu/webapps/login/'; my $agent = WWW::Mechanize->new( autocheck => 1 ); $agent->cookie_jar(HTTP::Cookies->new); $agent->get($base_url); die $agent->response->status_line unless $agent->success; #print $agent->content; $agent->set_fields( user_id => $user_name, password => $password ); #print $agent->content(); my $res = $agent->submit(); if ($res->is_success) { print $res->as_string; } else { print "Failed: ", $res->status_line, "\n"; }

And the HTML code contains:
<FORM ONSUBMIT="return validate_form(this)" METHOD="POST" ACTION="https://courses.northwestern.edu/webapps/login/" NAME="login" >

validate_form() is a Javascript method. Any help on this would be much appreciated.

Thanks in advance,
Ragas
  • Comment on How to login to a form which has Javascript OnSubmit method usirng Perl
  • Download Code

Replies are listed 'Best First'.
Re: How to login to a form which has Javascript OnSubmit method usirng Perl
by moritz (Cardinal) on Nov 04, 2007 at 07:47 UTC
    I don't know if there is a sufficient javascript interpreter in Perl, but you could look at the javascript function validate_form, and mimic its behaviour in Perl.

    It doesn't just validate the form as the name suggests, but encodes the password in base64, copies that to a hidden field of the form, removes the plain text entry of the password (I don't understand the paranoia - they are using ssl after all) and does some md5 magic.

      Thank you for the suggestion. I also considered that idea of mimicing the validate_form's behavior in my code. But, even then, hitting the OnSubmit Javascript method in the form is unavoidable before POSTing. Is there any way to bypass this OnSubmit at the client side and send the values directly to the server?
        Of course you can POST any data you like to a HTTP server, with LWP::UserAgent for example, and I'm quite sure WWW::Mechanize allows that as well. Your perl script doesn't hit any button, it just sends data to the server.

        All a server does is send data to a user agent - and that data has the form of HTML with embedded javascript - it doesn' t have any kind of control over how the user agent interprets the data.

        So all your script has to do is to send the same data as a web browser would do.

        If you don't want a general solution, but only one that works for you, there might be shortcut. If you look at that function again:

        function validate_form(form) { if ( form.user_id.value == "" || form.password.value == "" ) { alert( "Enter a username and password." ); return false; } //short-cut if challenge/response is disabled. if ( !_useChallenge ) { form.encoded_pw.value = base64encode( form.password.value ); form.encoded_pw_unicode.value = b64_unicode( form.password.value ) +; form.password.value = ""; return true; } ... }

        There seems to be a chance that the system accepts login without challenge/response, so all you have to do is to login once manually, and record the the values that are stored in form.encoded_pw_unicode and in the username field, and then uses this data each time you want to log in.

Re: How to login to a form which has Javascript OnSubmit method usirng Perl
by fenLisesi (Priest) on Nov 04, 2007 at 10:44 UTC
Re: How to login to a form which has Javascript OnSubmit method usirng Perl
by warblecl0x (Initiate) on Nov 04, 2007 at 14:47 UTC
    hi,

    what I would suggest, is that you need to remember that Javascript is doing all of its work in the user's browser. And Perl is working on the server.
    what you do in this simple case... is have the submit button send the collected data to your script...

    wile the user is filling in the form, you have either a monolithic Javascript function or object or lots of little functions working on the input data.

    form fields have events... that you can attach these javascripts to... onBlur, onFocus, onClick on mouseOver, etc... so, let's say you want to check that the name and the password are at least filled in...:

    you add a handler function which checks just that... however you will have to use logic.. and some human interface ergonomics...
    so simple it is to have an annoying alert pop up... dig a little further and you can write directly into the page in an empty "div". and then finally, perhaps a global variable in Javascript... indicates whether all the tests are passed, and sure put them in the hidden fields to make url-encoded key/value pairs for your perl script to extract and perform all the real tests and send off to your data-store and finally let the user access the golden Home page of desire.

    If that last bit was not clearly enough written, try to let a javascript function actually make the call as to when to set the page location to your script-url

    once the perl script is satisfied... it then returns the URL for the page to which all this logging in was all about.

    in short, it is time to start handing off some of the work to the new kid -- JS.

    This is actually the only secret of WEB 2.0. and based on my searches on the net, PHP has taken over in most of those applications mostly because PERL is too powerful.

    The other answers to this question clearly illustrate the singlemindedness and perhaps stubborness of PERL programmers, sure perl can do it all, but if the pass and login have invalid format, or illegal characters, it seems best to let the user know before he wastes the server's time with silly stuff. And you can have perl generate all of the Javascript and html/css into the login window, so detect your browser variables, write a hash and track the user with cookies: these are the power of your Perl.

    In fact, since users are now running around with vast amounts of unused CPU power, have your PERL scripts send parts of huge problems out to user's javascipt engines and return the answers back to your script for futher routing and matrix processing, Turn your 1000's of clicks into a massively parallel network renderer...

    ...or, have I said too much?
      I think ragas wants to automate a login process with a perl client, and can't control the server side - that's why it is impossible to avoid dealing with javascript.

      All those explanations about web 2.0 , js and perl (not PERL, btw) are fine, but they don't help with the task at hand.

        I hoped that i could answer without any code snippets and yes perhaps I did extrapolate too far...
        <form method="POST" action="in(cgiurl)" name=form1 onSubmit="return Se +tPass();">

        is the basic line...

        in this case, set pass actually sets the cookie, however, if the cookie is wrong, the script won't work... lot's more could be done there... by JS... I let the perl script send the users to the templated login page... so a simple link from a public page or inline on that public page... it will work the same if you submit action to the actual address of your script. basically, your script has to deal with the info it gets... there is a data parsing function ... getdata ... called by the main() function
        if($usecgi){ use CGI; $query = new CGI; @names = $query->param; foreach $i (@names){ $in{$i} = $query->param("$i"); } } else{ # Read in text if ($ENV{'REQUEST_METHOD'} eq "GET") { $in = $ENV{'QUERY_STRING'}; } elsif ($ENV{'REQUEST_METHOD'} eq "POST") { for ($i = 0; $i < $ENV{'CONTENT_LENGTH'}; $i++) { $in .= getc; } } #.... followed by more code for parsing, splitting and assigning the n +ew variables

        this sample script parses the POST data if any... here there are 2 conditions which call the login template...

        ($in{'command'} eq 'login')&&(&Login); ($in{'command'} eq '')&&(&Login); #all require password below &GetLogin; ... # passing thru Get login allows access to more program functions and a +dditional pages. the login page actually calls the next page... via h +idden form field


        I don't see his code doing any of that. I see him trying to set up the page and sending them off to the secret content without having the server do any checking via perl.
        If it successfully presents the login page, then it will finally need to pass thru the login page to the goodies behind that. for security reasons, I can see why the javascript shoul disguise the username and password and then potentially send the data either as the cookie request or via plain text... with or without the benefits of ssl.

        I previously provided the basic hint to look at Javascript and let it do what it does and allow perl to do what it does...
        in my case, setPass() does the combination of verifying that user has filled both fields and alerting if otherwise... it then creates a hash and stores the cookies locally... when setpass is satisfied, it returns true. otherwise it returns false. and the action fails. and the hidden field provides the (CGI)action "manage" which can only be reached by passing thru the GetLogin/Cookie retrieval process. and continuing to have a cookie until the session expires. Any failures should take youu back to the login screen.
        sub GetCookies{ $cookies = $ENV{'HTTP_COOKIE'}; @allcookies = split(/;\s*/,$cookies); foreach $i (@allcookies){ ($name,$value) = split(/\s*=\s*/,$i); $cookie{$name}=$value; } }


        the perl script then knows how to extract and check the cookies , and finally, it checks the retrieved against the stored password and username
        sub GetLogin{ &GetCookies; $in{'UserName'} = $cookie{'UserName'}; $in{'PassWord'} = $cookie{'PassWord'}; if(!$in{'UserName'}){ &PageOut('t_login.htm'); exit; #back where you started, though the Javascript sees that it never com +es here } else{ (($in{'UserName'} ne $username)||(($in{'PassWord'} ne $password)))&&(& +PError("Error. Invalid username or password")); #here we need to check the final results after parsing the cookie info + and retrieving matches in the db of variables... we have no choice +but to return a server error message. } }


        one might substitute a Mechanized page for a template... so on the first call to the script, you build the page, and the second call to the script, checks the returned data...

        essentially, there is a shared variable space @in{''} which need be passed between the page (javascript) and the server (perl)

        I certainly will continue to maintain that "the old way" is to ignore javascript and just continue to pass the variables back to the serverside parsing and keep returning the errors that way. (ERRORS in actual password or login are unavoidably handled this way.)
        Yes, Perfect. You have put it short and sweet. I am trying to build a perl client for automating the login process and the Javascript and server side logic is not in my control. So I am looking for alternatives.