WWW::Mechanize and Cookies generated through Javascript

shanu_040 has asked for the wisdom of the Perl Monks concerning the following question:

Hi Monks,
I have a Mechanize-driven application. The server I'm talking to sets a cookie using javascript (rather than standard set-cookie), so Mechanize doesn't catch the cookie. I need to pass that cookie back on the next GET request.


sub handler {
    my $r = shift;
    my $req = Apache2::Request->new($r);
    my $mech = WWW::Mechanize->new(
                    agent => 'Mozilla/5.0',
                autocheck => 1,
        );
    
    $mech->cookie_jar(HTTP::Cookies->new);
    my $domain = 'http://somesite.org';
    my $uri = $req->param('uri');

    if(defined $uri){
        $mech->get($domain);
    }else{
        my $args = $r->unparsed_uri();
        my $base = "$domain$args";
        $mech->get($base);        
    }
    
    my $response  = $mech->response();
    my $html = $response->content;
    
    $r->content_type($mech->content_type);
    $r->content_encoding($response->content_encoding);

       print $html; 
        return Apache2::Const::OK;

}
[download]

Would really appreciate it if some one could guide me. Thanks Amit

Comment on WWW::Mechanize and Cookies generated through Javascript Download Code

Replies are listed 'Best First'.
Re: WWW::Mechanize and Cookies generated through Javascript by jethro (Monsignor) on Aug 03, 2010 at 11:03 UTC
This question comes up quite often here on perlmonks so you might search for relevant threads with Super Search AFAIK you could either decipher the javascript yourself and just mimic what the javascript does, or use WWW::Mechanize::Firefox which uses firefox for the execution of the javascript. Alternatively WWW::Selenium is able to execute the javascript too (see also the tutorial Using WWW::Selenium To Test Or Automate An Ajax Website)	[reply]
Re^2: WWW::Mechanize and Cookies generated through Javascript by shanu_040 (Sexton) on Aug 04, 2010 at 04:42 UTC
Hi, As I am using this code to create a proxy server through url rewriting. The code works fine for displaying requested site. But when I try to login through web-browser, then problem starts. And I find that is because of cooikes. Please guide me that I am on right track or not. I am trying something like URL Rewriting Any further guidance would appreciable. Thanks	[reply]
Re^3: WWW::Mechanize and Cookies generated through Javascript by jethro (Monsignor) on Aug 04, 2010 at 11:40 UTC
The link you posted only tells about how to reroute a connection attempt to a proxy, not how the proxy itself should work. It doesn't change the validity of my and rowdogs answers. Have you tried WWW::Scripter? Have you tried to disassemble the javascript code?	[reply]
Re^4: WWW::Mechanize and Cookies generated through Javascript by shanu_040 (Sexton) on Aug 04, 2010 at 12:09 UTC
Re^5: WWW::Mechanize and Cookies generated through Javascript by jethro (Monsignor) on Aug 04, 2010 at 15:45 UTC
Re^3: WWW::Mechanize and Cookies generated through Javascript by shanu_040 (Sexton) on Aug 04, 2010 at 12:59 UTC
I found that it not the cookies that causing the problem..It the post request. How can I handle post method in mechanize. I also like to know how it works..I posted the code which works in same. When I submit the login page then problem starts due to form method POST. If I replace post to get it works well	[reply]
Re: WWW::Mechanize and Cookies generated through Javascript by rowdog (Curate) on Aug 03, 2010 at 11:56 UTC
Jethro's advice is good but I would be careful about launching a browser out of a mod_perl instance. I would try to scrape the cookie from the source with a regex first, then turn to the heavier user agents. An alternative might be WWW::Scripter with WWW::Scripter::Plugin::JavaScript which only pulls in spidermonkey rather than a full browser.	[reply]