GaijinPunch has asked for the wisdom of the Perl Monks concerning the following question:

Hi

I'm actually using a WWW::Scripter, a subclass of WWW::Mechanize. So, my script for the most part works. I had an issue where Javascript detection was being used to detect non-browsers. My PHP Curl script which had worked fine for years all of a sudden broke. Using WWW::Scripter (recommended in a thread started here!) has basically saved me, although there are some pitfalls.

my $cookie_jar  = HTTP::Cookies->new( file => 'cookie_jar.dat', autosa
+ve => 1 );
my $w           = new WWW::Scripter(    agent => 'Mozilla/5.0 (X11; Li
+nux x86_64; rv:2.0) Gecko/20110411 Firefox/4.0',
                                        cookie_jar => $cookie_jar ) ;
$w->add_header( 'Accept' => 'text/html,application/xhtml+xml,applicati
+on/xml;q=0.9,*/*;q=0.8' );
$w->add_header( 'Accept-Language' => 'en-us,en;q=0.5' );
$w->add_header( 'Accept-Encoding' => 'gzip, deflate' );
$w->add_header( 'Encoding' => 'euc-jp' );
$w->add_header( 'Keep-Alive' => '115' );
$w->add_header( 'Connection' => 'keep-alive' );
$w->add_header( 'Cache-Control' => 'max-age=0' );
$w->add_header( 'Referer' => undef );
$w->use_plugin( 'JavaScript' );
$w->get( $page_url );

[download]
As you can see, I'm not doing anything ninja-like. Pretty standard. Using HttpFox and with Curl in debug mode, I saw that calling get( $page_url ) would send a couple of cookies, and authenticate my session. The essence of the script is to login. However (I assume due to the Javascript parsing) the performance is quite bad. Having to not go through the 2-page login process for each instance would help. Unfortunately I can't see if it's sending the cookie or not. I've got the following set, as per WWW::Mechanize's FAQ 
use LWP::Debug qw(+);

[download]
...but I'm not seeing any extra output. Am I missing something about setting the cookie jar? Am I missing something about LWP::Debug?

Happy Friday to each and all!

Replies are listed 'Best First'.
Re: WWW::Mechanize Cookie debugging
by Corion (Patriarch) on Jul 01, 2011 at 06:31 UTC

    I don't see anything immediately wrong with how you set up the cookie. I would look at what the script sends using Wireshark. If you don't see the cookie header, chances are that it's not getting (and thus, not sending) the cookie.

[reply]
Re: WWW::Mechanize Cookie debugging
by Anonymous Monk on Jul 01, 2011 at 06:33 UTC

    I've got the following set, as per WWW::Mechanize's FAQ use LWP::Debug qw(+);

    Maybe you should read perldoc LWP::Debug, because qw(+) hasn't worked in years, since 2008-12-05 Release 5.822

[reply]
      That would make sense... Would be nice of Mechinize's page was updated. O_o Will mess around w/ Fireshark. My guess is the cookie isn't being sent, but want to make sure.
[reply]
Re: WWW::Mechanize Cookie debugging
by Khen1950fx (Canon) on Jul 01, 2011 at 09:18 UTC
    I think that the problem is with LWP. I kept having problems with "add_cookie_header" every time I tried to get, so I added a couple of handlers. Here's what I did. 
    #!/usr/bin/perl

use strict;
use warnings;
use LWP;
use WWW::Scripter;
use HTTP::Cookies;

my $cookie_file = '/root/Desktop/cookie_jar.dat';

my $cookie_jar = HTTP::Cookies->new(
    file     => $cookie_file,
    autosave => 1
);
 
my $ua = LWP::UserAgent->new;
$ua->default_header('Accept-Encoding' =>
     scalar HTTP::Message::decodable());
$ua->add_handler("request_send",  sub { shift->dump; return });
$ua->add_handler("response_done", sub { shift->dump; return });
$ua->cookie_jar($cookie_jar);
$ua->get('http://www.kcra.com');

    [download]
    Don't forget to delete the cookie_jar.dat. The cookie expires in 10 years:).
[reply]
[d/l]
      Thanks, but I've still got the same error. I ran Wireshark and am seeing 'roughly' the same stuff being sent. Specifically, the cookies are sent which is curious. However, there's one that looks different:

      When I load the page in Firefox or Chrome, It shows I'm sending 'Connection: keep-alive'. In my script, I'm sending 'TE: deflate,gzip;q=0.3' which I'm not sending at all in a browser, along with 'Connection: keep-alive, TE, close'. That is the only difference I can find now. :(
[reply]

Back to Seekers of Perl Wisdom