rmperl has asked for the wisdom of the Perl Monks concerning the following question:

I'm using WWW::Mechanize::Firefox to login into a website that has a javascript heavy login process to access members only content. After the login the members content is mainly HTML. Mechanize::Firefox works for the login process but is painfully slow at scraping the site and I therefore want to use www::mechanize to do the main scraping job by using the same firefox session as I need to access the members content that I can only get when I am logged-in. I've had a go at this but it driving me mad. I'm not getting any of the members content when I switch the user agent to mechanize. Any ideas?

use WWW::Mechanize; use WWW::Mechanize::Firefox; # open firefox my $firefox = WWW::Mechanize::Firefox->new( create =>1, activate=>1, launch=>'/usr/bin/firefox', ); # stop timeout issues with firefox telnet my $url='some url'; my $new_timeout=100000; $firefox->repl->repl->client->{telnet}->timeout($new_timeout); $firefox->get($url); # log-in process $firefox->click({xpath=>'//*[@id="signInLink"]'}); sleep 60; my $username='username'; my $password='password'; # sign-in process $firefox->form_id('sign-in-form'); $firefox->set_visible($username, $password ); sleep 60; $firefox->click({xpath=>'//*[@id="button-sign-in"]'}); sleep 30; print "now logged-in\n"; # trying to get mechanize to use current mechanize::firefox session my $user_agent_name='Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:39.0) Ge +cko/20100101 Firefox/39.0'; my $agent = WWW::Mechanize->new(agent=>$user_agent_name); my $content = $agent->content(); print $content;

Replies are listed 'Best First'.
Re: www::mechanize and www:mechanize::firefox
by Corion (Patriarch) on Aug 08, 2015 at 13:02 UTC

    Most likely, you just have to extract the cookie(s) that the site stores when you've logged in with Firefox and put them into the cookie jar that WWW::Mechanize uses.

      Thanks. I wasn't sure that www::mechanize::firefox was able to save cookies.

        I've log-in into the site using www::mechanize::firefox and that works well. I'm now trying to get the cookies after logging in but I get the error message 'Can't call method "declare" on an undefined value at /usr/local/share/perl/5.18.2/HTTP/Cookies/MozRepl.pm line 108' and I'm not sure why. Any ideas on where I'm going wrong here?

        my $cookie_jar = HTTP::Cookies::MozRepl->new(); my $user_agent_name='Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:39.0) Ge +cko/20100101 Firefox/39.0'; my $agent = WWW::Mechanize->new( agent=>$user_agent_name, cookie_jar => $cookie_jar, autocheck => 1);