rmperl has asked for the wisdom of the Perl Monks concerning the following question:
I'm using WWW::Mechanize::Firefox to login into a website that has a javascript heavy login process to access members only content. After the login the members content is mainly HTML. Mechanize::Firefox works for the login process but is painfully slow at scraping the site and I therefore want to use www::mechanize to do the main scraping job by using the same firefox session as I need to access the members content that I can only get when I am logged-in. I've had a go at this but it driving me mad. I'm not getting any of the members content when I switch the user agent to mechanize. Any ideas?
use WWW::Mechanize; use WWW::Mechanize::Firefox; # open firefox my $firefox = WWW::Mechanize::Firefox->new( create =>1, activate=>1, launch=>'/usr/bin/firefox', ); # stop timeout issues with firefox telnet my $url='some url'; my $new_timeout=100000; $firefox->repl->repl->client->{telnet}->timeout($new_timeout); $firefox->get($url); # log-in process $firefox->click({xpath=>'//*[@id="signInLink"]'}); sleep 60; my $username='username'; my $password='password'; # sign-in process $firefox->form_id('sign-in-form'); $firefox->set_visible($username, $password ); sleep 60; $firefox->click({xpath=>'//*[@id="button-sign-in"]'}); sleep 30; print "now logged-in\n"; # trying to get mechanize to use current mechanize::firefox session my $user_agent_name='Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:39.0) Ge +cko/20100101 Firefox/39.0'; my $agent = WWW::Mechanize->new(agent=>$user_agent_name); my $content = $agent->content(); print $content;
|
|---|
| Replies are listed 'Best First'. | |
|---|---|
|
Re: www::mechanize and www:mechanize::firefox
by Corion (Patriarch) on Aug 08, 2015 at 13:02 UTC | |
by rmperl (Novice) on Aug 08, 2015 at 13:12 UTC | |
by rmperl (Novice) on Aug 08, 2015 at 13:53 UTC | |
by Corion (Patriarch) on Aug 08, 2015 at 14:12 UTC | |
by shagbark (Acolyte) on Apr 02, 2016 at 06:21 UTC |