giantpanda has asked for the wisdom of the Perl Monks concerning the following question:
Hello, I'm currently coding a web crawler and I've been using WWW::Mechanize::Firefox to extract data from pages that keep loading content via JavaScript. Here's the code for this part of the script:
use strict; use warnings; use WWW::Mechanize::Firefox; # Requires MozRepl addon for Firefox use WWW::Mechanize; use Date::Manip; use DateTime; open TXT, "<", "nickname.log"; while(! eof (TXT)){ my $nick = <TXT>; chomp $nick; my $j = 0; for $j(0..3){ open MATCH, "+<", "match.log"; my (@gotmatch) = <MATCH>; my ($date) = DateTime->now; my ($k) = 0; while($k < $j){ $date = $date->subtract( days => 7); $k++; } $date = $date->ymd; $url = "http://www.quakelive.com/#profile/matches/$nick/$date" +; my ($firemech) = WWW::Mechanize::Firefox->new(); $firemech->get($url); die "Cannot connect to $url\n" if !$firemech->success(); my ($retries) = 10; while ($retries-- and ! $firemech->is_visible( xpath => '/ +/*[@class="areaMapC"]' )) { sleep 1; } die "Timeout" unless $retries; my ($content) = $firemech->content(); while(($content =~ /class="areaMapC" id="([^<]+)_([^<]+)_([^<] ++)">/gsi)){ my ($game) = $1; my ($longid) = $2; my ($shortid) = $3; my ($matchid) = "$longid/$game/$shortid\n"; # Checks match.log for duplicates my ($flag) = 0; for my $l(0..$#gotmatch){ if($matchid eq $gotmatch[$l]){ $flag = 1; } } if($flag == 0){ print MATCH $matchid; $flag = 0; } } close MATCH; undef $firemech; $j++; } }
The rest of the script (using WWW::Mechanize) runs perfectly, but this part breaks it giving these errors:
(in cleanup) Can't call method "cmd" on an undefined value at /Library +/Perl/5.10.0/MozRepl/Client.pm line 186,<DATA> line 42 during global + destruction. (in cleanup) Can't call method "execute" on an undefined value at +/Library/Perl/5.10.0/MozRepl.pm line 372,<DATA> line 42 during globa +l destruction. (in cleanup) Can't call method "is_debug" on an undefined value at + /Library/Perl/5.10.0/MozRepl/Client.pm line 188,<DATA> line 42 duri +ng global destruction.
Note that it sometimes gives only one of them, sometimes all of them and that "is_debug" is the most rare of them. Also note that the username they pop out is different every run (sometimes the 6th, others the 15th, etc).
On StackOverflow I've been told to "undef $firemech", but this didn't solve the issue. Google hasn't been helpful either so far, nor has been my Perl professor.
Thanks.
|
|---|
| Replies are listed 'Best First'. | |
|---|---|
|
Re: MozRepl cleanup problem
by Corion (Patriarch) on Oct 30, 2010 at 12:26 UTC | |
by giantpanda (Initiate) on Oct 30, 2010 at 14:58 UTC | |
by Corion (Patriarch) on Oct 30, 2010 at 16:45 UTC | |
by giantpanda (Initiate) on Oct 30, 2010 at 17:41 UTC | |
by Corion (Patriarch) on Oct 31, 2010 at 09:12 UTC | |
|