Greetings fellow seekers. First-time poster here.
I need assistance in figuring out why this sub returns an undefined $rows after 5 loops.
What this sub should accomplish is to hit a webpage, parse the content from the tables, find a link at the bottom whose text is "more", and keep iterating until that "more" link doesn't exist. There are 10 pages it should be grabbing, but I get a TableExtract error after the 5th page. Data is written to my file successfully up to that point.
use strict;
use warnings;
use diagnostics;
use HTML::TableExtract;
use WWW::Mechanize;
use Time::HiRes;
my $random = rand(10);
my $huffdata = "C:/huff_data.txt";
open (MYFILE, "+>>", "$huffdata") or die "unable to open $huffdata $!"
+;
my $url = "http://fundrace.huffingtonpost.com/neighbors.php?type=name&
+lname=SMITH";
sub parse_and_save{
sleep($random);
my $mech = WWW::Mechanize->new;
$mech->get($url);
my $text = $mech->content;
my $te = HTML::TableExtract->new( headers => [qw(Donor Contribution Ad
+dress)] );;
$te->parse($text);
my $row;
foreach $row ($te->rows) {
print MYFILE join(",", @$row);
}
my @links = $mech->find_link( text_regex => qr/more/i ) or die "no lin
+ks found";
for (@links){
$url = $_->url_abs($/);
parse_and_save();
}
}
parse_and_save();
Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
Read Where should I post X? if you're not absolutely sure you're posting in the right place.
Please read these before you post! —
Posts may use any of the Perl Monks Approved HTML tags:
- a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
| |
For: |
|
Use: |
| & | | & |
| < | | < |
| > | | > |
| [ | | [ |
| ] | | ] |
Link using PerlMonks shortcuts! What shortcuts can I use for linking?
See Writeup Formatting Tips and other pages linked from there for more info.