Extracted HTML table

mcoblentz has asked for the wisdom of the Perl Monks concerning the following question:

I'm scraping a HTML table of Cruise Ships and their locations, etc., using HTML::TableExtract and I wanted to use a "less brittle" approach. I know the desired column names in the table, but trying to extract based on that is yielding zero results. I can get the table by defining the border but I didn't really want that.

# get the data from the web.  Typically this is:
# http://www.sailwx.info/shiptrack/cruiseships.phtml
# Either pass this in as --url <page_url> when invoking or just set it
+.

$cols = 'Ship,last reported (UTC),position,Callsign';
$url = "http://www.sailwx.info/shiptrack/cruiseships.phtml";

my $input;

my $out_fn = 'C:\\Program Files\\cron\\Cruise Ships\\ship_data.csv';
open(my $out_fh, '>', $out_fn)
   or die("Unable to create output file \"$out_fn\": $!\n");

my $m = WWW::Mechanize->new();
   $m->get($url);
   $input = $m->content;

my $te;
if ( defined ($cols)) 
{
  my @headers = split(/,/, $cols);

te = HTML::TableExtract->new( attribs => { border => 1 } );
} 
  else 
   {
    $te = new HTML::TableExtract( depth => $depth, count=>$count); 
   }
$te->parse($input);
[download]

Neither of the following two extraction approaches seemed to work on the HTML. I had to find the table by border (as you can see in above code). This results in something more fragile than I would like.

I thought at least one of these two lines would work but they don't:

$te = new HTML::TableExtract ( headers => [qw(Ship position)] );

$te = new HTML::TableExtract(headers=>\@headers);
[download]

I even tried this line...

$te = HTML::TableExtract->new( headers => \@headers);
[download]

... but it doesn't work either. Well, I didn't really think it was a problem with spaces. I don't know what the problem is.

Comment on Extracted HTML table Select or Download Code

Replies are listed 'Best First'.
Re: Extracted HTML table by pc88mxer (Vicar) on Mar 20, 2008 at 06:34 UTC
hmmm... could it be an OS issue? All of the above examples work for me on a Linux box. Exactly what failure are you seeing? My test code: use WWW::Mechanize; use HTML::TableExtract; $url = "http://www.sailwx.info/shiptrack/cruiseships.phtml"; my $input; my $m = WWW::Mechanize->new(); $m->get($url); $input = $m->content; my $te; # $te = HTML::TableExtract->new( attribs => { border => 1 } ); # works # $te = HTML::TableExtract->new(headers => ['Ship', 'position']); # wo +rks # $te = HTML::TableExtract->new(headers => [qw(Ship position)]); # wor +ks $te = new HTML::TableExtract ( headers => [qw(Ship position)] ); # wor +ks $te->parse($input); foreach my $row ($te->rows) { print Dumper($row); use Data::Dumper; } [download]	[reply] [d/l]
Re: Extracted HTML table by wfsp (Abbot) on Mar 20, 2008 at 07:14 UTC
This works for me (Win XP, Perl Activestate 5.8.8). `#!/usr/local/bin/perl use strict; use warnings; use Data::Dumper; use HTML::TableExtract; use WWW::Mechanize; $Data::Dumper::Indent = 1; my $url = "http://www.sailwx.info/shiptrack/cruiseships.phtml"; my $m = WWW::Mechanize->new(); $m->get($url); die unless $m->success; my $input = $m->content; my $te = HTML::TableExtract->new( headers => [qw(Ship position)] ) or die qq{$!}; $te->parse($input); foreach my $row ($te->rows) { print Dumper($row); }` [download] outputs (extract) `$VAR1 = [ 'Carnival Freedom', "N 18\x{b0}30', W 077\x{b0}06'" ]; $VAR1 = [ 'Carnival Glory', "N 19\x{b0}24', W 064\x{b0}48'" ]; $VAR1 = [ 'Azamara Journey', "S 35\x{b0}06', W 056\x{b0}48'" ];` [download]	[reply] [d/l] [select]
Re^2: Extracted HTML table by mcoblentz (Scribe) on Mar 20, 2008 at 14:24 UTC
Well, I'm scratching my head here. When I run any of the alternative queries based on "header" (ship position, etc.), I get an empty file. When I run the query based on table border, I get the ships. I don't make any other changes. Hmm. I'll keep looking. Thanks for checking.	[reply]
Re: Extracted HTML table by poolpi (Hermit) on Mar 20, 2008 at 07:34 UTC
# Are you behind a proxy? $m->proxy('http', ftp','http://myproxy.appli:8080/'); # Do you have something in $input ? $input = $m->content; Your code works for me too on my Linux and XP box PooLpi 'Ebry haffa hoe hab im tik a bush'. Jamaican proverb	[reply]