Mr Bigglesworth has asked for the wisdom of the Perl Monks concerning the following question:

Hi All

I am trying to get data out of some HTML tables, which is largely working. i am having 2 issues and I have no idea how to resolve, and Google has not helped. I am very new to PERL and programming so I don't even know what I should be looking for to solve this issue.

Issue 1: HTML::TableExtract cannot pick up any data for the first cell. I guess this is because this cell is a html link to a *.png file, for example (line 380):

<td><img class="minimizeStyle" src="http://www.risa.com.au/JockeySilks/58035.png" /></td>.

What I would like to do is populate the first cell in each row with the 58035.png (in this case).

Issue 2: Where HTML::TabelExtract find an empty cell, it ignores it, for example (line 391):

<td style="text-align:center"></td>

Instead of ignoring it I would like a "," to be inserted so that when (eventually) a csv file is created everything will be in line.

The code I have at this stage is:

use strict;
use warnings;
use HTML::TableExtract;
use LWP::Simple;
my $html = get("http://www.risa.com.au/FreeFields/Results.aspx?Key=201
+3Aug19,VIC,Echuca");
my $te = HTML::TableExtract->new;
$te->parse($html);
# Table parsed, extract the data
binmode STDOUT, ":utf8";
foreach my $table ( $te->tables ) {
        foreach my $row ($table->rows) {
        my @values = grep {defined} @$row;
        print "   ", join(',', @values), "\n";
        }# Foreach
       
    }# Foreach

[download]

I hope someone is able to point me in the right direction.

Cheers

Mr Bigglesworth

Replies are listed 'Best First'.
Re: HTML::TableExtract issues
by choroba (Cardinal) on Aug 24, 2013 at 07:43 UTC
    HTML::TableExtract in default mode only extracts text. You can tell it to return the HTML document tree instead by using 
    use HTML::TableExtract qw(tree);

    [download]

    HTML::TableExtract is not ignoring undefined cells, you are. By grep {defined} you tell it to throw them away. Instead, use something like 

    my @values = map defined($_) ? $_ : ' ', @$row;

    [download]
    لսႽ† ᥲᥒ⚪⟊Ⴙᘓᖇ Ꮅᘓᖇ⎱ Ⴙᥲ𝇋ƙᘓᖇ
[reply]
[d/l]
[select]

      Hi choroba

      Thank you for your reply.

      my @values = map defined($_) ? $_ : ' ', @$row;

      works perfectly. I see the problem with using grep in this case.

      Changing: use HTML::TableExtract;

      to: use HTML::TableExtract qw (tree);

      yields: HTML::ElementTable::DataElement=HASH(0x3275f84), which repeats with different letters and numbers in between the HASH(), which I assume is the data within each cell of each table. More Googling required!!!

      Cheers

      Mr Bigglesworth

[reply]
[d/l]
[select]
        "What I would like to do is populate the first cell in each row with the 58035.png (in this case)."

        One way is by using the tree mode and look_down like this.

        #!perl
use strict; use warnings;
use HTML::TableExtract 'tree';
use Text::CSV;
use LWP::Simple;

# input
my $html = get('your url');
my  $te = HTML::TableExtract->new();
$te->parse($html);

# output
my $csvfile = 'results.csv';
my $csv = Text::CSV->new ( { binary => 1, eol => "\n" } ) 
          or die "Cannot use CSV: ".Text::CSV->error_diag ();
open my $fh, '>:encoding(utf-8)', $csvfile or die "$csvfile : $!";

# process
my $count=0;
printf "%3s %4s %4s\n",'Tbl','Rows','Cols';
foreach my $ts ($te->tables){

  my $tree = $ts->tree();
  printf "%3d %4d %4d\n",++$count,$tree->maxrow,$tree->maxcol;

  foreach my $r (0..$tree->maxrow){
    my @cells=();
    
    # is col 1 an img ?
    my $x = $tree->cell($r,0)->look_down('src',qr/png$/);
    push @cells,(defined $x) ? $x->attr('src') : $tree->cell($r,0)->as
+_text;
    
    for my $c (1..$tree->maxcol){
      my $val = $tree->cell($r,$c)->as_text; 
      push @cells,$val;
    }
    $csv->print ($fh, \@cells);
  }
}
close $fh or die "$csvfile: $!";

        [download]

        Notice I have used Text::CSV rather than just adding commas between columns.

        poj
[reply]
[d/l]
[select]

        The documentation for HTML::TableExtract gives you an example using the tree option: 

        # Example: Extracting as an HTML::Element tree structure
 # Rather than extracting raw text, the html can be converted into a
 # tree of element objects. The HTML document is composed of
 # HTML::Element objects and the tables are HTML::ElementTable
 # structures. Using this, the contents of tables within a document ca
+n
 # be edited in-place.

 use HTML::TableExtract qw(tree);
 $te = HTML::TableExtract->new( headers => qw(Fee Fie Foe Fum) );
 $te->parse_file($html_file);
 $table = $te->first_table_found;
 $table_tree = $table->tree;
 $table_tree->cell(4,4)->replace_content('Golden Goose');
 $table_html = $table_tree->as_HTML;
 $table_text = $table_tree->as_text;
 $document_tree = $te->tree;
 $document_html = $document_tree->as_HTML;

        [download]
[reply]
[d/l]

Back to Seekers of Perl Wisdom