in reply to Fetching table from website using HTML::TableExtract

I don't think HTML::TableExtract can do that for you.

Why not fix the data in Perl once you've extracted it in table form?

  • Comment on Re: Fetching table from website using HTML::TableExtract

Replies are listed 'Best First'.
Re^2: Fetching table from website using HTML::TableExtract
by Anonymous Monk on Mar 29, 2017 at 12:02 UTC
    An alternative to the cell() method in HTML::TableExtract::Table is the space() method. It is largely similar to cell(), except when given coordinates of a cell that was covered due to rowspan or colspan effects, it will return the contents of the cell that was covering that space rather than undef. So if, for example, cell (0,0) had a rowspan of 2 and colspan of 2, cell(1,1) would return undef and space(1,1) would return the same content as cell(0,0) or space(0,0).
[reply]

      Can you please help me with some example ......i am first time using table extract ..

[reply]

        Here you go 

        #!/usr/bin/perl --
use strict; 
use warnings;
use HTML::TableExtract ;
Main( @ARGV );
exit( 0 );
sub Main {
    my $te = HTML::TableExtract->new( br_translate => 0 );
    $te->parse( SkewHtml() );
    $te->eof;
    my $tab = $te->first_table_found;
    for my $rxrow ( $tab->row_indices ){
        for my $rxcol ( $tab->col_indices){
            my $cell = $tab->space( $rxrow, $rxcol );
            print "row($rxrow)col($rxcol) = $cell\n";
        }
    }   
} ## end of Main
sub SkewHtml {
    return q{
<html><head><title>skew test</title></head>
<body>
<table border=1>
<tr>
    <td>head0</td>
    <td>head1</td>
    <td>head2</td>
    <td>head3</td>
</tr>
<tr>
    <td colspan=4>THIS IS A WHOLE ROW-CELL OF JUNK</td>
</tr>
<tr>
    <td rowspan=2>JUNK</td>
    <td>Tasty tidbit (1,1)</td>
    <td>JUNK</td>
    <td>Tasty tidbit (1,3)</td>
</tr>
<tr>
    <td colspan=2 rowspan=3>BIG JUNK</td>
    <td>Tasty tidbit (2,3)</td>
</tr>
<tr>
    <td>Tasty tidbit (3,0)</td>
    <td>Tasty tidbit (3,3)</td>
</tr>
<tr>
    <td>Tasty tidbit (4,0)</td>
    <td>Tasty tidbit (4,3)</td>
</tr>
<tr>
    <td colspan=2>JUNK BUTTON</td>
    <td>Tasty tidbit (5,2)</td>
    <td>Tasty tidbit (5,3)</td>
</tr>
</table>
</body>
</html>
    };
} ## end of SkewHtml
__END__
row(0)col(0) = head0
row(0)col(1) = head1
row(0)col(2) = head2
row(0)col(3) = head3
row(1)col(0) = THIS IS A WHOLE ROW-CELL OF JUNK
row(1)col(1) = THIS IS A WHOLE ROW-CELL OF JUNK
row(1)col(2) = THIS IS A WHOLE ROW-CELL OF JUNK
row(1)col(3) = THIS IS A WHOLE ROW-CELL OF JUNK
row(2)col(0) = JUNK
row(2)col(1) = Tasty tidbit (1,1)
row(2)col(2) = JUNK
row(2)col(3) = Tasty tidbit (1,3)
row(3)col(0) = JUNK
row(3)col(1) = BIG JUNK
row(3)col(2) = BIG JUNK
row(3)col(3) = Tasty tidbit (2,3)
row(4)col(0) = Tasty tidbit (3,0)
row(4)col(1) = BIG JUNK
row(4)col(2) = BIG JUNK
row(4)col(3) = Tasty tidbit (3,3)
row(5)col(0) = Tasty tidbit (4,0)
row(5)col(1) = BIG JUNK
row(5)col(2) = BIG JUNK
row(5)col(3) = Tasty tidbit (4,3)
row(6)col(0) = JUNK BUTTON
row(6)col(1) = JUNK BUTTON
row(6)col(2) = Tasty tidbit (5,2)
row(6)col(3) = Tasty tidbit (5,3)

        [download]
[reply]
[d/l]
Re^2: Fetching table from website using HTML::TableExtract
by sachin raj aryan (Sexton) on Mar 29, 2017 at 12:16 UTC
    actually not sure how to fix the data as data i am getting from webpage directly .
[reply]

      Corion doesn't mean you fix the data in the webpage directly, but after you have extracted it with HTML::TableExtract which is properly inserting empty fields for data cells skipped due to colspan or rowspan. But you have multiline fields:

      The second row of your $ts->rows is 

      Region,Level 31.03.2016,,Sanction/Renewal
01.04.2016 to 28.02.2017,,,Level 28.02.2017,,Sanction/Renewal
During Current Month

         ,,,Level 26.03.2017,,Growth
as on
26.03.2017,

      [download]

      After Level 31.02.2016 there's an empty field because of colspan="2". The next field is

      Sanction/Renewal
During Current Month

      [download]

      so all you have to do is removing trailing whitespace/newlines from each field:

      foreach my $ts ( $te->tables ) {
    print "Table (", join( ',', $ts->coords ), "):\n";
    foreach my $row ( $ts->rows ) {
       s/[\s\n]+\z/ for @$row; # <--- here
       # s/\n/ /gs for @$row; # uncomment if you want to convert
                              # multiline fields into single line
       $OUT-> print( join( ',', @$row ), "\n");
    }
}

      [download]
      perl -le'print map{pack c,($-++?1:13)+ord}split//,ESEL'
[reply]
[d/l]
[select]
        Hi i tried your code somewhat working but still one issue below is the output

        Amount in Crores,,,,,,,,,,,,,,

        Region,Level 31.03.2016,,Sanction/Renewal01.04.2016 to 28.02.2017,,,Level 28.02.2017,,Sanction/RenewalDuring Current Month,,,Level 27.03.2017,,Growthas on27.03.2017,

        if you see multiple commas in first line and more than one in second row
[reply]

In Section Seekers of Perl Wisdom