Re^3: Fetching table from website using HTML::TableExtract

Corion doesn't mean you fix the data in the webpage directly, but after you have extracted it with HTML::TableExtract which is properly inserting empty fields for data cells skipped due to colspan or rowspan. But you have multiline fields:

The second row of your $ts->rows is

Region,Level 31.03.2016,,Sanction/Renewal
01.04.2016 to 28.02.2017,,,Level 28.02.2017,,Sanction/Renewal
During Current Month

         ,,,Level 26.03.2017,,Growth
as on
26.03.2017,
[download]

After Level 31.02.2016 there's an empty field because of colspan="2". The next field is

Sanction/Renewal
During Current Month
[download]

so all you have to do is removing trailing whitespace/newlines from each field:

foreach my $ts ( $te->tables ) {
    print "Table (", join( ',', $ts->coords ), "):\n";
    foreach my $row ( $ts->rows ) {
       s/[\s\n]+\z/ for @$row; # <--- here
       # s/\n/ /gs for @$row; # uncomment if you want to convert
                              # multiline fields into single line
       $OUT-> print( join( ',', @$row ), "\n");
    }
}
[download]

perl -le'print map{pack c,($-++?1:13)+ord}split//,ESEL'

Comment on Re^3: Fetching table from website using HTML::TableExtract Select or Download Code

Replies are listed 'Best First'.
Re^4: Fetching table from website using HTML::TableExtract by sachin raj aryan (Sexton) on Mar 30, 2017 at 08:25 UTC
Hi i tried your code somewhat working but still one issue below is the output Amount in Crores,,,,,,,,,,,,,, Region,Level 31.03.2016,,Sanction/Renewal01.04.2016 to 28.02.2017,,,Level 28.02.2017,,Sanction/RenewalDuring Current Month,,,Level 27.03.2017,,Growthas on27.03.2017, if you see multiple commas in first line and more than one in second row	[reply]
Re^5: Fetching table from website using HTML::TableExtract by shmem (Chancellor) on Mar 30, 2017 at 09:45 UTC
if you see multiple commas in first line and more than one in second row What do you expect? You are basically outputting a CSV file in which each line has the same amount of fields, separated by commas. Nothing between two commas is an empty field produced by expanding field spanning captions of the original table denoted as e.g. `colspan="2"` in html. If those fields were omitted, the placement of the captions would not correspond to the columns of the table data. Load the output file as a spreadsheet (e.g. in Excel) and you will see the effect. I don't know the purpose of what you are doing. If you want to remove empty fields from the output, proceed as follows: `foreach my $ts ( $te->tables ) { print "Table (", join( ',', $ts->coords ), "):\n"; foreach my $row ( $ts->rows ) { s/[\s\n]+\z/ for @$row; s/\n/ /gs for @$row; $OUT-> print( join( ',', grep { length $_ } @$row ), "\n"); } }` [download] Again, load the data into a spreadsheet to see the difference. Compare to the original table on the website. perl -le'print map{pack c,($-++?1:13)+ord}split//,ESEL'	[reply] [d/l] [select]
Re^6: Fetching table from website using HTML::TableExtract by sachin raj aryan (Sexton) on Mar 30, 2017 at 11:54 UTC
thanks it worked with first solution , when i opened it in excel ..	[reply]