Great subject line... =)

You may wish to look at a HTML::Parser or a more specific module like HTML::TableExtract rather than throwing sed, awk and such at it. It has the annoying TD per line layout that will make it a real nightmare for breaking on columns. Worse, half the names are links and half aren't.

If you really wish to stick to tinkering with it yourself, at least there is only one table. You should be able to rip off the beginning to the <TABLE> tag, rip from the </TABLE> tag to the end, remove everything in <TH> containers, remove all <TDgt; and <TR> and </TD></TR> tags, change all remaining </TD<\n sequences into ";" or "|" or something you like.

And then, rewrite it everytime it breaks when they reformat that page.

--
$you = new YOU;
honk() if $you->love(perl)

I decided to use your post as an opportunity for myself to learn HTML::Parser.

Being my first time at this module, it took ~2 hours to write, with the last 10% of the code
taking 90% of the time...

( FYI: that was in getting the <a> tags within the <td> tags to parse correctly )

So, here ya go -- using HTML::Parser and LWP::UserAgent :

#!/usr/bin/perl -w

use strict;

use LWP::UserAgent;
use HTML::Parser;

my ( $href, $ua, $req, $resp,
     $tmp, $i, $p, $tr, @stats );

$href = "http://setiathome.ssl.berkeley.edu/stats/country_7.html";

$ua   = LWP::UserAgent->new();
$req  = new HTTP::Request('GET', $href);
$resp = $ua->request($req);

sub get_table_text {

   return unless $i < 3;

   my $self = shift;
   my $text = shift;

   $self->handler( text => sub { return if shift eq "" }, "dtext" );

   ( $text = $text ) =~ s/^\d+\)\s(.*)$/$1/;

   if    ( $i == 1 ) { $tmp .= $text; }
   elsif ( $i == 2 ) { chomp ($tmp .= ":$text") }

}

sub grab_href_text {

   my $self = shift;

   $self->handler( text => sub { return if shift eq "" }, "dtext" );

   $tmp .= shift;

}

sub end_table {

   return unless shift eq "tr";

   push(@stats, "$tmp\n");
   undef $tmp;
   $i = 0;

}

sub start {

   my ( $tag, $self ) = @_;

   return unless $tag =~ /^(tr|td|a)$/;

   $tag =~ /td/ && do {
      $i++;
      $self->handler( text => \&get_table_text, "self, dtext" );
   };

   $tag =~ /a/ && do {
      $self->handler( text => \&grab_href_text, "self, dtext" )
   };

   $self->handler( end => \&end_table, "tagname, self");

}


$p = HTML::Parser->new( api_version => 3 );

$p->handler( start => \&start, "tagname, self" );

$p->parse( $resp->{'_content'} );

print @stats;
[download]

Hope that's educational/usefull ... despite the conspicuous lack of comments!

(c8=

#!/usr/bin/perl -w

open TEXT, 'country_7.html' || die $!;

my $text;

while(<TEXT>) { ## lot of ram...
  chomp;
  $text .= $_;
}

$text =~ s/<\/table.*//;
my @lines = split /<tr>/im, $text;
shift @lines; ## drop trailing html
shift @lines; ## ...and <th>..

foreach my $line (@lines) {
    $line =~ s/<\/tr>//i;
    $line =~ s/<\/td>//gi;
    my @values = split /<td[\sa-zA-Z=]*>/im, $line;
    shift @values;
    print '|';
    foreach my $value (@values) {
        $value =~ s/<\/a>//;
        $value =~ s/&nbsp;//;
        $value =~ s/<a .+>//;
        print $value , "|";
    }
        print "\n";
}
[download]

#!/usr/bin/perl -w

use strict;
use LWP::Simple;

my $page = get ("http://setiathome.ssl.berkeley.edu/stats/country_7.ht
+ml");

my @lines = split /<tr>/, $page;

for (@lines) {
    # Take out the links (for the lines that have 'em)
    s/<a.*?>(.*)<\/a>/$1/g;
    # Take out the silly &nbsp;
    s/&nbsp;//g;
    # Match the 2 parts you want
    m/<td>(.*?)<\/td>.*?(\d+)/isg;
    # And print it
    print "$1 : $2\n";
}
[download]

Gary Blackburn
Trained Killer