I believe you are on the right track. The first thing that I see that you are doing wrong is at the following line:

my $stream = HTML::TokeParser->new('$filename')
  || die "Couldn't read HTML file $filename: $!";
[download]

Second, you have:

print URI->new_abs($next[0][2]{'href'}, $filename), "\n";
[download]

Lastly, you have next Token;, but you don't have a label Token, and you realistically don't need the next their either as it will immediately go into the next loop whether or not the next is there.

-enlil

If you switch to HTML::TokeParser::Simple, I think you'll be happy with how much clearer the logic is.

use strict;
use HTML::TokeParser::Simple;
use LWP::Simple;
use URI;

my $url = 'http://www.reuters.com/newsEarlierArticles.jhtml?type=busin
+essNews';

my $stream = HTML::TokeParser::Simple->new(\get($url))
  || die "Couldn't read $url: $!";

while(my $token = $stream->get_token) {
    next unless 
        $token->is_start_tag('td') 
            and 
        ($token->return_attr('class') || '') eq 'earlyHeadline';

    my $next = $stream->get_token;
    if ($next->is_start_tag('a'))  {
        print URI->new_abs($next->return_attr('href'), $url), "\n";
    }
}
[download]

Hey,

I adjusted my code correctly to extract the urls from the reauters headlines.

However, when printing out the urls it looks like:



http://www.reuters.com/newsArticle.jhtml;jsessionid=1GRGO0RUSCREMCRBAE
+0CFFA?type=businessNews&storyID=4512094告on=news 
http://www.reuters.com/newsArticle.jhtml;jsessionid=1GRGO0RUSCREMCRBAE
+0CFFA?type=businessNews&storyID=4512054告on=news 
http://www.reuters.com/newsArticle.jhtml;jsessionid=1GRGO0RUSCREMCRBAE
+0CFFA?type=businessNews&storyID=4512041告on=news
[download]

If you copy and paste one of those url's, it will bring you to a blank reuters template, part being because at the end part of the url where it has "告on=news", should really be "'&'section=news".

Somehow its translating the "'&'section=news" into "告on=news".

Could it be because I'm using MIME-Base32 and not MIME-Base64 module? --I'm on a Windows machine.

Adjusted code:

#!/usr/bin/perl -w

use strict;
use HTML::TokeParser;
use LWP::Simple;
use URI;
print "Content-type: text/html\n\n";

my $filename = 'temp.html';

open FH, ">$filename";
print FH get("http://www.reuters.com/newsEarlierArticles.jhtml?type=bu
+sinessNews");
close FH;

my $stream = HTML::TokeParser->new($filename)
  || die "Couldn't read HTML file $filename: $!";

while(my $token = $stream->get_token) {

    if ($token->[0] eq 'S' and $token->[1] eq 'td' and
       ($token->[2]{'class'} || '') eq 'earlyHeadline') {

my(@next) = ($stream->get_token);

if ($next[0] and $next[0][0] eq 'S' and $next[0][1] eq 'a' and defined
+ $next[0][2]{'href'} )  {
    #early headline found for business section/grab a href portion
            print URI->new_abs($next[0][2]{'href'}, 'http://www.reuter
+s.com/'), "\n";
        }

    }

  }
[download]

Thank you,
Anthony

I used it when I was doing something very similar to what you're doing for parsing headlines on multiple web sites and it made the whole process quite easy.

Hope that helps!

Update: Thanks to suggestions from b10m and graff I'm including the code here so future monks can find it in a super search.

#!/usr/bin/perl -w

# HTML::TokeParser dumper
#
# quick & dirty code to print out TokeParser output

use strict;
use HTML::TokeParser;
use LWP::Simple;

print "Content-type: text/html\n\n";

my $filename = 'temp.html';

open FH, ">$filename";
print FH get("http://www.buchanie.co.uk/news.asp");
close FH;

my $stream = HTML::TokeParser->new($filename)
  || die "Couldn't read HTML file $filename: $!";

while(my $token = $stream->get_token) {
  if ($token->[0] eq "S"){
      print "Token:S 1:$token->[1]\n";
      foreach my $key(keys %{$token->[2]}){
              print "Key: $key Value: ${$token->[2]}{$key}\n";
              }
      print "3: @{$token->[3]}\n4: $token->[4]\n\n";
      }
  elsif ($token->[0] eq "E"){
         print "Token:E 1:$token->[1] 2: $token->[2]\n\n";
         }
  elsif ($token->[0] eq "T"){
         print "Token:T 1:$token->[1]\n\n";
         }
  elsif ($token->[0] eq "C"){
         print "Token:C 1:$token->[1]\n\n";
         }
  elsif ($token->[0] eq "D"){
         print "Token:D 1:$token->[1]\n\n";
         }
  else {print "Unknown token $token\n\n";}
  }
[download]

Rather than providing a link to your scratchpad, why not post that code in some more stable wing of the Monastery (or include it in your reply), to make it a stable reference? People are likely to find this thread in a search for tips on HTML parsing at any time over the coming months or years, and you're likely to have put something else on your scratch pad by then...

Actually b10m suggested the same thing so I'm taking the advice of both of you and updating my node. :)

There is no emoticon for what I'm feeling now.

Hello,

-Sheep