Okay I know stuff like this has been done before (Grab weather from yahoo, etc.) but I thought I'd post it anyway for some feedback. This script grabs the major headlines off of http://news.yahoo.com/ and prints them in their own table. Works great by calling it via an exec SSI or a system call in perl. I could have probably gotten away with using just LWP::Simple and I probably didn't have to use HTML::TokeParser but it makes it a little more custamizable, i.e. it'd be easy to strip the image tags etc. 
#!/usr/bin/perl

use strict;
use LWP::UserAgent;
use HTML::TokeParser;

### Main Program ###

my $ua = LWP::UserAgent->new;
my $rq = HTTP::Request->new('GET', 'http://news.yahoo.com');
my $r = $ua->request($rq);

my ($content, $parser, $token, $token2);
if ($r->is_success) { # parse the contents of the page
    $content = $r->content;
    $parser = HTML::TokeParser->new(\$content);
    while ($token = $parser->get_token) {
        if ($token->[0] eq 'T' && $token->[1] =~ /Top (\s+|\n)Story/) 
+{
            print "<table border=0 cellspacing=0 cellpadding=3 width=\
+"100%\"><tr bgcolor=dcdcdc><td><font face=\"arial,helvetica\"><b>";
            print $token->[1];
            while ($token2 = $parser->get_token) {
                if ($token2->[0] eq 'S') {
                    print $token2->[4];
                } elsif ($token2->[0] eq 'E') {
                    print $token2->[2];
                } else {
                    print $token2->[1];
                }
                last if $token2->[1] =~ /More\s+Health\s+Headlines/;
            }

        }
    }
    print "</b></a></font></td></tr></table>";
}

[download]

Replies are listed 'Best First'.
RE: Get yahoo's news headlines
by merlyn (Sage) on Jul 12, 2000 at 00:26 UTC
[reply]
      I believe Yahoo makes their headlines available via RSS, which is a whole lot simpler to extract than this.
      Merlyn, this is very interesting to me. I haven't been able to find out anything on Yahoo's site about this. Can you please write any more details you have here?

      I'd really like to see more companies distributing information in a more 'programmer-oriented' format, but I suppose they haven't got much incentive to do it. How often have my scripts been broken by sites changing their format? Only recently my SMS script has been broken (see my home node for info). Bravo to Fedex for being the only company I know that makes their services available through an API -- they provide TrackAPI and ShipAPI. Without companies producing APIs etc like this, or content in a standardised structured format, the full benefit of the 'information revolution' (apologies for using cheesy marketing phrase) cannot be realised.

      On a side note, does anyone have any idea how to parse RDF in Perl? I was under the impression that RDF was just another XML DTD, but at the W3C site I couldn't find out much about this, and although they did have some Perl code, it wasn't very useful. Any help is appreciated.

      <rant;> Thank you to DMOZ etc for providing no real help with parsing their RDF dir dumps </rant>

[reply]
        Try XML::RSS. It uses XML::Parser.

        Example from the docs: 

            my $rss = new XML::RSS;
    $rss->parsefile("fm.rdf");

    foreach my $item (@{$rss->{'items'}}) {
        print "title: $item->{'title'}\n";
        print "link: $item->{'link'}\n\n";
    }

        [download]
[reply]
[d/l]

Back to Cool Uses for Perl