I eventually came up with the idea of using the command line web browser Lynx to parse the HTML from Amazon.com for me. If you call Lynx with the '-dump' option, Lynx will parse the HTML and provide you with a nicely formatted stream devoid of HTML tags. If you pipe this stream into your Perl script, regexps and conditional statements are all that's needed for you to extract the data you require.

Although my Amazon.com script was quite long and contained a lot of if...elsif statements, I feel that it was a hell of a lot better than struggling with pure regexps and/or HTML::Parser.

My code is along the lines of:

$Amazon_URL = "http://www.amazon.com/exec/obidos/asin/";
$ISBN = "";

### Insert some code to get the ISBN

$Amzon_URL .= $ISBN;

open(FILEHANDLE, "lynx -dump $Amazon_URL|") 
    or die ("Can't get book data!");
@book_data = <FILEHANDLE>;
close(FILEHANDLE);
[download]

Then, you can just feed the @book_data array into your own parser procedure.

For the parser procedure, I looped through the @book_data array look for what I called "markers" in the parsed HTML. These are just bits of text that occur near the data you are looking for. Using a marker, I extracted the data fields I wanted by using offsets from the line on which the marker occured.

In my opinion, for complicated HTML web pages, this method is a lot easier than using plain regexps or HTML::Parser. However, for simpler pages, HTML::Parser and/or regexps are all that's really needed.

I hope this helps.

I agree, Lynx -dump makes life a lot easier (lazyness being my personal favourite virtue!). On the other hand, I'm not doing anything complex with the page... anyway, here's a simple example...

#!/usr/bin/perl -w
use strict;

use Mail::Mailer;

my $recipient ="my email address";
my $sender = "trains\@myhost.co.uk";
my $subject = "trains";

my $mailer = Mail::Mailer->new("sendmail");
$mailer->open({From => $sender,
                To => $recipient,
                'Content-Type' => "text/plain",
                Subject => $subject }) or die "can't open sendmail";

open (LYNX,'lynx -dump http://www.londontransport.co.uk/rt_home.shtml 
+|');

while (<LYNX>) {
        s/(\[.*\])//g ;
        print $mailer $_ if (/Real time news/ .. /References/);
}

close(LYNX);

$mailer->close();
[download]

(the regexp with the square brackets just gets rid of the image text, otherwise you get a certain amount of '[spacer.gif]')

andy.

Well modules are a great thing in many cases but I don't think a module knows what I want to grab off weatherchannel.com. I would rather some more hardcoding than module use because I don't want to save anything to my system as it says in that documentation. Instead I would rather have it like statswhore.pl. It grabs the information and tells a couple stats in a nicely printed fashion. Thanx for your help but it was a little off what I was looking for :-)

Wanna be perl hacker.
Dave AKA damian
I encourage you to email me

I can't speak for Beatnick, but my impression is that the suggestion is to use WebFetch as a basis to fill your own needs. Download it, play with it, see what comes close to doing what you want, copy it and modify. It's a great way to learn and it gives you the opportunity to give back to the project (chances are if you find a certain application useful (e.g. Weather Channel, TV listings, whatever), someone will too.)

To answer your original question, some combination of LWP and HTML::Parser (or one of its subclasses) will do what you want.