in reply to Perl spider

Well, slurping through text is what Perl was built for.

If most of what you need is already on CPAN, it's a sure win.