I use Perl to get (a subset of) a well-known newspaper to read on my Palm. Let's call it the Yew Nork Times.

Currently I do this what I think of as "the dumb way", which is, log in via WWW::Mechanize, get() various pages, grab links from one particular chunk of the HTML, append the magic 'print-friendly' request to the query string, get() and save the result -- and it all works well enough. If they change their code, I notice because the download fails, I fiddle with the script a bit, and get back on track.

But years of hanging around with the Monks have made me see that it's a little kludgy. It's not a solution I can easily update, it relies on using regexes not a parser, it's not a solution I could pass to anyone else, and it's not a solution I can generalise (I also like to read nolaS magazine which would prefer me to pay for this service, and I have another script for that).

So, what's the best way to write a scraper? I'm seeing some kind of general scraping module, which would open a particular file or data structure, as in, YNT.scrape would somehow contain a hash with all the details for the YNT, and nolaS.scrape the details for nolaS and so on.

Essentially the problem looks like: 

with       (an arbitrary number of starting pages)
get        (a number of links matching a certain regex or
            HTML-parsing expression, which might be 
            different for different pages or a default)
optionally (adding certain flags to query strings
            or transforming URLs)
save to    (a specified HD location with a specified 
            filename)

[download]
and the solution would be one where you would just swap out the YNT.scrape for a new one, or edit it, rather than have to dig into the code.

($_='kkvvttuu bbooppuuiiffss qqffssmm iibbddllffss')
=~y~b-v~a-z~s; print

In reply to How To Write A Scraper? by Cody Pendant

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.