in reply to Need to Improve Scraping Speed

There are many possibilities, but the simplest are: don't fetch old data (which probably doesn't change), or ask the site owner to provide an easily archive which can easily be downloaded.

If that's both not possible, you could send multiple requests in parallel. But that should really be a last resort.

Replies are listed 'Best First'.
Re^2: Need to Improve Scraping Speed
by cheech (Beadle) on Dec 03, 2009 at 19:59 UTC
    you could send multiple requests in parallel. But that should really be a last resort.

    Can you point me to some info on parallel requests and why it should be a last resort please?

    Thanks

[reply]

In Section Seekers of Perl Wisdom