in reply to Perl spider
I think another choice is more important:
1) use a browser + Selenium. Heavy on resources, but comes with a lot of features that can dramatically cut development time per website.
2) use lightweight modules like WWW::Mechanize. Cheap on resources, but you will get guaranteed headaches for sites that are heavy on obscure javascript.
Crawling 300 websites once a week is not that heavy, so I would go for option 1.
|
|---|