in reply to Perl spider

I have some experience with crawling in both languages, and I found that there's not much difference between Java and Perl when it comes to performance.

I think another choice is more important:

1) use a browser + Selenium. Heavy on resources, but comes with a lot of features that can dramatically cut development time per website.

2) use lightweight modules like WWW::Mechanize. Cheap on resources, but you will get guaranteed headaches for sites that are heavy on obscure javascript.

Crawling 300 websites once a week is not that heavy, so I would go for option 1.