Re^2: HTTP::Lite GET - too many requests?

OK, i'll have to explain this a little more. Im doing a reaserach project for my MA in sociology. I contacted site owner almost half year ago, he promised to give me access to their database. That would make my life a lot easier. The problem is that the site owner doesn't know really much about programming (he's a journalist) so that site is managed by some outsourcing guys, which *magically* haven't had time for past 6 months to do this. So, having a permission to access their data i decided to write a crawler do get this data. Do you know if it's possible to use proxy or SOCKS to get around limit of connection per IP which is most probably set on the server?

Comment on Re^2: HTTP::Lite GET - too many requests?

Replies are listed 'Best First'.
Re^3: HTTP::Lite GET - too many requests? by Corion (Patriarch) on Jul 14, 2008 at 06:15 UTC
Simply don't hammer the site. Make your requests slower, by sleeping between requests. You should sleep at least as long as it took for the last request to get processed. All other "circumvention ideas" will only lead to an arms race between you and the hosting people. Note that the hosting people have no interest in your task. They likely only care about keeping the website up and bots from crawling the website. Test your crawler on a local copy of some pages.	[reply]
Re^3: HTTP::Lite GET - too many requests? by Anonymous Monk on Jul 14, 2008 at 06:15 UTC
Its possible, but its unethical, and it could also include jail time.	[reply]