in reply to Re^3: Question regarding web scraping
in thread Question regarding web scraping
Can I also ask (just one last question, sorry!), I've been reading that it's good protocol to slow down requests to avoid the website potentially banning the IP address. I inserted a simple "sleep 60" into the code that you have every kindly written for me. This seems to be working very well and has successfully staggered every request by one minute.
However, I was then told that the pauses between requests should be random (as opposed to patterned every 60 seconds).
Do you have any thoughts on this?
Thanks again for all of your help!
|
|---|
| Replies are listed 'Best First'. | |
|---|---|
|
Re^5: Question regarding web scraping
by Corion (Patriarch) on Oct 23, 2016 at 08:21 UTC |