Re^2: ...How to parse search engine results fast?

Replies are listed 'Best First'.
Re^3: ...How to parse search engine results fast? by Fletch (Bishop) on Feb 03, 2005 at 16:46 UTC
I have an idea that the best architecture in the world will process 0 results a second if the site won't return any because the querying IP or network's been blocked because it's not following the site's rules (and doing that for a major search engine is sure to get you in on your hosting provider's or ISP's good side).	[reply]
Re^3: ...How to parse search engine results fast? by Grygonos (Chaplain) on Feb 03, 2005 at 16:28 UTC
I would familiarize yourself with the robots.txt document for each site you are scraping. Google's robots.txt Grygonos	[reply]