Re: ...How to parse search engine results fast?

You might get better results from Google using their API (not to mention that scraping them is against their Terms of Service . . .).

Comment on Re: ...How to parse search engine results fast?

Replies are listed 'Best First'.
Re^2: ...How to parse search engine results fast? by A200560 (Novice) on Feb 03, 2005 at 15:58 UTC
thanks, but my question have an architectural flavor... Do you have some idea?	[reply]
Re^3: ...How to parse search engine results fast? by Fletch (Bishop) on Feb 03, 2005 at 16:46 UTC
I have an idea that the best architecture in the world will process 0 results a second if the site won't return any because the querying IP or network's been blocked because it's not following the site's rules (and doing that for a major search engine is sure to get you in on your hosting provider's or ISP's good side).	[reply]
Re^3: ...How to parse search engine results fast? by Grygonos (Chaplain) on Feb 03, 2005 at 16:28 UTC
I would familiarize yourself with the robots.txt document for each site you are scraping. Google's robots.txt Grygonos	[reply]