in reply to Extract file links from Google search results

One thing you should also keep in mind is that according to Google's TOS, they can start blocking any requests that appear to be coming from a script or as part of a search engine agglomerator. The limit, to be fair, is quite high- since some of us are perfectly capable of generating five thousand legitimate google searches in a day- but if you are building this as part of a web app that might rack up hundreds of searches a minute, you may find google no longer responds. That's why they have the Search API, which, surprise, limits you to a few thousand requests per day. :(
  • Comment on Re: Extract file links from Google search results

Replies are listed 'Best First'.
Re^2: Extract file links from Google search results
by Scott7477 (Chaplain) on Jun 25, 2007 at 15:22 UTC
    Your point is well taken; how I get my Google pages to process is through manually downloading them as I am well aware of their TOS. Clearly, using the search API would be a cleaner way to accomplish my task; I was trying for a quick and dirty solution. I happen to have an API key myself; just haven't had the time to put together an app using it yet. Interesting to know that they won't cut you off too quickly...