in reply to Re: ...How to parse search engine results fast?
in thread ...How to parse search engine results fast?

Ciao, I don't think so, for example why dogPile download 200 results from 4 different search engines in 1 second and with my P4 1.7 GHZ, 512 MB RAM, 100Mbit (totally free machine) I download from google top 100 in 1,5 sec?


Hardware matter in a high load envirnment...
  • Comment on Re^2: ...How to parse search engine results fast?

Replies are listed 'Best First'.
Re^3: ...How to parse search engine results fast?
by saskaqueer (Friar) on Feb 04, 2005 at 02:53 UTC

    Obviously, dogPile doesn't submit a search to each of the search engines every time you enter something into dogPile. For one, that would be very foolish for speed (as your problem is showing). Surely dogPile saves the results it fetches from the engines and reuses that the next time someone else queries for the same search.

    So for example, you search for 'hello world'. dogPile sees that these search terms haven't been fetched before, so dogPile queries all the search engines. Next time someone searches for 'hello world', dogPile doesn't need to refetch the search results since it cached them on its own servers.