in reply to Newbie question in threading

See Re: How to create thread pool of ithreads for a good skeleton of how to use a pool of worker threads. I think that you'd be better served by not using threads but an asynchrous HTTP framework like AnyEvent::HTTP or POE to drive the communication, as multiple threads won't gain you much performance, as most of your processing will be limited by the throughput of the network connection between your machine and the the server(s).

Also note that when hitting a host repeatedly, make sure to throttle your requests to pause at least as long as the last request took before you launch a new request, so the target machine does not get overloaded.

As a last note, scraping Google is against Googles terms of use, so you might or might not violate local law.

Replies are listed 'Best First'.
Re^2: Newbie question in threading
by karthick (Initiate) on Apr 09, 2010 at 11:36 UTC
    thanks corion..!! The google url in the code is not for scrapping .I honestly respect terms of use of google.And am going through the alternatives u suggested. Thanks Again..!!