in reply to Re^2: screen scraping google
in thread screen scraping google

That part is primitive and based on the user-agent string that LWP sends in its headers identifying iteself as lwp/some_version. You will get the expected response if you masquarade as IE but of course that is immoral, possibly illegal but as far as I know not fattening.

use LWP::UserAgent; use Data::Dumper; my $ua = LWP::UserAgent->new; $ua->agent('Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; compaq) +'); my $request = HTTP::Request->new( 'GET', 'http://www.google.com/search +?q=google+api' ); my $response = $ua->request( $request ); print Dumper $response;

If you are interested in the throttling see Throttling Apache or mod_throttle or Google for terms like 'throttle bandwidth shaping'

cheers

tachyon