Google Not Liking Repetitive GETs

DougMcq has asked for the wisdom of the Perl Monks concerning the following question:

I'm using WWW::Mechanize and having an intermittent problem when executing a series of Google searches. Currently less than 100 but wish to scale significantly. It's a state problem of some sort. If I wait a bit, the script will execute properly. The error message follows "Error GETing htIndexRedirect?tps://ipv4.google.com/sorry/continue...ECHOS-MY-REQUESTED-URL....Service Unavailable at script line nnn"

 #!/usr/bin/perl
 
     # create a new browser
    use WWW::Mechanize;
    my $mech = WWW::Mechanize->new();
    $mech->cookie_jar(HTTP::Cookies->new());
[download]

sub GETURL {
    $start_time = time;
    $output_buffer = "";
    print OUTDATA "Retrieving $_\n";
    if ($mech->get("$_")) {
        $_ = $mech->content();
        $time = time-$start_time;
        print OUTDATA "Seconds $time\n";
        while (s/\n\n/\n/) {}
        }
    else { print OUTDATA " ERROR\n";}
    sleep (1);
    }
[download]

Comment on Google Not Liking Repetitive GETs Select or Download Code

Replies are listed 'Best First'.
Re: Google Not Liking Repetitive GETs by Anonymous Monk on Mar 11, 2015 at 00:02 UTC
It's a state problem of some sort. If I wait a bit, the script will execute properly. The error message follows "Error GETing htIndexRedirect?tps://ipv4.google.com/sorry/continue...ECHOS-MY-REQUESTED-URL....Service Unavailable at script line nnn" Its like when you walk up to a bartender, order a drink, and he responds with "I'll take your order in a minute", there is nothing for you to do but wait on bartender (google) `$mech->cookie_jar(HTTP::Cookies->new());` $mech already did this for you, its $mech not LWP	[reply] [d/l]
Re^2: Google Not Liking Repetitive GETs by DougMcq (Initiate) on Mar 28, 2015 at 01:59 UTC
IMHO there must be some other factor at play. I slowed this down to one GET request per minute. Hard to believe with the infrastructure Google has they can't even keep up with one glacial-paced user ;-)	[reply]
Re^3: Google Not Liking Repetitive GETs by choroba (Cardinal) on Mar 28, 2015 at 02:07 UTC
The problem is not the speed of a user, but the number of users. That's why they limit the number of requests per time. لսႽ† ᥲᥒ⚪⟊Ⴙᘓᖇ Ꮅᘓᖇ⎱ Ⴙᥲ𝇋ƙᘓᖇ	[reply]
Re^3: Google Not Liking Repetitive GETs by Anonymous Monk on Mar 28, 2015 at 08:37 UTC
IMHO there must be some other factor at play You are free to believe anything you want :) Google is teling you slow your roll ... you can always try the support forum	[reply]