in reply to automated search using perl

There is a small trick with Google search, you don’t need to submit form if you want to search any keyword.
Use a URL like: "http://www.google.com/search?q=<keyword>"
For example http://www.google.com/search?q=perl
One thing to remember here is to encode the keyword.
i.e use http://www.google.com/search?q=perl%20monk
And not http://www.google.com/search?q=perl monk

I have never used module WWW::Google::API so not sure about it. Though if it exists and can ease the extraction of links than better use it.

Regards,
Ashish

Replies are listed 'Best First'.
Re^2: automated search using perl
by Anonymous Monk on Jul 21, 2009 at 02:29 UTC
    The trick you mention using the url "http://www.google.com/search?q=perl" (and various extensions) works just fine from Firefox or IE browsers.

    HOWEVER, this fails when I call the same url using perl as in: 

    my $ua = new LWP::UserAgent;
my $req = HTTP::Request->new(GET => $url);
my $res = $ua->request($req);

    [download]
    Instead google returns a long error message of which I have exercepted the following key snippet:

    Your client does not have permission to get URL xxxx from this server. (Client IP address: xxx.xxx.xxx.xxx). Please see Google's Terms of Service posted at http://www.google.com/terms_of_service.html

    I suppose that Google recognizes that I am not using a standard browser and assumes (perhaps correctly) that I must be violating their TOS. Is there any way to get around this? i.e. can I make my perl code look like a valid browser? I would prefer a simple method that doesn't require in-depth knowledge of (or access to) Google APIs or other specialized perl modules.

[reply]
[d/l]
      Google (or any server) can detect the client from 'user-agent' header. Default value for LWP::UserAgent is 'libwww-perl/#.##'(refer LWP::UserAgent) It can be changed by setting agent attribute to any valid user-agent string. Here is one example: 
      my $ua = new LWP::UserAgent;
$ua->agent('Mozilla/5.0');
#$ua->agent('Checkbot/0.4 ');
#$ua->agent('Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1');
#$ua->agent('FireFox 2');
my $req = HTTP::Request->new(GET => $url);
my $res = $ua->request($req);
if ($res->is_success) {
    print $res->content;  # or whatever
}
else {
    die $res->status_line;
}

      [download]
      Regards,
      Ashish
[reply]
[d/l]
Re^2: automated search using perl
by ashish.kvarma (Monk) on Jul 14, 2009 at 09:10 UTC
    In case you are looking for the "Advance search" just do a random search and have a look at the URL in the browser. I believe it will be self explanatory.
    Regards, Ashish
[reply]

In Section Seekers of Perl Wisdom