Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

Hello again, What I thought would be easy is truning out hard. I have read the perldoc for lwp, but I'm confused on several issues. I have the following code as a start: Is it possible to submit text and automate the search button from googles site? 
use LWP::UserAgent;
$ua = LWP::UserAgent->new;

my $req = HTTP::Request->new(POST => 'http://www.google.com/');
$req->content_type('application/x-www-form-urlencoded');
$req->content('match=?');

my $res = $ua->request($req);
print $res->as_string;

[download]
I guess I don't understand the matching criteria, and the idea of usinf POST to submit a form Thank You

Replies are listed 'Best First'.
•Re: LWP post
by merlyn (Sage) on Jun 18, 2002 at 15:11 UTC
    See perldoc lwpcook, which reads in part: 
        Lazy people use the HTTP::Request::Common module to set up a suita
+ble
    POST request message (it handles all the escaping issues) and has 
+a
    suitable default for the content_type:

      use HTTP::Request::Common qw(POST);
      use LWP::UserAgent;
      $ua = LWP::UserAgent->new;

      my $req = POST 'http://www.perl.com/cgi-bin/BugGlimpse',
                    [ search => 'www', errors => 0 ];

      print $ua->request($req)->as_string;

    [download]

    -- Randal L. Schwartz, Perl hacker

[reply]
[d/l]
Re: LWP post
by smitz (Chaplain) on Jun 18, 2002 at 15:46 UTC
    Bit OT, but you should probably be aware that Google dont like this sort of thing. Many /WWW::Google/ type module authors have had requests for their code to be removed.
    If you want to do google searches from within your code, consider taking a look at Google API.

    SMiTZ
[reply]
      Yes, this is true. Google does not like these things. However I do have a small script that uses GET method that could be of some use to the person who asked the question. Use it only for learning and as smitz said, have a look at Google API. 
      use URI;
use LWP::UserAgent;

my $what_i_want_to_find = 'perl monks';

$ua = LWP::UserAgent->new;
# google doesn't like it when user agent is libwww perl, so
# I change it to Mozilla/5.0
$ua->agent('Mozilla/5.0');

my $uri = URI::new;
$uri->query_form('q'=>$what_i_want_to_find);
my $url = 'http://www.google.com/search' . $uri->as_string;
my $req = HTTP::Request->new(GET => $url);
my $res = $ua->request($req)->as_string;
$res =~ /\<p\>\<a href\=([^>]+)\>/;
print $1;

      [download]
      I hope this was useful.
[reply]
[d/l]
        Thanks moxliukas, I now understand the Get method, Do you know if the Post method works in a similar fashion when trying to automate the submit button on a form?
[reply]
      I'd like to add that google is likely to use some defense mechanisms against automatic queries via HTTP. WWW::Automate tests itself against http://google.com/ page and I get lots of failures described as "Connection reset by peer" when running several test sessions in sequence.
[reply]

Back to Seekers of Perl Wisdom