Gary Yang has asked for the wisdom of the Perl Monks concerning the following question:

I am learning LWP Post, by going through an examples on the web. Perl & LWP. I try to search AbeBooks for the book "Codex Seraphinianus" from the main page (AbeBooks). Here is the code that I am using: 

#!/usr/bin/perl -w

use strict;
use LWP

my $cookie_jar = HTTP::Cookies->new(file => "lwp_cookies.dat", autosav
+e => 1);

my $browser = LWP::UserAgent->new;

$browser->cookie_jar($cookie_jar);


my $response = $browser->get('http://www.abebooks.com',
   ["sts" => "t",
    "an" => "",
    "tn" => "Codex Seraphinianus",
    "kn" => "",
    "isbn" => ""
   ]
);

die "Error: ", $response->status_line, "\n" unless $response->is_succe
+ss;

my $out_file = "result_seraph.html"; # where to save it
open(OUT, ">$out_file") || die "Can't write-open $out_file: $!";

binmode(OUT);
print OUT $response->content;
close(OUT);

[download]

But this just sends me back the content of the search page not the result page. Anyone knows what the problems are? Anyone can show me the working code?

Replies are listed 'Best First'.
Re: LWP Post does not work.
by runrig (Abbot) on Jun 14, 2011 at 00:24 UTC
    Read the source of the page again, and check the url that the form submits to.
[reply]
      Eh, Gary
      Take a look again at what runrig said, the action on that search form appears to be"/servlet/SearchResults" not "/" and if you modify your script to make the request to the right place it works (provided you add the semicolon after use LWP ;)

      print "Good ",qw(night morning afternoon evening)[(localtime)[2]/6]," fellow monks."
[reply]
[d/l]
[select]
Re: LWP Post does not work.
by ikegami (Patriarch) on Jun 14, 2011 at 00:23 UTC

    You said "Post" in your subject and you're using the arguments for post, but you actually called get. Did you mean to call post?

[reply]
[d/l]
[select]
Re: LWP Post does not work.
by 7stud (Deacon) on Jun 14, 2011 at 07:37 UTC

    But this just sends me back the content of the search page not the result page. Anyone knows what the problems are?

    You seem to have a fundamental misunderstanding about how forms work. Forms gather name/value pairs from the form input elements and send them to the url specified in the form tag's action attribute. The name/value pairs that you are interested in generating come from this form:

    <form action="/servlet/SearchResults" method="post" name="homepageSear
+ch">

    [download]

    How do I know that? I clicked on View Source for abebook's homepage. So you have to send the name value pairs to the url specified for the action attribute--which you should note is a relative url. Try the following url instead of http://www.abebooks.com:

    http://www.abebooks.com/servlet/SearchResults

    Note also that the form specifies 'post', so you need to generate a post request--not a get request.

[reply]
[d/l]
[select]
Re: LWP Post does not work.
by Anonymous Monk on Jun 14, 2011 at 01:00 UTC

    Someone suggested to use POST. I changed the code to post. It still does not work. Any clue? 

    #!/usr/bin/perl -w

use strict;
use LWP;

my $browser = LWP::UserAgent->new;

my $response = $browser->post('http://www.abebooks.com',
   ["sts" => "t", 
    "an" => "", 
    "tn" => "Codex Seraphinianus",
    "kn" => "",
    "isbn" => ""
   ]
);

die "Error: ", $response->status_line, "\n" unless $response->is_succe
+ss;

my $out_file = "result_seraph.html"; # where to save it
open(OUT, ">$out_file") || die "Can't write-open $out_file: $!";

binmode(OUT);
print OUT $response->content;
close(OUT);

    [download]
[reply]
[d/l]
      It still does not work. Any clue?

      Yeah, explain what "does not work" means :)

[reply]

Back to Seekers of Perl Wisdom