drblove27 has asked for the wisdom of the Perl Monks concerning the following question:

So I have been trying to learn LWP Post, by going through a examples on the web. Except it seems like every example on the web, the page no longer exists (well specifically the page that I am supposed to post to)... I am hoping someone could tell me what I am doing wrong as I try to modify this example for Perl & LWP.

I am attempting to search AbeBooks for the book "Codex Seraphinianus" from the main page (AbeBooks). Here is the code that I am using: 

#!/usr/bin/perl -w
use strict; 
use LWP;

my $browser = LWP::UserAgent->new( agent => 'Mozilla/5.0 (Windows; U; 
+Windows NT 6.0; en-US; rv:1.9) Gecko/2008052906 Firefox/3.0' );
$browser->cookie_jar({});
my $response = $browser->post(
  'http://www.abebooks.com/servlet/SearchEntry',
   # That's the URL that the real form submits to.
  [
    "bi" => "0",
    "bx" => "off",
    "ds" => "30",
    "sortby" => "2",
    "sts" => "t",
    "tn" => "Codex+Seraphinianus",
    "x" => "43",
    "y" => "10",
  ]
);
print $response->content;

[download]
But this just sends me back the content of the advanced search page not the results page. I guess I really have two questions,

1) Is there a way to check the http header that goes to the request? I want to check the header versus when I do it in the Firefox browser (with the idea that will help me debug my code in the future)

2) How do I fix my code to get back to the proper results page?

Updated For what it is worth, there is JavaScript on the page, so that is a fun kick in the pants...

Cheers

Replies are listed 'Best First'.
Re: LWP Post Question
by marcussen (Pilgrim) on Jul 17, 2008 at 01:36 UTC
    Although my headers look slightly different to yours (no bi, bx, ds, etc) I found that a GET request does the trick. Here is my tamper data transcript: 
    1:24:34.883[278ms][total 278ms] Status: 302[Moved Temporarily]
POST http://www.abebooks.com/servlet/SearchResults Load Flags[LOAD_DOC
+UMENT_URI  LOAD_INITIAL_DOCUMENT_URI  ] Content Size[20] Mime Type[te
+xt/plain]
   Request Headers:
      Host[www.abebooks.com]
      User-Agent[Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9) Gec
+ko/2008061017 Firefox/3.0]
      Accept[text/html,application/xhtml+xml,application/xml;q=0.9,*/*
+;q=0.8]
      Accept-Language[en-us,en;q=0.5]
      Accept-Encoding[gzip,deflate]
      Accept-Charset[ISO-8859-1,utf-8;q=0.7,*;q=0.7]
      Keep-Alive[300]
      Connection[keep-alive]
      Referer[http://www.abebooks.com/]
      Cookie[visited=false; H9U4=2-2-4AF2B353F25D1563A7EFF64C9C8C966F1
+CC47B51D9B4FC20BF7EC22C1159C36E-62EFF0DD7924B97457302616162F154647325
+FF38A3CBE949CD1AE8180BAE484; H9U4SE=1; abe_vc=1]
   Post Data:
      sts[t]
      tn[Codex+Seraphinianus]
      x[53]
      y[15]
   Response Headers:
      Location[http://www.abebooks.com/servlet/SearchResults?sts=t&tn=
+Codex+Seraphinianus&x=53&y=15]
      Content-Type[text/plain; charset=ISO-8859-1]
      Content-Encoding[gzip]
      Content-Length[20]
      Expires[Thu, 17 Jul 2008 01:24:29 GMT]
      Cache-Control[max-age=0, no-cache, no-store]
      Pragma[no-cache]
      Date[Thu, 17 Jul 2008 01:24:29 GMT]
      Vary[Accept-Encoding]
      Connection[keep-alive]

    [download]

    Notice the 302 response?

    So I modified the url to look for some made up book title; http://www.abebooks.com/servlet/SearchResults?sts=t&tn=Story+of+the+sea+urchin&x=53&y=15 And sent a GET request... voila!

    Confucius says kill mosquito unless cannon
[reply]
[d/l]
      Do you suppose that there is a way to do this with post? (Not to be picky, but trying to learn the madness of both methods)... But this does actually beg the question, if the form on the web page is a post, why is get working here?
[reply]
        The form is a POST, but the site sends back a 302, and then the browser sends a GET. Notice the contents of the browser URL box after the request...it has all the parameters in the query string of the URL. Why it does it that way, I dunno. But the answer is to just send a GET from LWP in the first place.
[reply]
Re: LWP Post Question
by runrig (Abbot) on Jul 17, 2008 at 00:06 UTC
    1) Get the Live HTTP Headers plugin for Firefox.
[reply]
      I am using Tamper Data plugin for Firefox, and I want to compare the headers that I have for when I do it from the browser versus what the code gets (thinking that the problem with the code is the incorrect header), but I don't know how to get the header from the code, i.e. the post command. Thanks for answering!
[reply]
Re: LWP Post Question
by Anonymous Monk on Jul 17, 2008 at 03:25 UTC
    LWP comes with lwp-request program, very useful 
    > lwp-request -USexd -m get http://cpan.org/
LWP::UserAgent::new: ()
LWP::UserAgent::request: ()
LWP::UserAgent::send_request: GET http://cpan.org/
LWP::UserAgent::_need_proxy: Not proxied
LWP::Protocol::http::request: ()
LWP::Protocol::collect: read 782 bytes
LWP::Protocol::collect: read 1880 bytes
LWP::Protocol::collect: read 1256 bytes
LWP::UserAgent::request: Simple response: OK
GET http://cpan.org/
User-Agent: lwp-request/5.810

GET http://cpan.org/ --> 200 OK
Connection: close
Date: Thu, 17 Jul 2008 03:20:43 GMT
Accept-Ranges: bytes
ETag: "a9dc69-f4e-487d100c"
Server: Apache/1.3.37
Content-Length: 3918
Content-Type: text/html
Content-Type: text/html
Last-Modified: Tue, 15 Jul 2008 21:01:00 GMT
Client-Date: Thu, 17 Jul 2008 03:24:58 GMT
Client-Peer: 66.39.76.93:80
Client-Response-Num: 1
Link: <mailto:cpan@perl.org>; rev="made"
Title: CPAN

    [download]
    You can see request headers (option -U). If you examine lwp-request source, you'll see 
        if ($options{'u'} || $options{'U'}) {
        my $url = $response->request->url->as_string;
        print "$method $url\n";
        print $response->request->headers_as_string, "\n" if $options{
+'U'};
    }

    [download]
    More details in HTTP::Response documentation.
[reply]
[d/l]
[select]
Re: LWP Post Question
by Gary Yang (Acolyte) on Jun 14, 2011 at 00:19 UTC

    Anyone has the working Perl code for this topic? I still do not get it.

[reply]

Back to Seekers of Perl Wisdom