hhalpin has asked for the wisdom of the Perl Monks concerning the following question:

Fellow monks,

As explained in an earlier post, am trying to automate my usage of an on-line database using LWP. This is my first time using LWP, so if I say anything silly please have patience :)

I found the responses to my last post to be quite good, and now I have the following as code: 

use HTTP::Request::Common qw(POST);
use LWP::UserAgent;
$ua = new LWP::UserAgent;
my $req = POST 'http://lsa.colorado.edu/cgi-bin/LSA-sentence.html',
                    [ txt1 => 'I hate Jimmy. I love Jimmy.'];
$content = $ua->request($req)->as_string;
open(OUTPUT, ">>result.html");
print OUTPUT $content;

[download]

Which manages to retrieve the web-page LSA-sentence.html, but not the resulting web-page produced by the CGI ":http://lsa.colorado.edu/cgi-bin/LSA-sentence-x.html". I believe this is b/c the web-page features a "Submit" button am not pressing. I believe this is it in HTML from the web page: 

INPUT TYPE="submit"  VALUE="Submit Texts">
INPUT TYPE="reset"  VALUE=" Reset to Defaults ">

[download]
How do I press those buttons, if that is needed, or is there some other way to get my precious resulting web-page? I assume it's in my request "hash" - but how? Once I get it - I assume it will be in content - I would like just to print it to a file for later processing.

Thanks!

Harry

Replies are listed 'Best First'.
Re: Problem retrieving Web-page using LWP - help!
by chromatic (Archbishop) on Nov 30, 2000 at 22:40 UTC
    In your previous post, you show the HTML of the first page as containing the following:

    ACTION="http:/cgi-bin/database-#result.html"

    I suspect if you substitute this for the first parameter to your POST call above (making it into a complete URI, that is), you will have more success.

    The action attribute of a form tag tells the user agent (browser) where to connect to send the form data. I'm not sure (at the moment) how to send a submit value without a name, so I'm dismissing that line of reasoning. :)

[reply]
[d/l]
Re: Problem retrieving Web-page using LWP - help!
by swiftone (Curate) on Nov 30, 2000 at 22:59 UTC
    In addition to chromatic's comment, you'll also need to define the LSAspace variable that page wants. This script works for me: 
    #!/usr/bin/perl -w
use strict;
use HTTP::Request::Common qw(POST);
use LWP::UserAgent;

my $ua = new LWP::UserAgent;
my $req = POST 'http://lsa.colorado.edu/cgi-bin/LSA-sentence-x.html',
        [ txt1 => 'I hate Jimmy. I love Jimmy.', 
          LSAspace =>'General_Reading_up_to_1st_year_college'];
my $content = $ua->request($req)->as_string;

print $content;

    [download]
[reply]
[d/l]
Re: Problem retrieving Web-page using LWP - help!
by kilinrax (Deacon) on Nov 30, 2000 at 23:02 UTC
    Try this: 
    #!/usr/bin/perl -w
 
use strict;
use LWP::UserAgent;
use URI::URL;
 
my $ua = new LWP::UserAgent;
 
my $page = "http://lsa.colorado.edu/cgi-bin/LSA-sentence-x.html";
 
my $uri = new URI::URL;
$uri->query_form("LSAspace" => "General_Reading_up_to_1st_year_college
+ (300 factors)",
                 "LSAFactors" => "",
                 "txt1" => "I hate Jimmy. I love Jimmy.");
 
my $req = new HTTP::Request POST => $page;
$req->content_type('application/x-www-form-urlencoded');
$req->content($uri->equery);
 
my $res = $ua->request($req);
 
if (!$res->is_success) {
  die "Fetch failed\n";
}
 
my $content = $res->content;
print $content;

    [download]
[reply]
[d/l]
Re: Problem retrieving Web-page using LWP - help!
by chipmunk (Parson) on Nov 30, 2000 at 22:43 UTC
    I believe that the default name for a submit button is "submit". So, try adding submit => 'Submit Texts' to your list of parameters.
[reply]
Re: Problem retrieving Web-page using LWP - help!
by wardk (Deacon) on Nov 30, 2000 at 23:18 UTC
    Looks like your URL is an HTML page. Are you sure you are not wanting to invoke the actual script? You appear to be getting the form.
[reply]

Back to Seekers of Perl Wisdom