dneedles has asked for the wisdom of the Perl Monks concerning the following question:

I am likely missing something very basic. The first program works fine. It reads the page, detects whether a test is started or not and either starts or stops the test. The second program correctly reads the pages and detects whether a test is started or not, but the form doesn't seem to work and the values do not get updated. Any ideas?

First Program

#!/usr/bin/perl
use WWW::Mechanize; ## Automate Grabbing Webpages
use strict;

my $url = "http://10.10.10.10/bat.html";
my $mech = WWW::Mechanize->new(autocheck => 1);
$mech->default_headers->authorization_basic("user", "pass");
$mech->get($url);
my $page = $mech->content;

## INITIAL OUTPUT
print "INITIAL REQUEST\n";
$mech->dump_all(undef, 1);
print $page;
print "\n\n\n";

my $resp2;

## STATE: TEST (CANNOT START OR CHECK, BUT CAN STOP)
if ($page=~/reset.*?disabled.*?Reset/) {
  print "TEST STOP\n";
  $resp2=$mech->submit_form(
     form_number => 1,
     fields      =>{ a => 2,
                     b => '3,2,2',
                   },
  );
## STATE: NO TEST (CAN START, CHECK, BUT NOT STOP)
} else {
  print "TEST START\n";
  $resp2=$mech->submit_form(
     form_number => 1,
     fields      => { c => 49.0,
                      d => 0.3,
                      e => 45.0 ,
                      a => 1,
                      b => '3,2,2',
                    },
  );
}

## AFTER FORM FILL OUT
$mech->dump_all(undef, 1);
print $resp2->content;
print "\n\n\n";

[download]
Second program

#!/usr/bin/perl
use HTTP::Request::Common qw(POST);
use LWP::UserAgent;
use strict;

my $ua = LWP::UserAgent->new();
#$ua->credentials(
#    '10.10.10.10:80',
#    'PSC500',
#    'user' => 'pass'
#);

# log in to authentication page.
my $req = HTTP::Request->new(GET => 'http://10.10.10.10/bat.html');
$req->authorization_basic('user', 'pass');

## INITIAL OUTPUT
print "INITIAL REQUEST\n";
my $resp = $ua->request($req);
my $content = $resp->as_string;
print $content;
print "\n\n\n";


my $req2;
## STATE: TEST (CANNOT START OR CHECK, BUT CAN STOP)
if ($content=~/reset.*?disabled.*?Reset/) {
  print "TEST STOP\n";
  $req2= POST 'http://10.10.10.10/bat.html',
               [ a => 2, b => '3,2,2', ];
## STATE: NO TEST (CAN START, CHECK, BUT NOT STOP)
} else {
  print "TEST START\n";
  $req2= POST 'http://10.10.10.10/bat.html',
               [ c => 49.0, d => 0.3,
                 e => 45.0 , a => 1,
                 b => '3,2,2', ];
}
$req2->authorization_basic('user', 'pass');

my $resp2 = $ua->request($req2);
my $content = $resp->as_string;
print $content;
print "\n\n\n";

[download]
I also tried the second program with a number of variations: 
$req2= HTTP::Request->new(POST => 'http://10.10.10.10...
Used ua->credentials(...
etc.

[download]
I got the same result where the page was read but the form values did not post.

Any ideas?

Replies are listed 'Best First'.
Re: A case where Mechanize works, LWP doesn't
by ahmad (Hermit) on Feb 09, 2010 at 06:09 UTC

    With mechanize you are selecting the form you want to post, while in your http request you'll have to do it in your own

    You'll have to take a look at the form you're submitting and make sure that it does submit the form to 'bat.html' which is unlikely to be the case.

    Look for <form method='post' action='HERE's the action url you'll have to submit your data to'>

[reply]
[d/l]
      Thanks, but this isn't the case. There is only one form which is being submitted to the correct page: 

      function BATSubmit()
{
   if (checkfields()==false)
      return false;
    handle=EventHandlingNum(document.getElementById('c').value,documen
+t.getElementById('d').value,document.getElementById('e').value,1);
    document.batForm.a=handle;
}

<form name="batForm" ACTION="/bat.html" METHOD="POST" onSubmit="return
+ BATSubmit()">

      [download]
[reply]
[d/l]
Re: A case where Mechanize works, LWP doesn't
by ikegami (Patriarch) on Feb 09, 2010 at 07:24 UTC
    Compare the requests used. The request that produced a response is available via one of the methodsthe request method of the response.
[reply]
[d/l]
      I don't understand what you are saying. Can you be more specific perhaps refer to the example or provide one of your own?
[reply]
        How does the output of 
        print $resp2->request->as_string;

        [download]
        differ?
[reply]
[d/l]
Re: A case where Mechanize works, LWP doesn't
by Gangabass (Vicar) on Feb 09, 2010 at 06:35 UTC

    Cookies, hidden form inputs and as ahmad said check form action attribute.

[reply]
      Again, Mechanized works, the other doesn't so if there was cookies or other nuances it would affect both. What does Mechanize do that LWP doesn't?

      The form idea was good, but didn't pan out. Or how did I alter the approach from one to the other?

[reply]

        While LWP has cookie support, it is not enabled by default. While LWP has support for sending the Referer header, it does not do so by default. WWW::Mechanize basically provides defaults as a web browser would have them to LWP.

        As was mentioned below, get a network sniffer and compare the stream that your LWP implementation sends agasint what your WWW::Mechanize implementation sends. The differences are likely what makes your LWP implementation fail.

[reply]
[d/l]
Re: A case where Mechanize works, LWP doesn't
by planetscape (Chancellor) on Feb 09, 2010 at 18:46 UTC

    Use something like Wireshark or Live HTTP Headers to find out what is really going on, not what you think is going on.

    HTH,

    planetscape
[reply]
      Sigh. I figured I made a typo or forgot a call. Yep loaded wireshark. I guess LWP isn't as straight forward as I thought. Is LWP normally this difficult or is this just a bad one-off. I cannot imagine debugging a large program via sniffer traces.
[reply]
        WWW::Mechanize uses LWP, so your question makes no sense.
[reply]
      Ok. Interesting results on the Sniffer trace. It appears Mechanize is handling a redirect while LWP is not. Also the encapsulation of the POST on Mechanize is application/x-www-form-urlencoded while LWP just has variable: value in separate lines. So it appears Mechanize is doing much more work under the covers. I'll see how I can manipulate LWP to mimic Mechanize.
[reply]

        Why don't you just use WWW::Mechanize? It's based on LWP, so basically to make LWP::UserAgent behave like WWW::Mechanize means just taking the code of WWW::Mechanize.

[reply]

Back to Seekers of Perl Wisdom