WWW::Mechanize::Firefox not filling out form

Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

I'm having trouble getting WWW::Mechanize::Firefox to submit data to a site that uses JS on the return page that prevents WWW::Mechanize from being useful. Below is the code I'm trying to use for WWW::Mechanize::Firefox. The browser loads the initial page, but the form is never filled out before the submit button is clicked. This is on a Windows XP system with the latest version of Firefox, Strawberry PERL and MinGW.

use strict;
use WWW::Mechanize::Firefox;

my $url   = "http://tools.immuneepitope.org/tools/bcell/iedb_input";
my $agent = WWW::Mechanize::Firefox->new(
    activate  => 1,
    autoclose => 0,
);

my $method = 49;
my $seq = "ARRRSFASDATRASDFSDARASDAGADFGASDRFREWFASCDSAGAREW";

$agent->get($url);
if ( $agent->success() ) {
    sleep(5);
    print "Retrieved $url\n";
    $agent->form_name('form1');
    $agent->set_fields( 'sequence' => $seq,'method' => $method);
    sleep(5); 
    $agent->submit();
}
[download]

Comment on WWW::Mechanize::Firefox not filling out form Download Code

Replies are listed 'Best First'.
Re: WWW::Mechanize::Firefox not filling out form by Corion (Patriarch) on Dec 08, 2010 at 08:45 UTC
This is a deficiency and maybe even a bug in `->set_fields()`. I think `->set_fields` needs more tests, as `<input type="radio">` fields are not really supported by `->field` or `->select`. As a workaround, the following script works for me. It uses `->field()` to set the sequence and explicitly `->click`s on the radio input, because that's what the Javascript code expects to happen. use strict; use WWW::Mechanize::Firefox; my $url = "http://tools.immuneepitope.org/tools/bcell/iedb_input"; my $agent = WWW::Mechanize::Firefox->new( activate => 1, autoclose => 0, ); my $method = 49; my $seq = "ARRRSFASDATRASDFSDARASDAGADFGASDRFREWFASCDSAGAREW"; $agent->get($url); if ( $agent->success() ) { print "Retrieved $url\n"; $agent->form_name('form1'); #$agent->set_fields( 'sequence' => $seq ); $agent->field( 'sequence' => $seq ); my $q = sprintf '//input[@name="method" and @value="%s"]', $method +; my $method_field = $agent->xpath( $q, single => 1 ); $agent->click( $method_field, synchronize => 0 ); #$agent->submit(); } [download] Update: Confirmed: `->set_fields()` has a bug in it and never worked at all. The next release (0.41) will have tests and likely a somewhat more working version of `->set_fields` but you should do without it for the time being. Thanks for providing such a convenient, almost self-contained example that replicates the problem.	[reply] [d/l] [select]
Re^2: WWW::Mechanize::Firefox not filling out form by Anonymous Monk on Dec 10, 2010 at 16:22 UTC
Thanks for this. The form now shows data being entered into it, but then Firefox (32-bit Windows version 3.6.12) starts eating up an entire CPU core on my computer and never moves on from clicking the radio button in the script. I can tell that it is the interaction of this script with Firefox since if I break (^C) the script, Firefox immediately drops in CPU usage.	[reply]
Re^3: WWW::Mechanize::Firefox not filling out form by Corion (Patriarch) on Dec 10, 2010 at 18:59 UTC
I can't reproduce your problem. My script, as posted above, does not do that, in fact, it never submits because that's commented out. To help me help you better, you could post a minimal version of your script that still reproduces the problem but is no longer than 20 lines. That would enable me, again, to reproduce your problem and maybe find a diagnosis that helps us determine the best approach to a solution.	[reply]
Re^4: WWW::Mechanize::Firefox not filling out form by Anonymous Monk on Dec 10, 2010 at 19:07 UTC
Re^5: WWW::Mechanize::Firefox not filling out form by Corion (Patriarch) on Dec 10, 2010 at 19:44 UTC
Re: WWW::Mechanize::Firefox not filling out form by 7stud (Deacon) on Dec 08, 2010 at 03:10 UTC
I'm having trouble getting WWW::Mechanize::Firefox to submit data to a site that uses JS on the return page that prevents WWW::Mechanize from being useful. That's the way it goes. Mechanize cannot predict what js will do. If a website author does not want robots to ineract with his website, he can make it almost impossible to do so. An example: unbeknownst to you a webpage's js adds "_abc" to the end of all form fields and then a program on the server checks for that suffix when examining the data. You can use Mechanize to fill out the form data, but unless you examine the js and discover that the js adds "_abc" to the end of each field, the data that you submit with Mechanize will be faulty.	[reply]
Re^2: WWW::Mechanize::Firefox not filling out form by ikegami (Patriarch) on Dec 08, 2010 at 03:25 UTC
He knows. That's why he's not using Mechanize.	[reply]