ASPX Form Post

LeoCoder has asked for the wisdom of the Perl Monks concerning the following question:

Hello Monks - I am trying to retrieve the table data from this page http://bogc.dnrc.mt.gov/WebApps/DataMiner/Wells/WellCompletions.aspx using mechanize. I am not able to get the result page after post. I am passing all parameters with the form. But it is not working. Please assists me to solve this invisible bug. :)

use strict;
use warnings;
use LWP::Simple;
use WWW::Mechanize;

#Access to the root page
    my $agent;
    $agent = WWW::Mechanize->new();
    my $rootpage = "http://bogc.dnrc.mt.gov/WebApps/DataMiner/Wells/We
+llCompletions.aspx";

#Get the root page
    $agent->get($rootpage);

#Get Form data
    $agent->form_name('aspnetForm');
    my $eventtarget = $agent->field('__EVENTTARGET');
    my $eventarg = $agent->field('__EVENTARGUMENT');
    my $lastfocus = $agent->field('__LASTFOCUS');
    my $viewstate = $agent->field('__VIEWSTATE');
    my $viewstategen = $agent->field('__VIEWSTATEGENERATOR');
    my $eventValidation = $agent->field('__EVENTVALIDATION');
    
#Construct fields 
    my $fields =  {
        'ctl00$ScriptManager1' => 'ctl00$SiteContentPlaceHolder$FormFi
+lterNoNav1$UpdatePanelFormFilterNoNav|ctl00$SiteContentPlaceHolder$Fo
+rmFilterNoNav1$btnSearch',
        '__EVENTTARGET' => $eventtarget,
        '__EVENTARGUMENT' => $eventarg,
        '__LASTFOCUS' => $lastfocus,
        '__VIEWSTATE' => $viewstate,
        '__VIEWSTATEGENERATOR' => $viewstategen,
        '__EVENTVALIDATION' => $eventValidation,
        'ctl00$SiteContentPlaceHolder$FormFilterNoNav1$cboFilterOption
+' => 'Dt_Effect|date',
        'ctl00$SiteContentPlaceHolder$FormFilterNoNav1$cboFilterType' 
+=> '<',
        'ctl00$SiteContentPlaceHolder$FormFilterNoNav1$txtFilter' => '
+09/06/2016',
        'ctl00$SiteContentPlaceHolder$FormFilterNoNav1$txtFilterString
+' => '',
        'ctl00$SiteContentPlaceHolder$FormFilterNoNav1$btnSearch' => '
+Search'
    };
#Submit request to get data.
    $result = $agent->submit_form(
                form_name => 'aspnetForm',
                fields => $fields,
                );
    $agent->save_content("C:/" . "Completions.html");
[download]

Comment on ASPX Form Post Download Code

Replies are listed 'Best First'.
Re: ASPX Form Post by choroba (Cardinal) on Sep 08, 2016 at 14:46 UTC
Is JavaScript involved in the submission? WWW::Mechanize doesn't support it, as documented. ($q=q:Sq=~/;[c](.)(.)/;chr(-\|\|-\|5+lengthSq)`"S\|oS2"`map{chr \|+ord }map{substrSq`S_+\|`\|}3E\|-\|`7**2-3:)=~y+S\|`+$1,++print+eval$q,q,a, [download]	[reply] [d/l]
Re^2: ASPX Form Post by LeoCoder (Initiate) on Sep 08, 2016 at 14:52 UTC
Thanks. I am not sure about java script. Is there any other package that replaces mechanize and have all feature for handling web request. Additional information: java script are used to fill the form parameter during page load. I am getting those parameter results and passing them again while posting.	[reply]
Re^3: ASPX Form Post by Corion (Patriarch) on Sep 08, 2016 at 18:42 UTC
If the page needs Javascript, you will need to find a version of WWW::Mechanize that supports Javascript. See for example Win32::IEAutomation, WWW::Mechanize::Firefox or WWW::Mechanize::PhantomJs. Alternatively, find out what the Javascript does and do the same with Perl instead of Javascript.	[reply]
Re^4: ASPX Form Post by LeoCoder (Initiate) on Sep 08, 2016 at 20:26 UTC
Re^4: ASPX Form Post by LeoCoder (Initiate) on Sep 08, 2016 at 20:43 UTC
Re: ASPX Form Post by hippo (Archbishop) on Sep 08, 2016 at 18:33 UTC
But it is not working. In which case the problem is probably somewhere between "your computer isn't switched on" and "your ISP is filtering your traffic". Please assists me to solve this invisible bug. :) Please assist us to assist you by at least describing the symptoms in detail. Post the exact text of the error message received. Say which versions of perl and the 2 modules you are using (and maybe explain why you are using LWP::Simple when the rest of your code doesn't mention it). Say which O/S you are running. In essence, give some detail. Thanks.	[reply]
Re^2: ASPX Form Post by LeoCoder (Initiate) on Sep 08, 2016 at 20:55 UTC
Let me explain the problem. I am trying to extract the table data from that website. It requires current date in the textbox and click search. I used Firefox to inspect the network data that are passed to server. using that data, I constructed the parameters manually and submit the form. but, server is not throwing any error and result. I have shared the code here to get some thoughts from perlmonks about whether anything is missing in this code or what should i do to scrape ASPX webpage. I am new to Perl programming. My system details Windows, ActiveState Perl 4.14.	[reply]
Re^3: ASPX Form Post by hippo (Archbishop) on Sep 09, 2016 at 08:20 UTC
Thanks for elaborating. Unfortunately the only extra information here about what actually goes wrong is this bit: but, server is not throwing any error and result which isn't very clear because while it says what doesn't happen, it fails to mention what does actually happen. Does the script hang? Does it die? Does it silently finish? What did you do to try to find the root cause? Have you been through the Basic debugging checklist? My system details Windows, ActiveState Perl 4.14. That's an almost unfeasibly old version of perl and certainly the code which you have posted would not even compile on such an ancient version. Did you mean 5.14 instead?	[reply]
Re^4: ASPX Form Post by Anonymous Monk on Jun 21, 2018 at 18:01 UTC