Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

Hi,

I am trying to scrape this page and seems i cant get the valid '__VIEWSTATE' or even '__EVENTVALIDATION' (fields), here's my code:

use strict; use warnings; use HTML::TreeBuilder; use WWW::Mechanize; use URI::Escape; my $agent = WWW::Mechanize->new(); $agent->get('http://www.usgbc.org/myUSGBC/Members/MembersDirectory.asp +x?PageID=2011&CMSPageID=140'); $agent->form_name('frmMemberDirectory'); $agent->field('ddlCategory', ''); $agent->field('ddlState', ''); $agent->field('ddlCountry', 'US'); $agent->field('txtCompanyName', ''); $agent->click('btnSearch'); my $node = HTML::TreeBuilder->new(); $node->parse( $agent->content ); my $val_id = $node->find_by_attribute( id => '__EVENTVALIDATION'); my $validation = $val_id->attr('value'); my $state_id = $node->find_by_attribute( id => '__EVENTVALIDATION'); my $view_state = $state_id->attr('value'); $validation = uri_escape( $validation ); $view_state = uri_escape( $view_state ); my $id = '__EVENTTARGET=dgMembers%24ctl01%24ctl11&__EVENTARGUMENT=&__V +IEWSTATE='. $view_state . '&txtCompanyName=&ddlCategory=&ddlState=&ddlCountry=US&__EVENT +VALIDATION='. $validation; $agent->post('http://www.usgbc.org/myUSGBC/Members/MembersDirectory.as +px?PageID=2011&CMSPageID=140', Content => $id); print $agent->content();

Someone from this page said that 'encode' will solve the problem, i just have no idea what he means with 'encode'.

thanks

Replies are listed 'Best First'.
Re: scraping ASP page, __VIEWSTATE problem
by Anonymous Monk on Aug 07, 2009 at 18:57 UTC

    Update, here's the code

    #!/usr/bin/perl use strict; use warnings; use HTML::TreeBuilder; use WWW::Mechanize; use URI::Escape; my $agent = WWW::Mechanize->new(); $agent->get('http://www.usgbc.org/myUSGBC/Members/MembersDirectory.asp +x?PageID=2011&CMSPageID=140'); $agent->form_name('frmMemberDirectory'); $agent->field('ddlCategory', ''); $agent->field('ddlState', ''); $agent->field('ddlCountry', 'US'); $agent->field('txtCompanyName', ''); $agent->click('btnSearch'); my $node = HTML::TreeBuilder->new(); $node->parse( $agent->content ); my $val_id = $node->find_by_attribute( id => '__VIEWSTATE'); my $validation = $val_id->attr('value'); my $state_id = $node->find_by_attribute( id => '__EVENTVALIDATION'); my $view_state = $state_id->attr('value'); $validation = uri_escape( $validation ); $view_state = uri_escape( $view_state ); my $id = '__EVENTTARGET=dgMembers%24ctl01%24ctl11&__EVENTARGUMENT=&__V +IEWSTATE='. $view_state . '&txtCompanyName=&ddlCategory=&ddlState=&ddlCountry=US&__EVENT +VALIDATION='. $validation; $agent->post('http://www.usgbc.org/myUSGBC/Members/MembersDirectory.as +px?PageID=2011&CMSPageID=140', Content => $id); print $agent->content();
Re: scraping ASP page, __VIEWSTATE problem
by Anonymous Monk on Aug 08, 2009 at 08:37 UTC

    problem solved

    Change dgMembers%24ctl01%24ctl11 to dgMembers$ctl01$ctl10 and i'll get pagination 2, (for next paginations change to dgMembers$ctl01$ctl11)

    my $id = '__EVENTTARGET=dgMembers$ctl01$ctl10&__EVENTARGUMENT=&__VIEWS +TATE='. $view_state . '&txtCompanyName=&ddlCategory=&ddlState=&ddlCountry=US&__EVENT +VALIDATION='. $validation;

      You was right. But you still don't need to use POST. This is work too:

      #!/usr/bin/perl use strict; use warnings; use WWW::Mechanize; my $agent = WWW::Mechanize->new(); $agent->get('http://www.usgbc.org/myUSGBC/Members/MembersDirectory.asp +x?PageID=2011&CMSPageID=140'); $agent->form_name('frmMemberDirectory'); $agent->field('ddlCountry', 'US'); $agent->click('btnSearch'); $agent->submit_form( form_name => "frmMemberDirectory", fields => { '__EVENTTARGET' => 'dgMembers$ctl01$ctl01', '__EVENTARGUMENT' => "", }, ); open my $test, ">", "test.htm" or die $!; print $test $agent->content(); close $test;
Re: scraping ASP page, __VIEWSTATE problem
by Anonymous Monk on Aug 08, 2009 at 03:31 UTC
    If __VIEWSTATE and __EVENTVALIDATION are form fields, get rid of all your HTML::Tree nonsense and read WWW::Mechanize
Re: scraping ASP page, __VIEWSTATE problem
by Anonymous Monk on Aug 08, 2009 at 03:32 UTC
    Someone from this page said that 'encode' will solve the problem, i just have no idea what he means with 'encode'.

    encode isn't mentioned on that page at all.

A reply falls below the community's threshold of quality. You may see it by logging in.