papaii has asked for the wisdom of the Perl Monks concerning the following question:

while running the code i am recieving the below mentioned error message

C:\Users\kericson\Documents\DataFiles\MA_2012Plus>perl fetch_quotes.pl 09/2012 Fetch data for "01001"_30_na_1.html Found Session Id = DC02F201A46E46360672E5279A5D96B6 Can't call method "header" on an undefined value at C:/Perl64/lib/WWW/Mechanize.pm line 2467.

I checked the webpage code and found the continue button is present outside the <form></form> section.

The link is https://www.mahealthconnector.org/portal/site/connector/menuitem.55b6e23ac6627f40dbef6f47d7468a0c/

The code of webpage is 

      
      <div style="display: none;" class="frame" id="frame3">
        <div class="frameContent">
            <h3>Is your household income more than <span id="yearly_po
+verty_level">0</span> a year?</h3>
            <p><strong>YES:</strong> Click Continue to compare plans f
+rom Commonwealth Choice.<br>

            <strong>NO:</strong> Learn more about <a href="/portal/sit
+e/connector/template.MAXIMIZE/menuitem.3ef8fb03b7fa1ae4a7ca7738e6468a
+0c/?javax.portlet.tpst=2fdfb140904d489c8781176033468a0c_ws_MX&javax.p
+ortlet.prp_2fdfb140904d489c8781176033468a0c_viewID=content&javax.port
+let.prp_2fdfb140904d489c8781176033468a0c_docName=CommCareOverview&jav
+ax.portlet.prp_2fdfb140904d489c8781176033468a0c_folderPath=/About Us/
+CommonwealthCare/">subsidized plans from Commonwealth Care</a>.</p>  
+          
        </div><!-- end frameContent -->
        <div class="frameButton">
            <a id="backToFrame2Btn" href="/portal/site/connector/menui
+tem.879f6b190ed7ac06a7ca7738e6468a0c/" title="Back">Go Back</a>
            <a id="continueBtn" href="/portal/site/connector/template.
+MAXIMIZE/menuitem.55b6e23ac6627f40dbef6f47d7468a0c/?javax.portlet.beg
+CacheTok=com.vignette.cachetoken&javax.portlet.pst=1dd8f1b14739404575
+c4c2100ce08041_ws_MX&javax.portlet.prp_1dd8f1b14739404575c4c2100ce080
+41=connector_currentstep%3Dconnector-step1%26viewID%3Dsubmit_process&
+javax.portlet.endCacheTok=com.vignette.cachetoken">
                <img alt="Continue" src="/portal/images/connector_imag
+es/homepage2010/button-continue.png">

            </a>
        </div><!-- end frameButton -->
      </div><!-- end frame3 -->

[download]

Below is my perl code which is actually fetching data from the above mentioned website. 

sub fetchCoveragePlans

{

    my @subData = @_;



    my $familySize = $subData[3];  #  01001,42,42,2

    

    my $cookie_jar = HTTP::Cookies->new(

        file => 'cookies.dat',

        autosave => 1,

    );



    trace("Fetch data for " . getFileName(@subData));

    my $mech = WWW::Mechanize->new(cookie_jar => $cookie_jar, quiet =>
+ 0,);

    

    $mech->agent_alias( 'Windows IE 6' );

    

#    $mech->conn_cache(LWP::ConnCache->new);



    # Fetch - Find Insurance Page

    $mech->get($HOMEURL);

    

    my $sessionId;

    my $response = $mech->response();

    if ($mech->response->status_line != STATUS_OK)

    {

        trace("Error fetching Page 1. Status = ". $mech->response->sta
+tus_line .". Try after sometime.");

        return;

    }

    ##----------------------------------------------------------------
+-----------------------------------##

    for my $key ( $response->header_field_names() ) {

        if ($key =~ /Set-Cookie/is) {            

            $sessionId = $response->header( $key );

            if ($sessionId =~ /JSESSIONID=(.*?);/)

            {

                $sessionId = $1;

            }

        }

    }

    trace("Found Session Id = $sessionId");   

    ##----------------------------------------------------------------
+-----------------------------------##

    my $savePage = "$HOMEDIR/$currDir/".getFileName(@subData);

    my $html = $mech->content;

    ##----------------------------------------------------------------
+-----------------------------------##

    my $jsURL = "https://www.mahealthconnector.org/portal/dwr/call/pla
+incall/__System.pageLoaded.dwr";



    my $response = $mech->post($jsURL,

            [ 'callCount'        =>    "1",

              'windowName'        =>    "",

              'c0-scriptName'    =>    "__System",

              'c0-methodName'    =>    "pageLoaded",

              'c0-id'        =>    "0",

              'batchId'        =>    "0",

              'page'        =>    "/portal/site/connector/menuitem.55b
+6e23ac6627f40dbef6f47d7468a0c/",

              'httpSessionId'    =>  $sessionId,

              'scriptSessionId'    =>    "",

            ]);





    if ($mech->response->status_line != STATUS_OK)

    {

        trace("Error fetching Page 5a. Status = ". $mech->response->st
+atus_line .". Try after sometime.");

        return;

    }

    ##----------------------------------------------------------------
+-----------------------------------##

    # dwr.engine.remote.handleNewScriptSession("71A70B8E00AB93A125F4D9
+86E5D7B66D");

    my  $scriptSessionId;

    if ($html =~ /handleNewScriptSession\("(.*?)"/ )

    {

        $scriptSessionId = $1;

    }

    ##----------------------------------------------------------------
+-----------------------------------##

    my $jsURL4='https://www.mahealthconnector.org/portal/dwr/call/plai
+ncall/LandingDelegate.getPovertyLevelForFamilySize.dwr';

    my $response = $mech->post($jsURL4, 

            [ 'callCount'        =>    "1",

              'windowName'        =>    "",

              'c0-scriptName'    =>    "LandingDelegate",

              'c0-methodName'    =>    "getPovertyLevelForFamilySize",

              'c0-id'            =>    "0",

              'c0-param0'       =>  "number:2",

              'c0-param1'       =>  "string:INDIVIDUAL",

              'batchId'            =>    "1",

              'page'            =>  "/portal/site/connector/menuitem.5
+5b6e23ac6627f40dbef6f47d7468a0c/",

              'httpSessionId'    =>  $sessionId,

              'scriptSessionId'    =>  $scriptSessionId,

            ]);

    ##----------------------------------------------------------------
+-----------------------------------##

    #dwr.engine.remote.handleCallback("1","0",{minAnnIncome:45396.00,m
+inMonthlyIncome:3783.00});



    my $redirectConnector;

    $html=$mech->{content};

    if($html=~/handleCallback\(\s*["](.*?)["]\s*[,]\s*["](.*?)["]\s*[,
+]\s*["{](.*?)[}"]\s*\)/is)

    {

        $redirectConnector = $3;            

    }

    ##----------------------------------------------------------------
+-----------------------------------##

    my $url3='https://www.mahealthconnector.org/portal/site/connector/
+menuitem.55b6e23ac6627f40dbef6f47d7468a0c/';

    $mech->get($url3);

    if ($mech->response->status_line != STATUS_OK)

    {

        trace("Error fetching Page 5a. Status = ". $mech->response->st
+atus_line .". Try after sometime.");

        return;

    }        

    ##----------------------------------------------------------------
+-----------------------------------##

    #$mech->find_link( text_regex => qr/download/i );

    ####my $link=$mech->find_link( id_regex => qr/continue/i );

        my $link= $mech->click_button( value => "Continue" );

    my $continueLink='https://www.mahealthconnector.org'.$link->url;

    $mech->get($continueLink);

    if ($mech->response->status_line != STATUS_OK)

    {

        trace("Error fetching Page 5a. Status = ". $mech->response->st
+atus_line .". Try after sometime.");

        return;

    }        

    my $html = $mech->content;

    ##----------------------------------------------------------------
+-----------------------------------##

    ###$link= $mech->find_link( text_regex => qr/Browse\s+now/i );

        $link= $mech->click_button( value => "Shop Now" ); 

    my $browseLink='https://www.mahealthconnector.org'.$link->url;

    $mech->get($browseLink);

    if ($mech->response->status_line != STATUS_OK)

    {

        trace("Error fetching Page 3. Status = ". $mech->response->sta
+tus_line .". Try after sometime.");

        return;

    }

    ##----------------------------------------------------------------
+-----------------------------------##

    my $form1_fields = { 'js_enabled' => 'true' };   

    my $response = $mech->submit_form(    form_name     => 'js_form',

                        fields        => $form1_fields);



    if ($mech->response->status_line != STATUS_OK)

    {

        trace("Error fetching Page 3. Status = ". $mech->response->sta
+tus_line .". Try after sometime.");

        return;

    }

    ##----------------------------------------------------------------
+-----------------------------------##

    #trace("Fetching page with all family information");    

    my %form2_fields = getConnector2Fields(@subData);



    sleep(5);



    my $response = $mech->submit_form(    form_name     => 'connector_
+step2a_form',

                        fields        => \%form2_fields);

    if ($mech->response->status_line != STATUS_OK)

    {

        trace("Error fetching Page 3. Status = ". $mech->response->sta
+tus_line .". Try after sometime.");

        return;

    }

    ##----------------------------------------------------------------
+-----------------------------------##

    my $mechClone = $mech->clone();

    my $jsURL3='https://www.mahealthconnector.org/portal/dwr/call/plai
+ncall/__System.pageLoaded.dwr';

    my $response = $mechClone->post($jsURL3,

            [ 'callCount'        =>    "1",

              'windowName'        =>    "",

              'c0-scriptName'    =>    "__System",

              'c0-methodName'    =>    "pageLoaded",

              'c0-id'        =>    "0",

              'batchId'        =>    "0",

######              'page'        =>    "/portal/site/connector/templa
+te.PAGE/menuitem.99762c230fb3e650dbef6f47d7468a0c/?javax.portlet.tpst
+=1dd8f1b14739404575c4c2100ce08041&javax.portlet.prp_1dd8f1b1473940457
+5c4c2100ce08041=connector_currentstep%3Dconnector-step2a-load-rates%2
+6viewID%3DMY_PORTAL_VIEW&javax.portlet.begCacheTok=com.vignette.cache
+token&javax.portlet.endCacheTok=com.vignette.cachetoken",

                          'page'                =>    "/portal/site/co
+nnector/template.PAGE/menuitem.99762c230fb3e650dbef6f47d7468a0c/?java
+x.portlet.tpst=1dd8f1b14739404575c4c2100ce08041&javax.portlet.prp_1dd
+8f1b14739404575c4c2100ce08041=connector_currentstep%3Dconnector-step2
+a%26viewID%3DMY_PORTAL_VIEW&javax.portlet.begCacheTok=com.vignette.ca
+chetoken&javax.portlet.endCacheTok=com.vignette.cachetoken",

              'httpSessionId'    =>      $sessionId,

              'scriptSessionId'    =>    "",

            ]);

    ##----------------------------------------------------------------
+-----------------------------------##

    # dwr.engine.remote.handleNewScriptSession("71A70B8E00AB93A125F4D9
+86E5D7B66D");

    $html=$mechClone->{content};

    if($html=~/handleNewScriptSession\(\s*"(.*?)"\s*\)/is)

    {

        $scriptSessionId = $1;

    }

    ##----------------------------------------------------------------
+-----------------------------------##

    my $jsURL4='https://www.mahealthconnector.org/portal/dwr/call/plai
+ncall/NonGroupPlanPremiumsDelegate.fetchHealthPlans.dwr';

    my $response = $mechClone->post($jsURL4, 

            [ 'callCount'        =>    "1",

              'windowName'        =>    "",

              'c0-scriptName'    =>    "NonGroupPlanPremiumsDelegate",

              'c0-methodName'    =>    "fetchHealthPlans",

              'c0-id'        =>    "0",

              'batchId'        =>    "1",

######              'page'                =>  "/portal/site/connector/
+template.PAGE/menuitem.99762c230fb3e650dbef6f47d7468a0c/?javax.portle
+t.tpst=1dd8f1b14739404575c4c2100ce08041&javax.portlet.prp_1dd8f1b1473
+9404575c4c2100ce08041=connector_currentstep%3Dconnector-step2a-load-r
+ates%26viewID%3DMY_PORTAL_VIEW&javax.portlet.begCacheTok=com.vignette
+.cachetoken&javax.portlet.endCacheTok=com.vignette.cachetoken",

                          'page'                =>    "/portal/site/co
+nnector/template.PAGE/menuitem.99762c230fb3e650dbef6f47d7468a0c/?java
+x.portlet.tpst=1dd8f1b14739404575c4c2100ce08041&javax.portlet.prp_1dd
+8f1b14739404575c4c2100ce08041=connector_currentstep%3Dconnector-step2
+a%26viewID%3DMY_PORTAL_VIEW&javax.portlet.begCacheTok=com.vignette.ca
+chetoken&javax.portlet.endCacheTok=com.vignette.cachetoken",

              'httpSessionId'    =>      $sessionId,

              'scriptSessionId'    =>    $scriptSessionId,

            ]);

    ##----------------------------------------------------------------
+-----------------------------------##

    # dwr.engine.remote.handleCallback("1","0","connector-step3a");

    my $redirectConnector;

    $html=$mechClone->{content};

    if($html=~/handleCallback\(\s*["](.*?)["]\s*[,]\s*["](.*?)["]\s*[,
+]\s*["{](.*?)[}"]\s*\)/is)

    {

        $redirectConnector = $3;            

    }

    if($redirectConnector!~/connector-step3a/is)

    {

        trace("A network error occurred during your request.  Please t
+ry again later.");

        return;

    }

    undef($mechClone);

    ##----------------------------------------------------------------
+-----------------------------------##

    my $form1_fields = {

                      'connector_redirect_to_page' => $redirectConnect
+or,

                   };   

    $mech->submit_form(    form_name     => 'loadingrates',

                        fields        => $form1_fields);



    ##----------------------------------------------------------------
+-----------------------------------##

    my $form5_fields = {

                          'ben_pkg_choice' => 'All'

                       };   

    sleep(2);

    $mech->submit_form(    form_name     => 'connector_step3a_form',

                        fields        => $form5_fields);

                    

    if ($mech->response->status_line != STATUS_OK)

    {

        trace("Error fetching Page 5. Status = ". $mech->response->sta
+tus_line .". Try after sometime.");

        return;

    }

    ##----------------------------------------------------------------
+-----------------------------------##

    my $mechClone = $mech->clone();

    my $jsURL3='https://www.mahealthconnector.org/portal/dwr/call/plai
+ncall/__System.pageLoaded.dwr';



    my $response = $mechClone->post($jsURL3,

            [ 'callCount'        =>    "1",

              'windowName'        =>    "",

              'c0-scriptName'    =>    "__System",

              'c0-methodName'    =>    "pageLoaded",

              'c0-id'        =>    "0",

              'batchId'        =>    "0",

                          'page'        =>    "/portal/site/connector/
+template.PAGE/menuitem.99762c230fb3e650dbef6f47d7468a0c/?javax.portle
+t.tpst=1dd8f1b14739404575c4c2100ce08041&javax.portlet.prp_1dd8f1b1473
+9404575c4c2100ce08041=connector_currentstep%3Dconnector-step4soa%26vi
+ewID%3DMY_PORTAL_VIEW&javax.portlet.begCacheTok=com.vignette.cachetok
+en&javax.portlet.endCacheTok=com.vignette.cachetoken",

              'httpSessionId'    =>  $sessionId,

              'scriptSessionId'    =>    "",

            ]);

    ##----------------------------------------------------------------
+-----------##    

    # dwr.engine.remote.handleNewScriptSession("71A70B8E00AB93A125F4D9
+86E5D7B66D");

    $html=$mechClone->{content};

    if($html=~/handleNewScriptSession\(\s*"(.*?)"\s*\)/is)

    {

        $scriptSessionId = $1;

    }

    ##----------------------------------------------------------------
+-----------------------------------##



    my $jsURL2 = "https://www.mahealthconnector.org/portal/dwr/call/pl
+aincall/BrowsePlansDelegate.filterAndSortHealthPlans.dwr";

    

    my $response = $mechClone->post($jsURL2,

            [     'callCount'            => "1",

                'windowName'        => "",

                'c0-scriptName'        => "BrowsePlansDelegate",

                'c0-methodName'        => "filterAndSortHealthPlans",

                'c0-id'                => "0",

                'c0-param0'            => "string:Bronze%7CSilver%7CGo
+ld",

                'c0-param1'         => "string:",

                'c0-param2'         => "string:",

                'c0-param3'         => "string:",

                'c0-param4'         => "string:0",

                'batchId'        => "1",

                                'page'            => "/portal/site/con
+nector/template.PAGE/menuitem.99762c230fb3e650dbef6f47d7468a0c/?javax
+.portlet.tpst=1dd8f1b14739404575c4c2100ce08041&javax.portlet.prp_1dd8
+f1b14739404575c4c2100ce08041=connector_currentstep%3Dconnector-step4s
+oa%26viewID%3DMY_PORTAL_VIEW&javax.portlet.begCacheTok=com.vignette.c
+achetoken&javax.portlet.endCacheTok=com.vignette.cachetoken",

                'httpSessionId'        => $sessionId,

                'scriptSessionId'    => $scriptSessionId,

            ]);

    ##----------------------------------------------------------------
+-----------##    

    my $retVal = savePageData($savePage, $mechClone->content());

    

    if ($retVal == 1) {

        trace("Saved Final Page : $savePage\n");

    }

    else {

        trace("Error retrieving data for : $savePage\n");

    }    

    ##----------------------------------------------------------------
+-----------##    

    undef($mechClone);

    undef($mech);

    ##----------------------------------------------------------------
+-----------##    

}

[download]

what changes should i perform to make this continue button to be found by my perl code. i am new to perl cpan so please bear with me

Replies are listed 'Best First'.
Re: error received in WWW::Mechanize
by Corion (Patriarch) on Aug 04, 2012 at 08:27 UTC

    The first step is to reduce your program to a short, self-contained runnable program that still exhibits the problem. This will eliminate much of the unnecessary setup and also show us which modules you load into your program, and also at which line the erroneous call actually happens.

[reply]
    A reply falls below the community's threshold of quality. You may see it by logging in.
Re: error received in WWW::Mechanize
by aitap (Curate) on Aug 04, 2012 at 10:23 UTC
    You can use Perl debugger (perl -d myscript.pl, commands t, b, c, n, s) to trace execution of your code and find the cause of this exception.
    Sorry, your code example does not allow us to help you further.
    Sorry if my advice was wrong.
[reply]
[d/l]
[select]

Back to Seekers of Perl Wisdom