in reply to Re^4: WIN32::IE::Mechanize - can't follow link - JavaScript involved
in thread WIN32::IE::Mechanize - can't follow link - JavaScript involved

You are right> I didn't notice that the image tag was inside a

<a HREF="JavaScript:JSubmitForm()" onMouseOver="status='Perform Search'; return true" onMouseOut="status='';">

tag. When you click the image, it its the HREF attribute that fires the jSubmitForm() sub from javascript.

The "go" buttons are interchangeable because thet all call the same JS function.

So we return to my previous post: if you get the linked javascript you'll find the source of the function:

function JSubmitForm()
{
    document.Content.Name.value = document.Search1.Name.value;
    document.Content.CrdNumber.value = document.Search2.CrdNumber.valu
+e;        
    document.Content.SecNumber.value = document.Search3.SecNumber.valu
+e;
    
    if (eval("document.Search1.SearchType[0].checked") == true) 
    {
        document.Content.SearchType.value = 1;
    }
    if (eval("document.Search1.SearchType[1].checked") == true)
    {
        document.Content.SearchType.value = 2;
    }
    if (eval("document.Search1.SearchType[2].checked") == true)
    {
        document.Content.SearchType.value = 3;
    }
    
    if (eval("document.Search4.NumRows[0].selected") == true) 
    {
        document.Content.NumRows.value = document.Search4.NumRows[0].v
+alue;
    }
    if (eval("document.Search4.NumRows[1].selected") == true) 
    {
        document.Content.NumRows.value = document.Search4.NumRows[1].v
+alue;
    }
    if (eval("document.Search4.NumRows[2].selected") == true) 
    {
        document.Content.NumRows.value = document.Search4.NumRows[2].v
+alue;
    }
    if (eval("document.Search4.NumRows[3].selected") == true) 
    {
        document.Content.NumRows.value = document.Search4.NumRows[3].v
+alue;
    }
    
    document.Content.submit();
    return false;
}

[download]

As you can see, the function does nothing more than setting the hidden fields SearchType and NumRows to the appropriate values depending how the form was compiled and then submit the form.

This means that you should be able to completely bypass the downloading of the form, and produce a simple GET with the correct parameters that retrieves directly the result you want.

Rule One: Do not act incautiously when confronting a little bald wrinkly smiling man.

Replies are listed 'Best First'.
Re^6: WIN32::IE::Mechanize - can't follow link - JavaScript involved
by Ninth Prince (Acolyte) on May 22, 2008 at 19:39 UTC

    Okay, here is what I tried (see below). When I run this I just get the original web page that contains the form, but with the values filled in. The form never "fires" (i.e., I'm not getting through to the next page that I actually want).

    Here is my code.

    #!/usr/bin/perl -w

use LWP::UserAgent ;
use URI ;

my $url = URI->new( 'http://www.adviserinfo.sec.gov/IAPD/Content/Searc
+h/iapd_OrgSearch.aspx' ) ;

$url->query_form(
   'CrdNumber' => "144549",
   'NumRows' => "25",
   'SearchType' => "1",
) ;

my $response = LWP::UserAgent->new->get($url) ;


print $response->content();

exit ;

    [download]

    And here is the HTML that I get back. 

    

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">



<html>

    <head>

        <META content="text/html; charset=windows-1252" http-equiv=Con
+tent-Type>

        <title>Investment Adviser Search</title>

        <LINK href='/IAPD/Stylesheets/iapd.css' rel=stylesheet type=te
+xt/css media='screen'/>

        <LINK href='/IAPD/Stylesheets/iapd_print.css' rel=stylesheet t
+ype=text/css media='print'/>

        

    <script type="text/javascript" language='Javascript' src='/IAPD/In
+cludes/Validation/Search/iapd_OrgSearch.js'></script> 

      <script type="text/javascript" language='Javascript' src='/Iapd/
+Includes/iapd_WindowManagement.js'></script> 

    <script type="text/javascript" language='Javascript' src='/Iapd/In
+cludes/iapd_SetAndSub.js'></script> 

    <script language=javascript>

       function init() 

       {

            if(document.Content.Name != null)

                document.Content.Name.value = "";

            if(document.Content.CrdNumber != null)

                document.Content.CrdNumber.value = "";

            if(document.Content.SecNumber != null)

                document.Content.SecNumber.value = "";

       }

       window.onload = init; 

    </script>



    </head>



    <body bgColor=#ffffff leftmargin="0" topmargin="0">

        <a NAME="PageTop"></a>

        <table border="0" cellpadding="0" cellspacing="0" width="100%"
+ class="noprint">

            <tbody>

            <tr valign="top">

                <td height="92" rowSpan="5" nowrap><img alt="SEC Seal"
+ border="0" height="92" src="/IAPD/Images/SEC_bannerSealTop2.gif" wid
+th="95"><img alt="SEC Seal" border="0" height="92" src="/IAPD/Images/
+SEC_bannerSealTopRt.gif" width="26"></td>

                <td background="/IAPD/Images/SEC_stripedbgMain.gif" he
+ight="39" width="171"><img alt="" border="0" height="39" src="/IAPD/I
+mages/SEC_bannerFlagMain.gif" width="171"></td>

                <td align="right" background="/IAPD/Images/SEC_striped
+bgMain.gif" class="gray" valign="bottom" width="100%"></td>

                <td rowspan="4" width="126">

                    <table border="0" cellpadding="0" cellspacing="0">

                        <tbody>

                        <tr>

                            <td rowspan="3"><img alt="" height="90" sr
+c="/IAPD/Images/flag_lt.gif" width="7"></td>

                            <td><img alt="" height=12 src="/IAPD/Image
+s/flag_top.gif" width="112"></td>

                            <td rowspan=3><img alt="" height="90" src=
+"/IAPD/Images/flag_rt.gif" width="7"></td>

                        </tr>

                        <tr>

                            <td><img id="_ctl0_imgFlag" src="/IAPD/Ima
+ges/State_Regulation.gif" alt="State Regulation" border="0" height="7
+2" width="112" /></td>

                        </tr>

                        <tr>

                            <td><img alt="" height="6" src="/IAPD/Imag
+es/flag_bot.gif" width="112"></td>

                        </tr>

                        </tbody>

                    </table>

                </td>

            </tr>

            <tr valign="top">

                <td bgcolor="black"><img alt="" height="1" src="/IAPD/
+Images/pixel.gif" width="1"></td>

                <td bgcolor="black" height="34"><img alt="Investment A
+dviser Public Disclosure" border="0" height="34" src="/IAPD/Images/IA
+PDtitle.gif" width="391"></td>

            </tr>

            <tr>

                <td bgcolor="white" height="2"><img alt="" height="2" 
+src="/IAPD/Images/pixel.gif"></td>

                <td bgcolor="white" height="2"><img alt="" height="1" 
+src="/IAPD/Images/pixel.gif" width="1"></td>

            </tr>

            <tr>

                <td bgcolor="#324395" colspan="2" height="15"><img alt
+="" border="0" height="15" src="/IAPD/Images/SEC_bannerFlagBot.gif" w
+idth="50"></td>

            </tr>

            <tr>

                <td bgcolor="white" colspan="5" height="2"><img alt=""
+ height="2" src="/IAPD/Images/pixel.gif"></td>

                <td bgcolor="white" height="2"></td>

            </tr>

            </tbody>

        </table>

        

        <table id="_ctl0_tblMain" border="0" cellpadding="0" height="1
+00%" cellspacing="0" width="100%">

    <tr valign="top">

        <td bgcolor="#e1e1d6" colspan="3" valign="top" width="1%">

                    <table border="0" cellpadding="0" cellspacing="0" 
+width="150" class="noprint">

                        <tbody>

                        <tr>

                            <td><img src="/IAPD/Images/SEC_SealBotFron
+tB.gif" alt="SEC Seal"></td>

                        </tr>

                        <tr>

                            <td><img alt="" border="0" height="1" src=
+"/IAPD/Images/pixel.gif" width="5"></td>

                        </tr>

                        </tbody>

                    </table>

                    <table border="0" cellpadding="0" cellspacing="5" 
+width="130" class="noprint">

                        <tbody>

                        <tr>

                            <td>

                                <!-- Left Navigation -->

                                

                                    <table border="0" width="130" cell
+spacing="5" cellpadding="0">

  <tr>

    <td>

      <img src="/IAPD/images/IAPDhome2.jpg" alt="IAPD Main">

    </td>

  </tr>

  <tr>

    <td>

      <table border="0" width="130" cellspacing="3" cellpadding="0">

        <tr>

          <td>

            <a onMouseOver="status='Link To the SEC';return true" onMo
+useOut="status='';" class="Nav" href="javascript: JSub('/IAPD/Content
+/IAPDMain/iapd_SECRedirect.aspx');">Return to the SEC</a>

          </td>

        </tr>

        <tr>

          <td>

            <a onMouseOver="status='Link To the SEC IARD Page';return 
+true" onMouseOut="status='';" class="Nav" href="javascript: JSub('/IA
+PD/Content/IAPDMain/iapd_SECRedirect.aspx?dir=IARD');">Return to SEC 
+IARD Page</a>

          </td>

        </tr>

        <tr>

          <td>

            <a onMouseOver="status='Link To Sitemap';return true" onMo
+useOut="status='';" class="Nav" href="javascript: JSub('/IAPD/Content
+/IAPDMain/iapd_SiteMap.aspx');">Return to Sitemap</a>

          </td>

        </tr>

        <tr>

          <td>

            <a onMouseOver="status='Link To Investment Adviser Search'
+;return true" onMouseOut="status='';" class="NavSelected" href="javas
+cript: JSub('/IAPD/Content/Search/iapd_OrgSearch.aspx');">Investment 
+Adviser Search</a>

          </td>

        </tr>

      </table>

    </td>

  </tr>

</table>

<BR>

<BR>

                                

                                <!-- Left Nav Control Goes Here -->

                            </td>

                        </tr>

                        </tbody>

                    </table>

                </td>

        <td>

                    <table border="0" cellpadding="0" cellspacing="0" 
+width="100%">

                        <tr>

                            <td id="_ctl0_tdBackground" background="/I
+APD/Images/iapd-wmk2.jpg">

                                <img alt="" height="10" src="/IAPD/Ima
+ges/pixel.gif" width="540"><br />

                                <table border="0" cellPadding="0" cell
+Spacing="0" width="100%">

                                    <tbody>

                                    <tr>

                                        <td>

                                            <!-- Main Content -->

                                             

    <script language="Javascript">

        var bDoneLoading = "N"

    </script>

<table Border="0" cellspacing="0" cellpadding="0" height="100%" width=
+"100%">

  <tr valign="top" height="96%">

    <td width="100%" colspan="3">

      <DIV id="content">

        <div class="PageTitle">Investment Adviser Search</div>

        <input name="Save" type="Hidden" value="IAPDSearch">

        <table width="100%" border="0" cellpadding="0" cellspacing="0"
+>

          <tr>

            <td class="QueryTableTitle" colspan="5">Search for an Inve
+stment Adviser Firm:</td>

          </tr>

          <form name="Search1" method="Post" onSubmit="return JSubmitF
+orm();">

            <tr>

              <td colspan="5">

                <table border="0" cellPadding="3" cellSpacing="0" widt
+h="100%">

                  <tbody>

                    <tr>

                      <td class="QueryTableLabel" colspan="1" width="3
+0%">Firm Name:</td>

                      <td class="QueryTableLabel" colspan="1">

                        <input type="text" name="Name" value="" size="
+30" maxlength="64">

                        <a HREF="JavaScript:JSubmitForm()" onMouseOver
+="status='Perform Search'; return true" onMouseOut="status='';">

                          <img SRC="/IAPD/Images/go_off.gif" alt="Go" 
+name="go" onMouseover="JImgAct('go')" onMouseout="JImgInact('go')" BO
+RDER="0" align="top">

                        </a>

                      </td>

                    </tr>

                    <tr>

                      <td class="QueryTableLabel" width="30%" valign="
+top">Type of Firm Search:</td>

                      <td class="QueryTableLabel">

                        <input type="radio" name="SearchType" value="1
+" checked="true">Starts With<br><input type="radio" name="SearchType"
+ value="2">Contains<br><input type="radio" name="SearchType" value="3
+">Sounds Like</td>

                    </tr>

                  </tbody>

                </table>

              </td>

            </tr>

          </form>

          <tr>

            <td colSpan="5">

              <hr>

            </td>

          </tr>

          <form name="Search2" method="Post" onSubmit="return JSubmitF
+orm();">

            <tr>

              <td colspan="5">

                <table border="0" cellPadding="3" cellSpacing="0" widt
+h="100%">

                  <tbody>

                    <tr>

                      <td width="30%" class="BoldTD" colspan="2">Firm 
+IARD/CRD Number:</td>

                      <td colspan="3">

                        <input type="text" name="CrdNumber" value="144
+549" size="16" maxlength="12">

                        <a HREF="JavaScript:JSubmitForm()" onMouseOver
+="status='Perform Search'; return true" onMouseOut="status='';">

                          <img SRC="/IAPD/Images/go_off.gif" alt="Go" 
+name="go2" onMouseover="JImgAct('go2')" onMouseout="JImgInact('go2')"
+ BORDER="0" align="top">

                        </a>

                      </td>

                    </tr>

                  </tbody>

                </table>

              </td>

            </tr>

          </form>

          <tr>

            <td colSpan="5">

              <hr>

            </td>

          </tr>

          <form name="Search3" method="Post" onSubmit="return JSubmitF
+orm();">

            <tr>

              <td colspan="5">

                <table border="0" cellPadding="3" cellSpacing="0" widt
+h="100%">

                  <tbody>

                    <tr>

                      <td width="20%" class="BoldTD" colspan="1">Firm 
+SEC Number:</td>

                      <td class="BoldRightTD" width="10%">801-</td>

                      <td colspan="3">

                        <input type="text" name="SecNumber" value="" s
+ize="16" maxlength="16">

                        <a HREF="JavaScript:JSubmitForm()" onMouseOver
+="status='Perform Search'; return true" onMouseOut="status='';">

                          <img SRC="/IAPD/Images/go_off.gif" alt="Go" 
+name="go3" onMouseover="JImgAct('go3')" onMouseout="JImgInact('go3')"
+ BORDER="0" align="top">

                        </a>

                      </td>

                    </tr>

                  </tbody>

                </table>

              </td>

            </tr>

          </form>

          <tr>

            <td colSpan="5">

              <hr>

            </td>

          </tr>

          <form name="Search4">

            <tr>

              <td colspan="5">

                <table border="0" cellPadding="3" cellSpacing="0" widt
+h="100%">

                  <tbody>

                    <tr>

                      <td width="30%" colspan="2" class="BoldTD">Recor
+ds displayed per Page:</td>

                      <td colspan="3">

                        <select name="NumRows">

                          <option value="25">25</option>

                          <option value="50">50</option>

                          <option value="75">75</option>

                          <option value="100">100</option>

                        </select>

                      </td>

                    </tr>

                  </tbody>

                </table>

              </td>

            </tr>

          </form>

        </table>

        <br>

        <br>

        <br>

        <br>

        <form name="Content" method="Post">

          <input type="hidden" name="LinkPage" value="">

          <input type="hidden" name="PageType" value="Search">

          <input type="hidden" name="ORG_PK" value="000">

          <input type="hidden" name="STATE_CD" value="">

          <input type="hidden" name="Name" value="">

          <input type="hidden" name="SearchType" value="">

          <input type="hidden" name="CrdNumber" value="">

          <input type="hidden" name="SecNumber" value="">

          <input type="hidden" name="NumRows" value="">

          <input type="hidden" name="Save" value="IAPDSearch">

        </form>

      </DIV>

    </td>

  </tr>

</table>

<script language="Javascript">

        bDoneLoading = "Y"

    </script> 



                                        </td>

                                    </tr>

                                    </tbody>

                                </table>

                            </td>

        

                        </tr>

                    </table>

                </td>

    </tr>

    <tr>

        <td colspan="4" class="FooterMenuItem noprint">

                    

                        <!-- Navigation Footer -->

                        <table border="0" cellpadding="0" cellspacing=
+"0" width="100%">

  <tr class="noprint" align="top">

    <TD colspan="3" align="center" valign="top" width="150" Class="Bac
+kToTop">

      <a HREF="#PageTop" Class="FooterMenuItem" onMouseOver="status='S
+croll Back to Top of this Page';return true" onMouseOut="status=''">B
+ack to Top</a>

    </TD>

    <td Class="FooterMenuItem" align="Center">

      <div align="center">

        <b>IAPD Main</b>

      </div>

      <a class="FooterMenuItem" href="javascript: JSub('/IAPD/Content/
+IAPDMain/iapd_SECRedirect.aspx');" onMouseOver="status='Link To the S
+EC'; return true" onMouseOut="status=''">Return to the SEC</a>
           | 
       <a class="FooterMenuItem" href="javascript: JSub('/IAPD/Content
+/IAPDMain/iapd_SECRedirect.aspx?dir=IARD');" onMouseOver="status='Lin
+k To the SEC IARD Page'; return true" onMouseOut="status=''">Return t
+o SEC IARD Page</a>
           | 
       <a class="FooterMenuItem" href="javascript: JSub('/IAPD/Content
+/IAPDMain/iapd_SiteMap.aspx');" onMouseOver="status='Link To Sitemap'
+; return true" onMouseOut="status=''">Return to Sitemap</a>
           | 
       <a class="FooterItemSelected" href="javascript: JSub('/IAPD/Con
+tent/Search/iapd_OrgSearch.aspx');" onMouseOver="status='Link To Inve
+stment Adviser Search'; return true" onMouseOut="status=''">Investmen
+t Adviser Search</a> 
    
 </td>

  </tr>

</table>

                    

                </td>

    </tr>

</table>



    </body>

</html>

    [download]
[reply]
[d/l]
[select]

      It is because you make a GET and not a POST and you don't valorized some field.

      Replace this in your code:

      $url->query_form(
   'CrdNumber' => "144549",
   'NumRows' => "25",
   'SearchType' => "2",
   'ORG_PK' => "000",
   'Save' => "IAPDSearch",
) ;

my $response = LWP::UserAgent->new->post($url) ;

      [download]

      and it will work.

      Rule One: Do not act incautiously when confronting a little bald wrinkly smiling man.

[reply]
[d/l]
        Thanks again for all you help. It works perfectly now!
[reply]

In Section Seekers of Perl Wisdom