in reply to Re: WIN32::IE::Mechanize - can't follow link - JavaScript involved
in thread WIN32::IE::Mechanize - can't follow link - JavaScript involved

Oops, sorry about that. I don't know enough about JavaScript to even know that I should have given you all or the HTML. Anyway, here it is.

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">

<html>
    <head>
        <META content="text/html; charset=windows-1252" http-equiv=Con
+tent-Type>
        <title>Investment Adviser Search</title>
        <LINK href='/IAPD/Stylesheets/iapd.css' rel=stylesheet type=te
+xt/css media='screen'/>
        <LINK href='/IAPD/Stylesheets/iapd_print.css' rel=stylesheet t
+ype=text/css media='print'/>
        
    <script type="text/javascript" language='Javascript' src='/IAPD/In
+cludes/Validation/Search/iapd_OrgSearch.js'></script> 
      <script type="text/javascript" language='Javascript' src='/Iapd/
+Includes/iapd_WindowManagement.js'></script> 
    <script type="text/javascript" language='Javascript' src='/Iapd/In
+cludes/iapd_SetAndSub.js'></script> 
    <script language=javascript>
       function init() 
       {
            if(document.Content.Name != null)
                document.Content.Name.value = "";
            if(document.Content.CrdNumber != null)
                document.Content.CrdNumber.value = "";
            if(document.Content.SecNumber != null)
                document.Content.SecNumber.value = "";
       }
       window.onload = init; 
    </script>

    </head>

    <body bgColor=#ffffff leftmargin="0" topmargin="0">
        <a NAME="PageTop"></a>
        <table border="0" cellpadding="0" cellspacing="0" width="100%"
+ class="noprint">
            <tbody>
            <tr valign="top">
                <td height="92" rowSpan="5" nowrap><img alt="SEC Seal"
+ border="0" height="92" src="/IAPD/Images/SEC_bannerSealTop2.gif" wid
+th="95"><img alt="SEC Seal" border="0" height="92" src="/IAPD/Images/
+SEC_bannerSealTopRt.gif" width="26"></td>
                <td background="/IAPD/Images/SEC_stripedbgMain.gif" he
+ight="39" width="171"><img alt="" border="0" height="39" src="/IAPD/I
+mages/SEC_bannerFlagMain.gif" width="171"></td>
                <td align="right" background="/IAPD/Images/SEC_striped
+bgMain.gif" class="gray" valign="bottom" width="100%"></td>
                <td rowspan="4" width="126">
                    <table border="0" cellpadding="0" cellspacing="0">
                        <tbody>
                        <tr>
                            <td rowspan="3"><img alt="" height="90" sr
+c="/IAPD/Images/flag_lt.gif" width="7"></td>
                            <td><img alt="" height=12 src="/IAPD/Image
+s/flag_top.gif" width="112"></td>
                            <td rowspan=3><img alt="" height="90" src=
+"/IAPD/Images/flag_rt.gif" width="7"></td>
                        </tr>
                        <tr>
                            <td><img id="_ctl0_imgFlag" src="/IAPD/Ima
+ges/State_Regulation.gif" alt="State Regulation" border="0" style="he
+ight:72px;width:112px;" /></td>
                        </tr>
                        <tr>
                            <td><img alt="" height="6" src="/IAPD/Imag
+es/flag_bot.gif" width="112"></td>
                        </tr>
                        </tbody>
                    </table>
                </td>
            </tr>
            <tr valign="top">
                <td bgcolor="black"><img alt="" height="1" src="/IAPD/
+Images/pixel.gif" width="1"></td>
                <td bgcolor="black" height="34"><img alt="Investment A
+dviser Public Disclosure" border="0" height="34" src="/IAPD/Images/IA
+PDtitle.gif" width="391"></td>
            </tr>
            <tr>
                <td bgcolor="white" height="2"><img alt="" height="2" 
+src="/IAPD/Images/pixel.gif"></td>
                <td bgcolor="white" height="2"><img alt="" height="1" 
+src="/IAPD/Images/pixel.gif" width="1"></td>
            </tr>
            <tr>
                <td bgcolor="#324395" colspan="2" height="15"><img alt
+="" border="0" height="15" src="/IAPD/Images/SEC_bannerFlagBot.gif" w
+idth="50"></td>
            </tr>
            <tr>
                <td bgcolor="white" colspan="5" height="2"><img alt=""
+ height="2" src="/IAPD/Images/pixel.gif"></td>
                <td bgcolor="white" height="2"></td>
            </tr>
            </tbody>
        </table>
        
        <table id="_ctl0_tblMain" border="0" cellpadding="0" height="1
+00%" cellspacing="0" width="100%">
    <tr valign="top">
        <td bgcolor="#e1e1d6" colspan="3" valign="top" width="1%">
                    <table border="0" cellpadding="0" cellspacing="0" 
+width="150" class="noprint">
                        <tbody>
                        <tr>
                            <td><img src="/IAPD/Images/SEC_SealBotFron
+tB.gif" alt="SEC Seal"></td>
                        </tr>
                        <tr>
                            <td><img alt="" border="0" height="1" src=
+"/IAPD/Images/pixel.gif" width="5"></td>
                        </tr>
                        </tbody>
                    </table>
                    <table border="0" cellpadding="0" cellspacing="5" 
+width="130" class="noprint">
                        <tbody>
                        <tr>
                            <td>
                                <!-- Left Navigation -->
                                
                                    <table border="0" width="130" cell
+spacing="5" cellpadding="0">
  <tr>
    <td>
      <img src="/IAPD/images/IAPDhome2.jpg" alt="IAPD Main">
    </td>
  </tr>
  <tr>
    <td>
      <table border="0" width="130" cellspacing="3" cellpadding="0">
        <tr>
          <td>
            <a onMouseOver="status='Link To the SEC';return true" onMo
+useOut="status='';" class="Nav" href="javascript: JSub('/IAPD/Content
+/IAPDMain/iapd_SECRedirect.aspx');">Return to the SEC</a>
          </td>
        </tr>
        <tr>
          <td>
            <a onMouseOver="status='Link To the SEC IARD Page';return 
+true" onMouseOut="status='';" class="Nav" href="javascript: JSub('/IA
+PD/Content/IAPDMain/iapd_SECRedirect.aspx?dir=IARD');">Return to SEC 
+IARD Page</a>
          </td>
        </tr>
        <tr>
          <td>
            <a onMouseOver="status='Link To Sitemap';return true" onMo
+useOut="status='';" class="Nav" href="javascript: JSub('/IAPD/Content
+/IAPDMain/iapd_SiteMap.aspx');">Return to Sitemap</a>
          </td>
        </tr>
        <tr>
          <td>
            <a onMouseOver="status='Link To Investment Adviser Search'
+;return true" onMouseOut="status='';" class="NavSelected" href="javas
+cript: JSub('/IAPD/Content/Search/iapd_OrgSearch.aspx');">Investment 
+Adviser Search</a>
          </td>
        </tr>
      </table>
    </td>
  </tr>
</table>
<BR>
<BR>
                                
                                <!-- Left Nav Control Goes Here -->
                            </td>
                        </tr>
                        </tbody>
                    </table>
                </td>
        <td>
                    <table border="0" cellpadding="0" cellspacing="0" 
+width="100%">
                        <tr>
                            <td id="_ctl0_tdBackground" background="/I
+APD/Images/iapd-wmk2.jpg">
                                <img alt="" height="10" src="/IAPD/Ima
+ges/pixel.gif" width="540"><br />
                                <table border="0" cellPadding="0" cell
+Spacing="0" width="100%">
                                    <tbody>
                                    <tr>
                                        <td>
                                            <!-- Main Content -->
                                             
    <script language="Javascript">
        var bDoneLoading = "N"
    </script>
<table Border="0" cellspacing="0" cellpadding="0" height="100%" width=
+"100%">
  <tr valign="top" height="96%">
    <td width="100%" colspan="3">
      <DIV id="content">
        <div class="PageTitle">Investment Adviser Search</div>
        <input name="Save" type="Hidden" value="IAPDSearch">
        <table width="100%" border="0" cellpadding="0" cellspacing="0"
+>
          <tr>
            <td class="QueryTableTitle" colspan="5">Search for an Inve
+stment Adviser Firm:</td>
          </tr>
          <form name="Search1" method="Post" onSubmit="return JSubmitF
+orm();">
            <tr>
              <td colspan="5">
                <table border="0" cellPadding="3" cellSpacing="0" widt
+h="100%">
                  <tbody>
                    <tr>
                      <td class="QueryTableLabel" colspan="1" width="3
+0%">Firm Name:</td>
                      <td class="QueryTableLabel" colspan="1">
                        <input type="text" name="Name" value="" size="
+30" maxlength="64">
                        <a HREF="JavaScript:JSubmitForm()" onMouseOver
+="status='Perform Search'; return true" onMouseOut="status='';">
                          <img SRC="/IAPD/Images/go_off.gif" alt="Go" 
+name="go" onMouseover="JImgAct('go')" onMouseout="JImgInact('go')" BO
+RDER="0" align="top">
                        </a>
                      </td>
                    </tr>
                    <tr>
                      <td class="QueryTableLabel" width="30%" valign="
+top">Type of Firm Search:</td>
                      <td class="QueryTableLabel">
                        <input type="radio" name="SearchType" value="1
+" checked="true">Starts With<br><input type="radio" name="SearchType"
+ value="2">Contains<br><input type="radio" name="SearchType" value="3
+">Sounds Like</td>
                    </tr>
                  </tbody>
                </table>
              </td>
            </tr>
          </form>
          <tr>
            <td colSpan="5">
              <hr>
            </td>
          </tr>
          <form name="Search2" method="Post" onSubmit="return JSubmitF
+orm();">
            <tr>
              <td colspan="5">
                <table border="0" cellPadding="3" cellSpacing="0" widt
+h="100%">
                  <tbody>
                    <tr>
                      <td width="30%" class="BoldTD" colspan="2">Firm 
+IARD/CRD Number:</td>
                      <td colspan="3">
                        <input type="text" name="CrdNumber" value="" s
+ize="16" maxlength="12">
                        <a HREF="JavaScript:JSubmitForm()" onMouseOver
+="status='Perform Search'; return true" onMouseOut="status='';">
                          <img SRC="/IAPD/Images/go_off.gif" alt="Go" 
+name="go2" onMouseover="JImgAct('go2')" onMouseout="JImgInact('go2')"
+ BORDER="0" align="top">
                        </a>
                      </td>
                    </tr>
                  </tbody>
                </table>
              </td>
            </tr>
          </form>
          <tr>
            <td colSpan="5">
              <hr>
            </td>
          </tr>
          <form name="Search3" method="Post" onSubmit="return JSubmitF
+orm();">
            <tr>
              <td colspan="5">
                <table border="0" cellPadding="3" cellSpacing="0" widt
+h="100%">
                  <tbody>
                    <tr>
                      <td width="20%" class="BoldTD" colspan="1">Firm 
+SEC Number:</td>
                      <td class="BoldRightTD" width="10%">801-</td>
                      <td colspan="3">
                        <input type="text" name="SecNumber" value="" s
+ize="16" maxlength="16">
                        <a HREF="JavaScript:JSubmitForm()" onMouseOver
+="status='Perform Search'; return true" onMouseOut="status='';">
                          <img SRC="/IAPD/Images/go_off.gif" alt="Go" 
+name="go3" onMouseover="JImgAct('go3')" onMouseout="JImgInact('go3')"
+ BORDER="0" align="top">
                        </a>
                      </td>
                    </tr>
                  </tbody>
                </table>
              </td>
            </tr>
          </form>
          <tr>
            <td colSpan="5">
              <hr>
            </td>
          </tr>
          <form name="Search4">
            <tr>
              <td colspan="5">
                <table border="0" cellPadding="3" cellSpacing="0" widt
+h="100%">
                  <tbody>
                    <tr>
                      <td width="30%" colspan="2" class="BoldTD">Recor
+ds displayed per Page:</td>
                      <td colspan="3">
                        <select name="NumRows">
                          <option value="25">25</option>
                          <option value="50">50</option>
                          <option value="75">75</option>
                          <option value="100">100</option>
                        </select>
                      </td>
                    </tr>
                  </tbody>
                </table>
              </td>
            </tr>
          </form>
        </table>
        <br>
        <br>
        <br>
        <br>
        <form name="Content" method="Post">
          <input type="hidden" name="LinkPage" value="">
          <input type="hidden" name="PageType" value="Search">
          <input type="hidden" name="ORG_PK" value="000">
          <input type="hidden" name="STATE_CD" value="">
          <input type="hidden" name="Name" value="">
          <input type="hidden" name="SearchType" value="">
          <input type="hidden" name="CrdNumber" value="">
          <input type="hidden" name="SecNumber" value="">
          <input type="hidden" name="NumRows" value="">
          <input type="hidden" name="Save" value="IAPDSearch">
        </form>
      </DIV>
    </td>
  </tr>
</table>
<script language="Javascript">
        bDoneLoading = "Y"
    </script> 

                                        </td>
                                    </tr>
                                    </tbody>
                                </table>
                            </td>
        
                        </tr>
                    </table>
                </td>
    </tr>
    <tr>
        <td colspan="4" class="FooterMenuItem noprint">
                    
                        <!-- Navigation Footer -->
                        <table border="0" cellpadding="0" cellspacing=
+"0" width="100%">
  <tr class="noprint" align="top">
    <TD colspan="3" align="center" valign="top" width="150" Class="Bac
+kToTop">
      <a HREF="#PageTop" Class="FooterMenuItem" onMouseOver="status='S
+croll Back to Top of this Page';return true" onMouseOut="status=''">B
+ack to Top</a>
    </TD>
    <td Class="FooterMenuItem" align="Center">
      <div align="center">
        <b>IAPD Main</b>
      </div>
      <a class="FooterMenuItem" href="javascript: JSub('/IAPD/Content/
+IAPDMain/iapd_SECRedirect.aspx');" onMouseOver="status='Link To the S
+EC'; return true" onMouseOut="status=''">Return to the SEC</a>
           | 
       <a class="FooterMenuItem" href="javascript: JSub('/IAPD/Content
+/IAPDMain/iapd_SECRedirect.aspx?dir=IARD');" onMouseOver="status='Lin
+k To the SEC IARD Page'; return true" onMouseOut="status=''">Return t
+o SEC IARD Page</a>
           | 
       <a class="FooterMenuItem" href="javascript: JSub('/IAPD/Content
+/IAPDMain/iapd_SiteMap.aspx');" onMouseOver="status='Link To Sitemap'
+; return true" onMouseOut="status=''">Return to Sitemap</a>
           | 
       <a class="FooterItemSelected" href="javascript: JSub('/IAPD/Con
+tent/Search/iapd_OrgSearch.aspx');" onMouseOver="status='Link To Inve
+stment Adviser Search'; return true" onMouseOut="status=''">Investmen
+t Adviser Search</a> 
    
 </td>
  </tr>
</table>
                    
                </td>
    </tr>
</table>

    </body>
</html>

[download]

What is it that I'm looking for? Thanks.

  • Comment on Re^2: WIN32::IE::Mechanize - can't follow link - JavaScript involved
  • Download Code

Replies are listed 'Best First'.
Re^3: WIN32::IE::Mechanize - can't follow link - JavaScript involved
by psini (Deacon) on May 22, 2008 at 17:33 UTC

    Near the beginning of the file, there are six lines:

     <script type="text/javascript" language='Javascript' src='/IAPD/Inclu
+des/Validation/Search/iapd_OrgSearch.js'></script> 
      <script type="text/javascript" language='Javascript' src='/Iapd/
+Includes/iapd_WindowManagement.js'></script> 
    <script type="text/javascript" language='Javascript' src='/Iapd/In
+cludes/iapd_SetAndSub.js'></script> 
    <script language=javascript>

    [download]

    that tell the browser to load the corresponding scripts (eg /IAPD/Includes/Validation/Search/iapd_OrgSearch.js and the other two). If you follow the links (they are relative to the URL from which you got the HTML page) you can get the scripts.

    Then, in one the three scripts, you will find the javascript code which lays behind the page.

    Reading better the HTML I see a strange thing:

    <img SRC="/IAPD/Images/go_off.gif" alt="Go" name="go2" onMouseover="JImgAct('go2')" onMouseout="JImgInact('go2')" BORDER="0" align="top">

    is the definition of the object that you call "Button GO2", but it is not a button, it is a image and I see handlers only for onMouseover and onMouseut (called when the mous pass over the image and when it exits from the image). For this to act as a button, it should have a handler for onClick, too.

    Are you sure that this is the right object?

    Rule One: Do not act incautiously when confronting a little bald wrinkly smiling man.

[reply]
[d/l]
[select]

      No, I'm not sure, but it seems like the correct object to me.

      By way of background, this is a form that appears on the Securities and Exchange Commission's web site that investors can use to get information about investment advisors. The URL is http://www.adviserinfo.sec.gov/IAPD/Content/Search/iapd_OrgSearch.aspx.

      When I open this page up in a regular browser I fill in the field next to the label "CrdNumber" with a known value for an existing investment advisor (the value I use is 144549). To submit the form, I click on what looks like a "button" that is next to the field that I filled in. This process gets me to the page (information) that I really want. This is the process that I would like to automate over a large number of investment advisors

      Although I have never worked with JS, I have worked a little with Java, so it seemed odd to me that there was no "onClick()" method in the underlying source code. There was an earlier thread that I read that had a similar issue and the advice was that the object is not, in fact, a button. That thread suggested using ie->follow_link() which is why I tried adding it to my code.

      One other thing, although there are four (4) forms on the page, once I have filled in the CrdNumber field, if I click on ANY of the "submit" buttons (they all have different names) my submission works fine. The reasons for this are probably obvious to someone with more coding experience than myself, but it is a mystery to me.

      I'm going to try to follow those (relative) URLs from the JS lines. I will get back on that. Thanks for you advice.

[reply]
[d/l]

        You are right> I didn't notice that the image tag was inside a

        <a HREF="JavaScript:JSubmitForm()" onMouseOver="status='Perform Search'; return true" onMouseOut="status='';">

        tag. When you click the image, it its the HREF attribute that fires the jSubmitForm() sub from javascript.

        The "go" buttons are interchangeable because thet all call the same JS function.

        So we return to my previous post: if you get the linked javascript you'll find the source of the function:

        function JSubmitForm()
{
    document.Content.Name.value = document.Search1.Name.value;
    document.Content.CrdNumber.value = document.Search2.CrdNumber.valu
+e;        
    document.Content.SecNumber.value = document.Search3.SecNumber.valu
+e;
    
    if (eval("document.Search1.SearchType[0].checked") == true) 
    {
        document.Content.SearchType.value = 1;
    }
    if (eval("document.Search1.SearchType[1].checked") == true)
    {
        document.Content.SearchType.value = 2;
    }
    if (eval("document.Search1.SearchType[2].checked") == true)
    {
        document.Content.SearchType.value = 3;
    }
    
    if (eval("document.Search4.NumRows[0].selected") == true) 
    {
        document.Content.NumRows.value = document.Search4.NumRows[0].v
+alue;
    }
    if (eval("document.Search4.NumRows[1].selected") == true) 
    {
        document.Content.NumRows.value = document.Search4.NumRows[1].v
+alue;
    }
    if (eval("document.Search4.NumRows[2].selected") == true) 
    {
        document.Content.NumRows.value = document.Search4.NumRows[2].v
+alue;
    }
    if (eval("document.Search4.NumRows[3].selected") == true) 
    {
        document.Content.NumRows.value = document.Search4.NumRows[3].v
+alue;
    }
    
    document.Content.submit();
    return false;
}

        [download]

        As you can see, the function does nothing more than setting the hidden fields SearchType and NumRows to the appropriate values depending how the form was compiled and then submit the form.

        This means that you should be able to completely bypass the downloading of the form, and produce a simple GET with the correct parameters that retrieves directly the result you want.

        Rule One: Do not act incautiously when confronting a little bald wrinkly smiling man.

[reply]
[d/l]
[select]

      Okay, I followed the JavaScipt script that looked like it would be the right one and I think it is. Here is the content. You can see that it includes the JSubmitForm() function call that appears in the "go2" button (that isn't a button!?).

      browserName = navigator.appName;
browserVer = parseInt(navigator.appVersion);

if ((browserName == "Netscape" && browserVer >= 3) || 
        (browserName == "Microsoft Internet Explorer" && browserVer >=
+ 4))
{
    version = "n3";
}
else 
{
    version = "n2";
}

if (version == "n3")
{
    go_on                = new Image;
    go_on.src        = "/IAPD/Images/go_on.gif";
    go_off            = new Image;
    go_off.src    = "/IAPD/Images/go_off.gif"; 
    go2_on            = new Image;
    go2_on.src    = "/IAPD/Images/go_on.gif";
    go2_off            = new Image;
    go2_off.src    = "/IAPD/Images/go_off.gif";
    go3_on            = new Image;
    go3_on.src    = "/IAPD/Images/go_on.gif";
    go3_off            = new Image;
    go3_off.src    = "/IAPD/Images/go_off.gif";
}

/*********************************************************************
+*********
* Function: 
*  JImgAct
* 
* Purpose:
*  This function is called to activate an image.
*
**********************************************************************
+********/
function JImgAct(imgName)
{
    if (version == "n3") 
    {
        imgOn = eval(imgName + "_on.src");
        document [imgName].src = imgOn;
    }
}

/*********************************************************************
+*********
* Function: 
*  JImgInact
* 
* Purpose:
*  This function is called to inactivate an image.
*
**********************************************************************
+********/
function JImgInact(imgName)
{
    if (version == "n3")
    {
        imgOff = eval(imgName + "_off.src");
        document [imgName].src = imgOff;
    }
}

function JSubmitForm()
{
    document.Content.Name.value = document.Search1.Name.value;
    document.Content.CrdNumber.value = document.Search2.CrdNumber.valu
+e;        
    document.Content.SecNumber.value = document.Search3.SecNumber.valu
+e;
    
    if (eval("document.Search1.SearchType[0].checked") == true) 
    {
        document.Content.SearchType.value = 1;
    }
    if (eval("document.Search1.SearchType[1].checked") == true)
    {
        document.Content.SearchType.value = 2;
    }
    if (eval("document.Search1.SearchType[2].checked") == true)
    {
        document.Content.SearchType.value = 3;
    }
    
    if (eval("document.Search4.NumRows[0].selected") == true) 
    {
        document.Content.NumRows.value = document.Search4.NumRows[0].v
+alue;
    }
    if (eval("document.Search4.NumRows[1].selected") == true) 
    {
        document.Content.NumRows.value = document.Search4.NumRows[1].v
+alue;
    }
    if (eval("document.Search4.NumRows[2].selected") == true) 
    {
        document.Content.NumRows.value = document.Search4.NumRows[2].v
+alue;
    }
    if (eval("document.Search4.NumRows[3].selected") == true) 
    {
        document.Content.NumRows.value = document.Search4.NumRows[3].v
+alue;
    }
    
    document.Content.submit();
    return false;
}

      [download]

      With the JS code in hand, how do I attack my original problem of getting the form to submit? Thanks!

[reply]
[d/l]

        Well, when you click (any)one of the "go" buttons, the jSubmitForm function populate the hidden fields of this form:

        <form name="Content" method="Post">
          <input type="hidden" name="LinkPage" value="">
          <input type="hidden" name="PageType" value="Search">
          <input type="hidden" name="ORG_PK" value="000">
          <input type="hidden" name="STATE_CD" value="">
          <input type="hidden" name="Name" value="">
          <input type="hidden" name="SearchType" value="">
          <input type="hidden" name="CrdNumber" value="">
          <input type="hidden" name="SecNumber" value="">
          <input type="hidden" name="NumRows" value="">
          <input type="hidden" name="Save" value="IAPDSearch">
        </form>

        [download]

        and submit it to the site.

        All you have to do is to prepare a POST statement with the fields correctly populated and submit it directly.

        I advise you to see the CGI module, with that you can do it with little more than 4 lines of perl.

        Rule One: Do not act incautiously when confronting a little bald wrinkly smiling man.

[reply]
[d/l]

In Section Seekers of Perl Wisdom