So, I've tried a couple modules, and am not having luck completely completing my task with either.

I'm trying to make a script that navigates pages, submits data to both html and javascript forms, and returns the html of current page.

I tried WWW::Mechanize, but it doesn't support javascript. Then I tried Win32::IE::Mechanize, but discovered something strange -

When navigating the site normally clicking a link on the menu would open the requested page on another frame on the browser. Doing so with WWW::Mechanize brought gave you direct access to the document. When I use win32::IE::Mechanize to access the menu frame, and then follow the links, the page just refreshes to the main page. I tried typing url's straight into the browser and got the same results.

I have to assume this has something to do with the fact that Win32::IE:Mech is using a browser, instead of making a more direct request to the server? Do you have other ideas of reasons this could happen? Can you think of any ways to make the script access the documents I need?

If not, can any of you suggest a web automation tool that can follow links, parse javascripts, return a pages html or url, and fill javascripts and html forms without using a browser?

Thanks for any help!

***EDIT***

Oh, sorry, here's some code:

#!/usr/bin/perl -w


use lib "C:/Program Files/Perl Express/Scripts";
use WWW::Mechanize;
use LWP::Simple;

package ClanBot;


#Takes a username and a password, creates a Mechanize object
#and uses the Mechanize object to log in to KOL. It returns the Mechan
+ize #object
#if login was successful, otherwise it diplays "Wait 60 seconds" and d
+ies.
#This sub works with either WWW::Mechanize or WWW::IE::Mechanize
sub login {
    my(@login) = @_;

        my $mech = WWW::Mechanize->new();
        $mech->get( "http://www5.kingdomofloathing.com" );

        $mech->submit_form(
        form_number => 1,
        fields => {
                loginname    => $login[0],
                password    => $login[1],
            }
        );

        $url = $mech->uri();

        if($url =~ m/game.php/i) {
            return $mech;
        }
        else {
            print "Wait 60 seconds";
            die;
        }

        return 0;
}




#Takes a reference to a mech object, calls back until it is at
#game.php, then navigates links to the clan_detailedroster.php
#This works with WWW::Mechanize, but whether I'm following links or us
+ing 
#->get(url) with Win32::IE::Mechanize it always refreshes to game.php 
+#immediately
sub toroster {
    $page = $_[0];

        $url = $$page->uri();

        while(!($$page->uri() =~ m/game.php/)) {
        $$page->back();
        }

        $$page->follow_link(n=>1);
        $$page->follow_link(n=>7);
        $$page->follow_link(n=>7);
        $$page->follow_link(n=>3);

        return 1;
}

[download]

Also here's an example of some javascript from the page I'd like to automate

<!--
function addlist() {
    which=0;
    if(document.getElementById("item11").innerHTML.length<1) which=11;
    if(document.getElementById("item10").innerHTML.length<1) which=10;
    if(document.getElementById("item9").innerHTML.length<1) which=9;
    if(document.getElementById("item8").innerHTML.length<1) which=8;
    if(document.getElementById("item7").innerHTML.length<1) which=7;
    if(document.getElementById("item6").innerHTML.length<1) which=6;
    if(document.getElementById("item5").innerHTML.length<1) which=5;
    if(document.getElementById("item4").innerHTML.length<1) which=4;
    if(document.getElementById("item3").innerHTML.length<1) which=3;
    if(document.getElementById("item2").innerHTML.length<1) which=2;
    if (which>0) {
        var text="<b>Send: <input type=text size=2 value=1 name=howman
+y"+which+"> </b>";
        text=text+"<select name=whichitem"+which+"><option value=0>-se
+lect an item-</option><option value=71>wooden stakes (1)</option></se
+lect>";
        document.getElementById("item"+which).innerHTML=text;
    } else {
        alert('Eleven is enough.');
    }
}

[download]

It adds another drop down menu to a page that allows you to send items to people. Win32::IE::Mechanize could use this page, if it would navigate to it, which it wont. WWW::Mechanize couldnt use it, but it can navigate there.

In reply to Suggest a suitable module? by jdetloff

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.