diablo1024 has asked for the wisdom of the Perl Monks concerning the following question:

Hello Monks,

I'm having some trouble working with forms in HTML. My project is to automate login to certain web pages and download information from them. I have done this successfully for one website. However, two other sites are giving me problems. Here is some sample code:

$login_url = url 'https://webbroker1.tdwaterhouse.com/TD/Waterhouse/ie
+4x/logonForm.asp';
$ua = LWP::UserAgent->new;
my $login_req = GET $login_url;
my $login_res = $ua->request($login_req);
my $tb = HTML::TreeBuilder->new;
$tb->parse($login_res->content);
my @forms = @{$tb->extract_links(qw(FORM))};
my $f = HTTP::Request::Form->new($forms[0][1], $login_url);
$f->dump;

[download]

If you visit the website given, you will see a login page where you type your Account Number and Password to gain access to the site. However, in the form that is downloaded (printed to STDIO with the dump command), the only fields that show up are a bunch of hidden fields, not the "UserID" and "Password", which are the two fields I need to fill out. I am aware of how to handle multiple forms per page, and this page contains only one form. This exact same code worked fine on another website (https://login.fidelity.com/ftgw/Fidelity/RtlCust/Login/Init) but on this one it does not work. Thank you very much for your help!

-Patrick

Replies are listed 'Best First'.
Re: trouble filling out HTML form
by Ovid (Cardinal) on Oct 20, 2003 at 21:57 UTC

    Try WWW::Mechanize. In looking at the form, you would use the following code:

    use WWW::Mechanize;
my $browser = WWW::Mechanize->new;

$browser->get($url);
$browser->submit_form(
        form_name => 'logon',
        fields    => {
            UserID   => $user,
            Password => $pass
        });

    [download]

    I noticed that their HTML is wrong. They forgot to specify the type for the UserID input box. I don't know if that will cause a problem.

    Cheers,
    Ovid

    New address of my CGI Course.

[reply]
[d/l]

      Note that with Ovids solution you *may* have to parse out and include some or all of the hidden fields in the form as well as adding the user/pass fields, otherwise the login may fail.

      Additionally the login STATE may be maintained by hidden fields in subsequent pages (you need to parse them out and add them into subsequent post login requests) or a SESSION COOKIE which you need to accept, save and pass with each post login request.

      They forgot to specify the type for the UserID input box

      You don't have to, the practical default is text

      If you think about the transaction there is no data passed from browser to server about the type of field so provided the browser will render it you have no issue in the real world.

      cheers

      tachyon

      s&&rsenoyhcatreve&&&s&n.+t&"$'$`$\"$\&"&ee&&y&srve&&d&&print

[reply]
        Thanks very much, tachyon and Ovid. Mech is in fact much better suited to this task than the old HTTP::Request and HTML::TreeBuilder modules I was using before. You perl monks are wise indeed.
[reply]

Back to Seekers of Perl Wisdom