Fellow Monks,

I am trying to learn about all things webby, in particular LWP::UserAgent and HTML::Parser. I need help though as I find that my attempts at meaningful functionality are being frustrated by Javascript. Before I explain my problem, I would like to state that I am starting with zero net knowledge eg I have never used cgi, coded javascript or designed a web page say. Ok, so let me explain what's happening with an example...

The Project

As a learning project, I decided to try and write some code that would perform the following actions:

1. log on to my net banking site,

2. locate my account balances and recent transactions,

3. reconcile these with my budget and personal records, and finally

4. send a chatty email to my wife based on the results thus proving to her that my interest in perl can lead to practical benefits.

Initial Problem

I haven't been able to get past step 1. Firstly, I learnt about lwp and html via the pods and some basic code to be able to get pages and extract html. The future seemed bright and clear. Then I tried to apply this knowledge to the project. The first thing I noticed was that the start page was javascript. No matter, I'll just scan the source and extract what I need.

I then plugged the refernced url into my code and got the following... 

META HTTP-EQUIV="Pragma" CONTENT="no-cache"
META HTTP-EQUIV="REFRESH" REFRESH: 0; CONTENT="0; URL=https://www.xxxb
+ank.com//main.htm"

[download]

Looking good... so after finding out about what https stands for I was ready. Alas, all I got for my trouble was a 500: Unknown Error error. Here is my modest code pretty much equal to the pod example. 


my $start_page = 'https://www.xxxbank.com//main.htm';

use LWP::UserAgent;
use HTTP::Cookies;
$ua = LWP::UserAgent->new;
$ua->agent("Mozilla/8.0"); # pretend we are very capable browser
$ua->cookie_jar(HTTP::Cookies->new(file => "lwpcookies.txt",
                                  autosave => 1));
$req = HTTP::Request->new(GET => $start_page);
$res = $ua->request($req);
print "Error: " . $res->status_line . "\n" unless $res->is_success;
print $res->content;

[download]

The Question

I'm not sure where to go now. I have tried going to the next stage of the login where I found form data that I put in a POST request but it led to the same error. My feel is that pages with lots of Javascript are inaccessible to a humble perl hacker and will end in tears. Is the wide world out there a Java one or can someone counsel perl perseverence and eventual enlightenment. I'm happy to do the hard yards of understanding html, cgi and so on but I get a chill in my spine thinking about hacking through javascript.

In reply to LWP and Javascript by tonyday

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.