I was recently perusing my (sparse!) posting history on this site and had some fond memories triggered by this script I had written almost ten years ago. In the ten years since then screen scraping and web automation have made some great advances! WWW::Mechanize was a huge advance by providing such a programmer friendly API. More recently WWW:Scripter, in conjunction with plugin scripting engines, has essentially given Perl programmers a pure Perl "virtual browser" in which to data mine even the most horrendously JavaScript laden site! Just today I wrote the following code to download quotes for a portfolio of stocks on Google Finance. Now, I didn't really need to use WWW::Scripter. I could have used WWW::Mechanize to achieve the same thing, of course, but I wanted to experiment with WWW::Scripter a little. Having my script identify itself as Safari and the random sleeps between actions are just small ways to act more like a real human user logging in and clicking through. Again, no real reason other than to just have some fun. The fact remains, though, that if I streamlined this script I could have accomplished the act of logging into Google and clicking through to download the quotes in about a dozen lines of code! Furthermore, if Google decided to check to see if they were getting hit by a "real browser" with JavaScript checks a Perl programmer could still get the job done! 
#download_quotes.pl
#Downloads portfolio quotes in .csv format from Google Finance.
#Adam Russell(ac.russell@live.com)
#27 February 2011
use strict;
use warnings;
use WWW::Scripter;

my $w = new WWW::Scripter;
$w->agent_alias("Mac Safari");
 
$w->use_plugin("JavaScript");
$w->get("https://www.google.com/accounts/Login");
sleep_rand();
$w->form_name("gaia_loginform");
$w->field("Email",GOOGLE_LOGIN);
$w->field("Passwd", GOOGLE_PASSWORD);
$w->current_form->trigger_event("submit");
sleep_rand();
$w->get("http://www.google.com/finance");
sleep_rand();
$w->follow_link(id=>"nav-p");
sleep_rand();
$w->follow_link(id=>"nav-pf");
sleep_rand();
$w->follow_link(id=>"download");

print $w->content;

sub sleep_rand{
    sleep(rand()*3);
}

[download]
<jc> Why do people persist in asking me stupid questions?
<Petruchio> <insert mutually recursive response>
--an exchange from #perlmonks on irc.slashnet.org(2 March 2009 1345 EST)

In reply to Thinking about advances in web automation by adamcrussell

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.