Yes it's hard but, but yes you can "scrape" any website with Mechanize (LWP) as long as you're smart enough (besides knowing the perl apis, you have to know how to learn what the browser is doing, and how it's talking to the server, then emulate -- this involves knowing everything from HTTP to CGI to HTML to Javascript).

In my opinion, there's a need for a solid Win32 app to drive IE in a serious way...

I kind of think Mozilla makes a stronger candidate to be the browser used "to allow access to complex web environments involving non-vanilla pages" because besides being open source and well documented, it's also cross platform. Mozilla already makes it easy to scrape websites with the livehttpheaders extension, and I'm sure some people have already automated Mozilla (so it appears). Hey, maybe you could write a plugin like livehttpheaders to talk with a perl program?

Currently a bunch of wxWigets (formerly wxWindows) programmers have created wxMozilla, a component for embedding Mozilla into any wxWindows application. A whole bunch of wxPerl programmers are interested in seeing this ported to wxPerl, it just might happen one day :)

Which brings me to what already exists, which is Wx::ActiveX::IE - ActiveX interface for Internet Explorer. Sound useful? I'd says yes. Slap together a little wxPerl program (perhaps using wxGlade), add some event handlers and you're scraping :)

I asked a few months ago, but I'll toss out the question again

If you can't find it on CPAN/Sourceforge/Freshmeat, or find it using GOOGLE, chances are it doesn't exist. I'll bet unless someone really interested (like you) takes up the task (hint hint), a few months from now it still won't exist.

Which brings me back to WWW::Mechanize. Take a look at (Javascript::SpiderMonkey)Re: Passing other variables to start handler in HTML::Parser and Re: Perl/Tk and Javascript. All JavaScript::SpiderMonkey needs is a Document Object Model. As soon as somebody writes one (hint), i'm sure somebody will marry it to WWW::Mechanize somehow :)

I hope that's enough ideas :)

update: I just ran accross the PerlXPCOM idea, but there doesn't appear to be any code, darn :(

Perl & Mozilla
PerlXPCOM
Perl scripting
      <script language=“perl”>print “Yay!” if /\w+/; </script>
[download]