On being an IE browser, revisited

rkg has asked for the wisdom of the Perl Monks concerning the following question:

Earlier I discussed the pros and cons of OLE + IE to do web things, vs. Mechanize or LWP.

Screen scraping is horrible, and clearly not the Right Way to access web resources. But sometimes screen scraping is necessary to get things done.

For serious screen scraping (yuck), I'm more convinced OLE + IE is the way to go. Often the pages I need to access involve Javascript, pop-up confirmation menus, file selection windows, and file download windows.

To my understanding, LWP and its derivitatives can't handle this sort of complexity. And to my understanding, samie doesn't yet handle popup dialog screens, upload screens, or file downloads.

In my opinion, there's a need for a solid Win32 app to drive IE in a serious way, to allow access to complex web environments involving non-vanilla pages. I asked a few months ago, but I'll toss out the question again:

Does anyone know of a robust full-featured module for driving IE?

Thanks for suggestions --

rkg

Comment on On being an IE browser, revisited

Replies are listed 'Best First'.

Re: On being an IE browser, revisited
by PodMaster (Abbot) on Feb 29, 2004 at 16:12 UTC

In my opinion, there's a need for a solid Win32 app to drive IE in a serious way...

to allow access to complex web environments involving non-vanilla pages

livehttpheaders

Currently a bunch of wxWigets (formerly wxWindows) programmers have created wxMozilla, a component for embedding Mozilla into any wxWindows application. A whole bunch of wxPerl programmers are interested in seeing this ported to wxPerl, it just might happen one day :)

Which brings me to what already exists, which is Wx::ActiveX::IE - ActiveX interface for Internet Explorer. Sound useful? I'd says yes. Slap together a little wxPerl program (perhaps using wxGlade), add some event handlers and you're scraping :)

I asked a few months ago, but I'll toss out the question again

Which brings me back to WWW::Mechanize. Take a look at (Javascript::SpiderMonkey)Re: Passing other variables to start handler in HTML::Parser and Re: Perl/Tk and Javascript. All JavaScript::SpiderMonkey needs is a Document Object Model. As soon as somebody writes one (hint), i'm sure somebody will marry it to WWW::Mechanize somehow :)

I hope that's enough ideas :)

update: I just ran accross the PerlXPCOM idea, but there doesn't appear to be any code, darn :(

Perl & Mozilla
PerlXPCOM
Perl scripting
      <script language=“perl”>print “Yay!” if /\w+/; </script>
[download]

MJD says "you can't just make shit up and expect the computer to know what you mean, retardo!"
I run a Win32 PPM repository for perl 5.6.x and 5.8.x -- I take requests (README).
** The third rule of perl club is a statement of fact: pod is sexy.

[reply]
[d/l]

Re: On being an IE browser, revisited
by grinder (Bishop) on Feb 28, 2004 at 22:08 UTC

Often the pages I need to access involve Javascript, pop-up confirmation menus, file selection windows, and file download windows.

Have you looked at WWW::Mechanize? If you need to understand the finer points of Javascript/frame/cookie/et cetera interaction, have you looked at HTTP::Proxy?

What do you really want to do today, drive IE, or download web resources? Aren't you getting the means and the end confused?

[reply]

Re: Re: On being an IE browser, revisited

by rkg (Hermit) on Feb 28, 2004 at 22:36 UTC

I like it for interacting with nice vanilla web pages. I don't know how to use Mechanize to handle websites which pop up additional confirmation dialog screens. I'd prefer Mechanize to OLE + IE, but I've unable to drive complex websites with Mechanize. (Could be the problem is me, not Mechanize -- I'm well aware of that!)

What I really want to is use automated intelligent perl scripts to interact with complex HTTP and HTTPS web apps designed for IE or NN. I'd rather not use OLE + IE -- using IE is a hack; IE has a big memory and processor footprint; IE is Win32-only; IE object docs are cryptic; etc. But (again maybe due to my lack of skill with LWP and Mechanize), OLE seems the most feasible option....

[reply]

Re: Re: Re: On being an IE browser, revisited

by flyingmoose (Priest) on Feb 28, 2004 at 23:52 UTC

I was going to suggest one of the GuiTest modules, but that doesn't cover the screenscraping requirement. You'd have to code in capturing to a file. In general, I don't like anything that can't be done cleanly with a command line as well as a GUI, so perhaps it's time to get that pure-Perl javascript support into Mechanize (just kidding! just kidding!).

[reply]


Clear questions and runnable code get the best and fastest answer
	PerlMonks