in reply to LWP and Javascript

Do you just want to look at the new stuff DOM nodes created by the scripts or do you want to control the whole process with perl. If you just want to look (and mess about by hand) check out the firebug firefox extension.

If you want perl control, check out Mozilla::Mechanize::GUITester and/or Mozilla::Mechanize or WWW::Selenium. update2: Win32::IE::Mechanize

update: just to be clear: LWP does not support javascript at all, and as far as I know no-one is planning on adding javascript support. All options I linked drive conventional GUI browsers since that's a lot easier to build and support than creating your own compliant and effectively useful DOM/BOM with javascript support. I should know - I gave it a try once :-)

"What should it profit a man, if he should win a flame war, yet lose his cool?"

Replies are listed 'Best First'.
Re^2: LWP and Javascript
by sailortailorson (Scribe) on Mar 29, 2007 at 22:44 UTC
    Joost,

    Thanks. This all looks like the kind of things I need to add to a general toolbox for making webpages jump through hoops, mostly for automating testing, and computer aided manual testing. The specific task I wanted to solve is counting sponsored links (ads) on a web page. I cannot see them in LWP, only the Ajax that goes and gets them.

    For some reason, I have not even tried the mechanize modules. I guess I thought I was getting more control from LWP. I just heard about Selenium last week for the first time, so I will take a look at that too.

    In regard to your note about LWP, that confirms the sense I had about it. So I know now not to bark up that tree anymore.

    Thanks
[reply]
      WWW::Mechanize is a subclass of LWP::UserAgent, so more or less anything you can do with LWP::UA you can do with WWW::Mechanize. Mechanize is just easier to use when automating click-trails (start at link such-and-such, click link named "bla bla", fill in form field "thingy", submit - etc)

      The *::Mechanize::* modules linked in my post above mimic and extend WWW::Mechanize's behaviour but are probably not 100% API compatible - YMMV.

      "What should it profit a man, if he should win a flame war, yet lose his cool?"
[reply]
        I ported Mozilla::Mechanize from Win32::IE::Mechanize, which was in turn ported from WWW::Mechanize. You're right they're not 100% compatible, but mostly they are. I've meant to get a new release out of Mozilla::DOM and Mozilla::Mechanize for a long time, especially since Boris's Mozilla::* modules came out, but I have a little problem where all Gtk2::MozEmbed-based applications segfault for me on Ubuntu, so I'm not able to confirm that things work. (FWIW, I have seen https://launchpad.net/ubuntu/+source/firefox/+bug/26436 and I have set LD_LIBRARY_PATH and /etc/ld.so.conf and done /sbin/ldconfig, but things still segfault for me.)
[reply]

In Section Seekers of Perl Wisdom