Mechanize, Using WWW instead of IE

Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

I would like to avoid having to depend on IE for my utility I'm writing but unfortunately I can't figure out how to do the same with the WWW::Mechanize module. Does anyone know how to make the following code work under WWW::Mechanize? I've read that it doesn't support javascript so I'm not sure if it's possible or not...but with Perl I suspect it is possible somehow :)

use warnings;
use strict;

use Win32::IE::Mechanize;
my $ie;

my $id = ' 14632';
my $max_page = 9;

$ie = Win32::IE::Mechanize->new( visible => 0 );

my $url = 'http://myspace.com/Modules/ViewFriends/FriendsView.aspx?%3f
+fuseacion=user.viewfriends&friendID=' . $id;

$ie->get( $url );

sleep 1;

my $count = 0;
my $i = 0;
    
for (1..$max_page) {
    scrape_page($_, $max_page);
}

sub scrape_page {
    
    my ($page, $max) = @_;

    print "Looking at $page ...\n";

    if ($page > 1 ) {

    $ie->follow_link( url => 'javascript:__doPostBack(\'ctl00$cpMain$p
+agerBottom\',\'' . $page . '\')' );

    }

    sleep 1;

    my @data = split(/\n/, $ie->content());
    foreach (@data) {
    chomp;
    if (/<A href=\"http:\/\/profile\.myspace\.com\/index\.cfm\?fuseact
+ion=user.viewprofile&amp;friendid=(.*?)\">(.*?)<\/A><BR>/i) {
        print "ID: $1\n";
    }   
    }
}
[download]

Thanks

Comment on Mechanize, Using WWW instead of IE Download Code

Replies are listed 'Best First'.
Re: Mechanize, Using WWW instead of IE by Cody Pendant (Prior) on Jan 28, 2007 at 09:06 UTC
Short answer, without knowing what the javascript in question does -- whatever it does, it's communicating from your browser to the server. So get the LiveHTTPHeaders extension for Firefox, use that to figure out what it's actually doing, and use WWW::Mechanize to reproduce that. ($_='kkvvttuu bbooppuuiiffss qqffssmm iibbddllffss') =~y~b-v~a-z~s; print	[reply]
Re: Mechanize, Using WWW instead of IE by dorko (Prior) on Jan 28, 2007 at 05:21 UTC
`$ie->follow_link( url => 'javascript:__doPostBack(\'ctl00$cpMain$pagerBottom\',\'' . $page . '\')' );` You're right in that link is Javascript based and WWW::Mechanize can't handle Javascript. In fact, I've seen Win32::IE::Mechanize suggested for use when you're trying to deal with web pages that rely on Javascript (like yours does). Because Win32::IE::Mechanize uses Internet Explorer, it can handle Javascript. Cheers, Brent -- Yeah, I'm a Delt.	[reply] [d/l]
Re^2: Mechanize, Using WWW instead of IE by bart (Canon) on Jan 29, 2007 at 12:09 UTC
In a very recent node, Joost states that another contender; Mozilla::Mechanize, can also work with Javascript, but is based on the Gecko rendering engine instead — the core of Firefox and related browsers. The OP could make his script less dependant on MSIE, or on Windows, by allowing the user to choose between these two modules, and use whichever one is available. Judging by the docs of the latter, that appear to be largely copied from Win32::IE::Mechanize, so they claim themselves; they ought to be largely compatible. (Caveat: untested)	[reply]
Re: Mechanize, Using WWW instead of IE by andyford (Curate) on Jan 28, 2007 at 12:19 UTC
Although the module is currently having problems due to recent changes at http://myspace.com, WWW::Myspace will help you. Update: New version (0.62) should fix problems with the version I talked about above. non-Perl: Andy Ford	[reply]
Re: Mechanize, Using WWW instead of IE by mkirank (Chaplain) on Jan 28, 2007 at 15:50 UTC
Have you looked at Test::WWW::Selenium	[reply]
Re^2: Mechanize, Using WWW instead of IE by andyford (Curate) on Jan 29, 2007 at 00:35 UTC
I tried that with MySpace, but Selenium got really confused in flurry of weird frames and domain changes. Have you got it working with MySpace? non-Perl: Andy Ford	[reply]
Re^3: Mechanize, Using WWW instead of IE by mkirank (Chaplain) on Jan 30, 2007 at 02:26 UTC
I have not tried it with Myspace but we have an internal application that uses frames, to get selenium to work with frames we start selenium with the -multiWindow option.	[reply]