Special_K has asked for the wisdom of the Perl Monks concerning the following question:

I'm trying to navigate Facebook and access my friends list programatically using WWW::Mechanize::Firefox. I have successfully logged in and navigated to the friends page, but the problem is the friends are not all loaded at once. A portion of them are loaded when the page first loads, and then to get more to load you have to scroll to the bottom of the screen. Getting all friends to load therefore entails repeatedly scrolling to the "new bottom" of the screen each time a new set of friends loads from the previous scroll action. I did some searching on here and saw the suggestion to send javascript through WWW::Mechanize::Firefox, and tried doing this on the friends page:

$mech->eval('window.scrollTo(0, document.body.scrollHeight)');

[download]

But that didn't work - instead of scrolling the page down, it took me back to the main page (i.e. the newsfeed). Can anyone give me more detailed information regarding how to programatically scroll the page to the bottom using WWW::Mechanize::Firefox? Thanks

Replies are listed 'Best First'.
Re: WWW::Mechanize::Firefox and dynamic pages
by Loops (Curate) on Jul 28, 2013 at 00:28 UTC
    Don't know if it will trigger the necessary loading you're looking for, but the code below will load a page and scroll down 188 lines. Once you have the window object you can use any method listed here. There are a handful of methods that allow you to scroll based on different parameters.

    My guess that if this is going to work as you hope, you'll have to scroll incrementally and not just pop to the bottom of the document.

    use WWW::Mechanize::Firefox;
my $mech = WWW::Mechanize::Firefox->new(tab => 'current');
$mech->get('http://search.cpan.org/~corion/WWW-Mechanize-Firefox/lib/W
+WW/Mechanize/Firefox.pm');
my ($window,$type) = $mech->eval('window');
$window->scrollByLines(188);

    [download]
[reply]
[d/l]

      Wow, I had no idea I could access all those functions. That's actually very helpful in general for some other things I was working on. I'll have to try those scroll functions out.

[reply]

        UPDATE: Using the scroll function makes the bar scroll, but then when I call:

        $mech->content()

        The friends' names aren't there, even though they are there on the page and I can find them using Firebug. How do I get all the data to load when I call mech->content()?

[reply]
[d/l]

Back to Seekers of Perl Wisdom