scrolling in WWW:Mechanize::Firefox

msh210 has asked for the wisdom of the Perl Monks concerning the following question:

Venerable Monks,

Consider the page https://www.hy-vee.com/grocery/CT167/Pasta, if you please. You will see that a good portion of the page loads, and that, if you scroll to the bottom of the page, some more loads. Scroll to the bottom again, and even more loads. There must be some kind of JavaScript event triggered by scrolling to the bottom, though I seem to be inept at finding it using Firebug. But no matter: that's not my question.

My question is: How can I simulate such scrolling using WWW::Mechanize::Firefox? I wish to load more, and more, and more of the page before parsing it using the rest of my script.

$_="msh210";$"=$\;@_=@{[split//,uc]}[2,0];$_="@_$\1";$\=$/;++$_[0]for$...1;print lc substr crypt($_,"@_"),1,6

Comment on scrolling in WWW:Mechanize::Firefox Download Code

Replies are listed 'Best First'.
Re: scrolling in WWW:Mechanize::Firefox by Mr. Muskrat (Canon) on Jan 26, 2016 at 21:12 UTC
Check out WWW::Mechanize::Firefox and dynamic pages.	[reply]
Re^2: scrolling in WWW:Mechanize::Firefox by msh210 (Monk) on Jan 26, 2016 at 21:18 UTC
Yes, indeed. Thank you! `$_="msh210";$"=$\;@_=@{[split//,uc]}[2,0];$_="@_$\1";$\=$/;++$_[0]for$...1;print lc substr crypt($_,"@_"),1,6`	[reply] [d/l]
Re: scrolling in WWW:Mechanize::Firefox by thomas895 (Deacon) on Jan 26, 2016 at 21:18 UTC
Some browsers' developer tools provide a call trace for XHR requsts. If you're using Chrome or something Chromium-based, you have this. Other browsers might have it too. I tried your webpage, and here's what I got. If you try this in your browser, click on the line numbers and you'll be taken to where the function is defined in the JavaScript. It looks like the function you want is called `FilterProducts`. You may not even need to dig into the JavaScript too much. The developer tools also tell you which URL was loaded, as seen in my screenshot. Can you figure out how to manipulate that URL to make it do what you want? -Thomas "Excuse me for butting in, but I'm interrupt-driven..."	[reply]