Navigating multiple layers webpage

ttran18 has asked for the wisdom of the Perl Monks concerning the following question:

I have a webpage that requires a login upon loading. After entering your credentials, you are taken to a page where there is a search box I want to fill out, as the search box is filled out, there are suggestions that start showing up, you'll have to pick from the dropdown list (it's an input box, I'm guessing javascript is populating a list you can choose from). I want the result of the search box.

In short:

Log in
Enter data into search box
Pick from suggested results
Store results in variable

Right now I just use the WWW::Mechanize module and have the login part done. I do not know how to access the next page where I can enter data into a search field. I was thinking I could do something with $results.

Please advise.

my $mech = WWW::Mechanize->new();
$mech->get('http://loginpage/Account/LogOn');
my $result = $mech->submit_form(
    form_number =>  '1',
    fields => {
        'UserName'  => '$username',
        'Password'  => "$pw",
    },
);
[download]

Comment on Navigating multiple layers webpage Download Code

Replies are listed 'Best First'.
Re: Navigating multiple layers webpage by choroba (Cardinal) on Sep 04, 2015 at 23:34 UTC
WWW::Mechanize works like a browser. After submitting a form (unless JavaScript is involved which isn't supported) the $mech should "display" the next page. If you need to run JS, look into WWW::Mechanize::Firefox or WWW::Selenium etc. لսႽ† ᥲᥒ⚪⟊Ⴙᘓᖇ Ꮅᘓᖇ⎱ Ⴙᥲ𝇋ƙᘓᖇ	[reply]
Re: Navigating multiple layers webpage by ww (Archbishop) on Sep 04, 2015 at 21:44 UTC
... and I want a pony. (I don't suppose you're going to give me one, are you?) What have you tried? Have you read the module's documentation carefully enough to have any ideas beyond cut'n'pasting a really basic piece of code? Have you searched here for similar questions and answers -- especially about sites that us js? PM is NOT a code writing service. check Ln42! -->	[reply]
Re^2: Navigating multiple layers webpage by ttran18 (Initiate) on Sep 04, 2015 at 22:52 UTC
I've tried manipulating the return from the submit_form function with no success. I've googled multiple sites, many of which have their own different purpose and niches. I've tried more than just copying and pasting code, I didn't think it was worth posting all of my failed attempts. I've read the CPAN page and thought perhaps I can use click vs submit, or follow link methods, but I am stuck on whether I do that with the $mech variable or the $results, both of which I have tried. I don't want all the code written for me, just hints on a direction to go. And yes, I've tried looking on pm as well.	[reply]
Re: Navigating multiple layers webpage by Anonymous Monk on Sep 04, 2015 at 23:35 UTC
See https://metacpan.org/pod/WWW::Mechanize::FAQ#JavaScript, this is the beginning	[reply]
Re: Navigating multiple layers webpage by locked_user sundialsvc4 (Abbot) on Sep 05, 2015 at 16:04 UTC
To try to give you a little more direction on this ... in the aforementioned perldoc, there are two things to zero in on: (1) WWW::Mechanize doesn’t support JavaScript, and (2) JavaScript is critical to what this web page is doing. The page is sending AJAX requests to a host, and, based on the results obtained, it (JavaScript) is populating the box. Furthermore, only it knows how to do it, and when it has finished. If you simply used UserAgent to “fetch the page,” it would be kinda like exhuming a lifeless corpse. You would have the HTML and maybe even all the JavaScript, but no way to bring the thing to life, and unless it is “alive,” it’s useless to you. At the bottom of the perldoc section is a paragraph entitled, Which modules work like Mechanize and have JavaScript support?“ The one I’m most familiar with is WWW::Selenium, which actually is an interface to the Selenium web-testing framework. Selenium spawns an actual web-browser process (of your choice ...) and uses it to access the site. The browser is controlled and monitored by Selenium, but it is an actual browser and therefore able to execute scripts that are part of the page. Perl, in turn, is able to control Selenium.
Re^2: Navigating multiple layers webpage by ttran18 (Initiate) on Sep 20, 2015 at 03:28 UTC
Thanks for the lead! This sounds like what I'm looking for. Though I'll be making hundreds of calls, hopefully the load won't be too heavy with all the openning and closing of browsers.	[reply]