jQuery Perl Interface

BrentD has asked for the wisdom of the Perl Monks concerning the following question:

I have a process I am trying to automate. Once a week, a third party sends me an e-mail with a randomly generated login and password to download a zip file of documents. I have managed to get the login working using HTTP::Cookies and LWP::UserAgent, but the file that is downloaded contains a bunch of JavaScript rather than links. The page apparently relies heavily on JQuery to generate the list of files. (Redacted image of interface available here: https://imagebin.ca/v/3pRrSI2qr3sH) The code that, I believe, populates the table is this snippet here:

<script type="text/javascript" src="/WebInterface/custom.js"></script>
    <script type="text/javascript">
        $(document).ready(function() {
            $('#filesContainer').fileTree({
                root: '/',
                overrideFromHash: true,
                expandSpeed: 1000,
                collapseSpeed: 1000,
                multiFolder: true,
                customData: true
            }, function(link) {
                window.open(link);
            });
[download]

I need to be able to somehow parse this page to get a list of the files that they have available for download, then download the file. Directory listing is turned off on the server, so I can't just log in and then do a list. Is there a way I can convert this into a URL I can send to the server to get the file list?

Comment on jQuery Perl Interface Download Code

Replies are listed 'Best First'.
Re: jQuery Perl Interface by Corion (Patriarch) on Jan 25, 2018 at 20:21 UTC
Use the network inspection tools of your browser to see what data actually gets sent or retrieved. From the looks of it, some data is passed to some `fileTree` plugin for jQuery, and most likely somewhere else in the HTML, that data is fetched or it is fetched when clicking on a link. As an alternative, you could try to use Perl to automate your browser, for example using WWW::Mechanize::Chrome or WWW::Mechanize::PhantomJS.	[reply] [d/l]
Re: jQuery Perl Interface by MorayJ (Beadle) on Jan 25, 2018 at 21:44 UTC
Yes, there's nothing to actually parse in there to show you the URL. The code needs to be executed and that could be done in a browser driven by perl as in the above answer. But following up on the inspection tools, I would try inspect element on the link when you log in manually and see what its path is - that path might be the same each time. Can the third party give the file the same name each time, or a predictable iteration? Just guessing though.	[reply]