SergioQ has asked for the wisdom of the Perl Monks concerning the following question:

I have an earlier post about web scraping images using Perl and the WWW::Mechanize family for web pages that get built using JavaScript after they are loaded. Obviously a lot of hurdles with are beyond my capabilities.

So I have a new thought but not sure it's possible, I'll describe what I believe may be a kludgy solution, which I'll gladly take at this point.

Please remember, I have my own Ubuntu server sitting somewhere, and I will run this script from any laptop with a web browser.

•Have my Perl script generate an HTML form of my own design, and then output it to my browser
•Make sure my Perl script does not exit here
•Have my Perl script generate the proper URL for the page I want to scrape
•Have my Perl script redirect to the URL from the above line and open that in either a new browser window or tab
•Now have my Perl script end

Then my idea is that I would go to the redirected window, manually "View Source", copy it, and post it to the form generated by the above Perl script.

If this kludgy idea was an entry in a cheese contest, it would not just be tagged the winner, but it would be so smelly that all the judges would run out while trying not to puke. However, is this possible?

  • Comment on Can my Perl CGI script open a new browser tab or window?

Replies are listed 'Best First'.
Re: Can my Perl CGI script open a new browser tab or window?
by marto (Cardinal) on Dec 16, 2019 at 11:51 UTC

    To call this kludgy is being kind ;) You should explain exactly what it is you want to do, I suspect you want to download posters from this site? Have you considered examining how other software gets it's movie posters? Have you searched https://metacpan.org to see if an API already exists? My previous reply also asked you to ensure the JavaScript did indeed do what you assumed, is this the case?

      From my limited understanding, yes the page is constructed by javascript more than html.

        And the other points I raised?

Re: Can my Perl CGI script open a new browser tab or window?
by Corion (Patriarch) on Dec 16, 2019 at 11:47 UTC

    Yes, this is possible, but sounds like a very roundabout way of going at it.

    See the target="_blank" attribute for links and (I guess) HTML forms.

    See WWW::Mechanize::Chrome for a module by me which allows web scraping and running of Javascript. It needs Google Chrome or Chromium installed though.

      Forgive my being rep repetitive, but I wanted to make sure: My Perl script is on one machine and the browser (in this case Chrome) would be on another machine. So wanted to make sure it works that way. Or does the Perl script have to be local Chrome machine??
Re: Can my Perl CGI script open a new browser tab or window?
by bliako (Abbot) on Dec 17, 2019 at 10:43 UTC

    It would be helpful perhaps to monitor network traffic via firefox's developer tools (other browsers have similar i guess). you may be lucky that eventually the convoluted javascript-based, ajax-based webpage leads you to an image url which is straightforward to reconstruct.

    Ideally, your scraping script should be as autonomous as possible, so you can be at the beach when it scrapes from a datacenter dungeon - scripts like this kind of hell. Improving your skills to this end, I would personally consider it Life's purpose.

Re: Can my Perl CGI script open a new browser tab or window?
by harangzsolt33 (Deacon) on Dec 16, 2019 at 14:10 UTC

    Sounds like you want to download the content of web pages. Have you tried wget? It does exactly that.

    But to answer your original question (how do I open new tabs in the web browser?), you could insert the following code into the output HTML:

    <A HREF="javascript:OpenMyWindows();">Open My Windows</A> <SCRIPT> MyURLS = [ "https://www.perlmonks.com", "https://www.foxnews.com", "https://duckduckgo.com", "https://www.yahoo.com" ]; function OpenMyWindows() { while (MyURLS.length) window.open(MyURLS.shift()); } </SCRIPT>

    ^ This will create a link, and when you click that link, several new tab will be created. BUT your web browser will block them, because they are popups. So, you have to click on Allow popups from your site.

    You could create a popup or new tab as soon as the browser loads the page without having to click anywhere, but the browser will again block your popup:

    <SCRIPT> window.open("https://www.yahoo.com"); </SCRIPT>