vijneedshelp has asked for the wisdom of the Perl Monks concerning the following question:

Revered monks, I desperately need your advice on how I should scrape from pages with lots of javascript created HTML. So far I have learnt the ways of Mechanize::Firefox and its interaction with MozRepl. The trouble is that I need it to run in a headless environment. I could run Firefox under xvfb, but I don't know how to install the MozRepl extention in a headless environment. It also seem very heavy weight to run the entire Firefox application. Do you have any advice on a simpler method that would allow me to get the rendered HTML of a page? Or should I just suck it up and use MozRepl and figure out how to get it working in a headless environment. Thank you for your wisdom

Replies are listed 'Best First'.
Re: Suggestions for headless crawling
by Corion (Patriarch) on Oct 24, 2011 at 07:06 UTC

    There is WWW::Scripter, which has a Javascript plugin.

    For how to operate Firefox without a terminal, have you considered setting up Firefox with an X terminal and then launching it with the xvfb X terminal? Your system administrator can advise you on how to switch X terminals.

[reply]
      Thanks for your advice Corion!
[reply]

Back to Seekers of Perl Wisdom