Fetching web pages via Perl

mrguy123 has asked for the wisdom of the Perl Monks concerning the following question:

Hi Monks,
I am working on a tool that fetches results from different search engines and parses the results for display.
In the past I have mostly worked with the LWP::UserAgent module, and would like to also try some different modules as well.
I have heard good things about WWW:Mechanize and would like to hear any other recommendations or advice about similar modules that do a good job
Any suggestions?

Thanks,
Guy (mrguy123)

Unix is user friendly - it's just a bit more choosy about who it's friends are -- Gene Buckle

Comment on Fetching web pages via Perl

Replies are listed 'Best First'.
Re: Fetching web pages via Perll by Corion (Patriarch) on Oct 06, 2011 at 09:43 UTC
If you are targetting search engines, maybe WWW::Search may or may not be of interest. Other than that, there is little that beats the API of WWW::Mechanize. WWW::Mechanize has everything LWP::UserAgent has and a bit more. There is WWW::Scripter, a descendant of WWW::Mechanize. The Mechanize namespace is where almost all modules for web navigation live.	[reply]
Re: Fetching web pages via Perl by moritz (Cardinal) on Oct 06, 2011 at 10:03 UTC
I've made good experiences with Mojo::UserAgent. Perl 6 - second systems done right	[reply]
Re^2: Fetching web pages via Perll by mrguy123 (Hermit) on Oct 06, 2011 at 10:31 UTC
Thanks for the recommendations	[reply]