web spider

alokranjan has asked for the wisdom of the Perl Monks concerning the following question:

Replies are listed 'Best First'.
Re: web spider by 1nickt (Canon) on Nov 12, 2015 at 23:20 UTC
Yes, you can do anything in Perl that you want. You should start with perlintro and go from there. The first web programming you do, you'll want to look at LWP::UserAgent and friends. For spidering you'll probably want WWW::Mechanize. There are lots of articles and tutorials you can find on basic web programming with perl. Start reading and experimenting! Don't expect to finish in one day, and begin with the basics of Perl from the intros. Good luck! The way forward always starts with a minimal test.	[reply]
Re^2: web spider by Anonymous Monk on Nov 13, 2015 at 00:31 UTC
WWW::Mechanize is the friendlier version of LWP::UserAgent	[reply]
Re^3: web spider by 1nickt (Canon) on Nov 13, 2015 at 05:18 UTC
Agreed. But LWP is used all over the place and he's more likely to find it in his tutorials and books and classes. You'd be at a sore disadvantage if you didn't know some LWP in my opinion. But you may think me an old fart. The way forward always starts with a minimal test.	[reply]
Re^4: web spider by Anonymous Monk on Nov 13, 2015 at 08:53 UTC
Re^5: web spider by Your Mother (Archbishop) on Nov 13, 2015 at 16:13 UTC
Some notes below your chosen depth have not been shown here
Re: web spider by Discipulus (Canon) on Nov 13, 2015 at 10:11 UTC
yes there are so many option that is worth mentioning Web::Scraper too, the author is the brilliant father of the plack and PSGI. I had some understanding problem with that module, for sure for my limits but is fun and works. You can also be interested to a two parts article about modern perl web scraping and to general introduction on the matter. For my common usage i still use LWP::UserAgent. L* UPDATE 17 Nov 2015 minor typo fixed There are no rules, there are no thumbs.. Reinvent the wheel, then learn The Wheel; may be one day you reinvent one of THE WHEELS.	[reply] [d/l]
Re: web spider by hippo (Archbishop) on Nov 12, 2015 at 23:38 UTC
Is there any way to do the same in Perl. Yes: linklint.	[reply]
Re: web spider by stevieb (Canon) on Nov 12, 2015 at 23:17 UTC
There are many ways to do this with Perl. This isn't a code writing service though. Show us what you've tried, or at least what you've researched in your attempt.	[reply]