See merlyn's Web scraping with WWW::Mechanize (Apr 03) column for a scraper which comes with a full explanation of its code.
--shmem
_($_=" "x(1<<5)."?\n".q·/)Oo. G°\ /
/\_¯/(q /
---------------------------- \__(m.====·.(_("always off the crowd"))."·
");sub _{s./.($e="'Itrs `mnsgdq Gdbj O`qkdq")=~y/"-y/#-z/;$e.e && print}
| [reply] |
| [reply] |
In addition to the links you have already been given, had you even bothered to search CPAN for the term 'webreaper' you would have found webreaper. I am sure it comes to no surprise to anyone that you did not bother to search yourself before asking here.
Update: I did not inform Win that 'webreaper' was a 'commonly used term'. This is a complete fabrication.
Martin | [reply] |
I think you're being unfair to the OP. "Webreaper" is not the single, universal standard term for this kind of app that you think it is. I'm no newbie, but if
I were looking for it, "webreaper" would never have occurred to me as a search
term. Terms I think of are "crawler" and "spider". But then,
I'm also aware that wget does something like what the OP wants, and
wouldn't have looked any further. :-) (OTOH, "reaper" does seem to be the most
likely term in the OP's vocabulary...)
A word spoken in Mind will reach its own level, in the objective world, by its own wei ght
| [reply] |
I agree 100%, which is why I said "In addition to the links you have already been given....". The point I was raising was that the OP has a history of posting here without searching (or in some cases even thinking) about the task in hand, and how they can solve it using the tools available to them (Perl, Modules, third party apps, and so on..).
I agree with you, the term 'webreaper' is not something that instantly springs to mind, and I find myself thinking of the terms you would search for also. Having said that, the OP did not bother to search for either the term they used 'web reaper' or any other, dare I say more common terms for this technology.
Thanks
Martin
| [reply] |