Advatages of LWP?

ChrisJ-UK has asked for the wisdom of the Perl Monks concerning the following question:

Hi,

I am still building my little crawler.

At the moment I retrieve each page using the Lynx browser:

sub fetch_a_page {
    my($fetch_this)=@_;      # Holds URL of page to be fetched
    my($fetched_page);       # Holds contents of fetched page
    $fetched_page=`lynx -source $fetch_this`;
    return $fetched_page;
}
[download]

I am aware of the LWP module being able to fetch web pages. Is there any actual advantage to be had from using LWP rather than Lynx?

If I shifted the script from the current Unix server to Windows then I would have to include LWP but I don't forsee that ever happening.

Thanks.

Comment on Advatages of LWP? Download Code

Replies are listed 'Best First'.
Re: Advatages of LWP? by davorg (Chancellor) on Jul 28, 2004 at 13:06 UTC
Some that spring immediately to mind: LWP is a standard part of Perl (since 5.8.0) therefore it will always be installed whereas lynx is something extra to install. Using internal code is always safer and faster than shelling out to an external program. LWP can be used to give you more information - like the HTTP response header - which might be useful in the future. -- <http://www.dave.org.uk> "The first rule of Perl club is you do not talk about Perl club." -- Chip Salzenberg	[reply]
Re^2: Advatages of LWP? by hardburn (Abbot) on Jul 28, 2004 at 13:17 UTC
LWP is a standard part of Perl Hrm? I don't see it. The standard dist needs to get smaller, anyway. Although very useful, LWP is not such a critical peice of functionality that it needs to be in the core. ---- send money to your kernel via the boot loader.. This and more wisdom available from Markov Hardburn.	[reply]
Re^3: Advatages of LWP? by davorg (Chancellor) on Jul 28, 2004 at 13:24 UTC
Er.... no. It was libnet that was added, not libwww. Sorry about that. -- <http://www.dave.org.uk> "The first rule of Perl club is you do not talk about Perl club." -- Chip Salzenberg	[reply]
Re: Advatages of LWP? by hardburn (Abbot) on Jul 28, 2004 at 13:13 UTC
Yes: Calling external programs is always a security risk, especially without setting `$ENV{PATH}`. (It's up to you to balance this with other concerns). LWP will let you get at the HTTP headers It's easier to handle errors in LWP If you're willing to take the security risk, and don't need to do anything with HTTP headers, then you're still stuck with error handling. What if the page isn't available? What if your connection is down? What happens if Lynx isn't available? You mention porting to Windows. Lynx is just as unlikely to be there as LWP. ---- send money to your kernel via the boot loader.. This and more wisdom available from Markov Hardburn.	[reply] [d/l]
Re: Advatages of LWP? by nite_man (Deacon) on Jul 28, 2004 at 13:49 UTC
Sure, you can use external programms directly in many cases instead of using modules. But in my mind, using of modules is better because there are many things which already done: error catching, additional functionality some validations etc and many people test they functionality. In your case, you should do the same things (testing, error catching etc) in your application. So, no reasong do the same work. --- Schiller It's only my opinion and it doesn't have pretensions of absoluteness!	[reply]
Re: Advatages of LWP? by iburrell (Chaplain) on Jul 28, 2004 at 16:27 UTC
It is more efficient to use the Perl modules than the external programs. LWP requires loading some Perl modules. But it saves forking and running another program. For a single page, it probably doesn't matter. For multiple pages, the difference could be huge.	[reply]
Re: Advatages of LWP? by Aristotle (Chancellor) on Jul 28, 2004 at 16:27 UTC
My reason would be that `lynx` is really slow. If any external tool, I'd use `wget -qO -` to pull pages (or maybe `curl` if you have that; it's less common than `wget`). That said, I can't think of any particular reason in favour of using an external utility. I'd just `use LWP::Simple;` and call `get()`, which is one extra line to pull in the module and otherwise (at least) as simple. A reason against using backticks in general is security; in your case, if `$fetch_this` ever contains input from untrusted users. Makeshifts last the longest.	[reply] [d/l]