From bash to perl

paranoid times has asked for the wisdom of the Perl Monks concerning the following question:

I've got a script which is using some bash commands to suppliment my perl. I would like to turn the whole script into a perl script to make it a little more efficent, and to learn some more about perl. Here are the things that are going on in particular. I have the script take in a URL at the command line. Then I use wget to download the file at the end of the URL (a web page) and write it to a file, then I open the file and read through it with a while loop one line at a time. I went hunting around for a way for perl to download the webpage, but I haven't found anything. (My research abilities aren't that great with perl, I mostly just try google searches. I don't really know how CPAN works, and I know there is some manual that can be searched, but I don't know what it is called or how to search it. If anyone has pointers on that, I would be most greatful for the information). If I can get the webpage downloaded with perl, then I expect I could just avoid writing to a file, and then would be able to have an array read the whole thing in (If I recall from a previous script I made the array will default to breaking the elements into every new line.) So basically I'm wondering how to replace wget with a perl solution (I have a bunch of other questions, but I would rather beat on them myself for at least awhile, as I will learn more that way.) Thanks, Michael

Comment on From bash to perl

Replies are listed 'Best First'.
Re: From bash to perl by BUU (Prior) on Nov 08, 2006 at 01:57 UTC
You probably want LWP::Simple from the LWP distribution. LWP::Simple will export a function named "get" that takes a URI as an argument and returns the contents of that URI. As for modules in general, you should always be searching cpan at http://search.cpan.org.	[reply]
Re^2: From bash to perl by paranoid times (Acolyte) on Nov 08, 2006 at 02:51 UTC
Thanks I'll check into that. I've actually used that function once before but it was for sending data to a URI. I didn't know that it could be used the other way around, nifty.	[reply]
Re: From bash to perl by GrandFather (Saint) on Nov 08, 2006 at 02:05 UTC
You probably want to start with LWP to solve your ultimate problem, but take a trawl through the Tutorials section here first to get some of the background info. In particular yuou may find the Installing Modules section helpful. Perl has pretty good documentation available using `perldoc docname`. perl (`perldoc perl`) provides a list of the documents available. DWIM is Perl's answer to Gödel	[reply] [d/l] [select]
Re^2: From bash to perl by paranoid times (Acolyte) on Nov 08, 2006 at 02:54 UTC
I was poking around the tutorials a little earlier today. I think I was there with the wrong mind set though, as I was looking for an answer, more then information. I should probably take another look, just looking for more general stuff. I'll check into perldoc as well. Thank you.	[reply]
Re: From bash to perl by jwkrahn (Abbot) on Nov 08, 2006 at 02:06 UTC
You can use the LWP::Simple module to retrieve web pages. Also the libwww-perl cookbook has more details.	[reply]
Re^2: From bash to perl by paranoid times (Acolyte) on Nov 08, 2006 at 02:58 UTC
Thank you for the cookbook link.	[reply]
Re^3: From bash to perl by helgi (Hermit) on Nov 08, 2006 at 09:25 UTC
You can also run it on the command line on any computer that has a working perl installation: `[~]$ perldoc lwpcook` [download] or `C:\>perldoc lwpcook` [download] -- Regards, Helgi Briem hbriem AT f-prot DOT com	[reply] [d/l] [select]
Re: From bash to perl by graff (Chancellor) on Nov 08, 2006 at 03:00 UTC
I use wget to download the file at the end of the URL (a web page) and write it to a file, then I open the file and read through it with a while loop one line at a time. Next time, show us some of your code -- it's just better that way. So you'll start using LWP or LWP::Simple (and I'm not surprised you didn't find them... I still have trouble remembering how these module names relate to their function, let alone what "LWP" actually stands for). After that, the next thing you'll want to get acquainted with is the more appropriately named HTML::Parser. This will help tremendously in doing a proper job of whatever it is you are trying to do with the web pages that you are downloading -- and you won't even need to store the html data to a file first (though you probably want to do that anyway), because HTML::Parser can work directly on the scalar variable that holds the full content of the page (after LWP assigns the fetched web-page content to that variable). Good luck.	[reply]
Re^2: From bash to perl by Anonymous Monk on Nov 08, 2006 at 09:35 UTC
I still have trouble remembering how these module names relate to their function, let alone what "LWP" actually stands for LWP stands for `libwww-perl`, the second word mentioned in the DESCRIPTION section of the LWP manual page.	[reply]
Re: From bash to perl by fenLisesi (Priest) on Nov 08, 2006 at 07:47 UTC
WWW::Mechanize may help. Cheers.	[reply]
Re: From bash to perl by Khen1950fx (Canon) on Nov 08, 2006 at 06:00 UTC
I think that WWW-PkgFind might help you. To quote the docs: "package_retriever scans a web or ftp site for newly posted files and downloads them to a local filesystem. It then prints out the file names of the files it downloads to stdout, suitable for passing to other tools in a pipe". Also, it'll create subdirectories; it uses regular expressions to find "wanted" or "unwanted" files; it can rename packages after download, etc.	[reply]