50-50 has asked for the wisdom of the Perl Monks concerning the following question:

Hi, I'm still pretty much a newbie in Perl. All I've done so far is automate my site using DBI and CGI. I'd like to learn a few more things at once, and I have a project idea that I think will go nicely with Perl.

I want to write a script that will log on to a specified website directory, and download all the files in that directory of a user specified type. Of course this will only work on websites that allow you to view those directories anyways. My main reason for wanting a script like this is to automate the dl of image files. So if a gaming site or something has a directory full of screen shots, you can just fire up your script, and have it download them all for you, instead of having to go to the directory, and manually save each one.

Can you guys point me in the right direction as to what modules are best suited for this? I was looking into IO::Socket::INET, IO::File, File::Listing, and Net::FTP for various options. I'm not familiar with these much at all, and would like some input on which ones would be better suited for the task, etc. I'd also like to have some kind of file xfer status displayed to the user, either in the console, or on a webpage if I make it web-based.

Thank you very much for any information you can give.

Replies are listed 'Best First'.
Re (tilly) 1: Learning Project
by tilly (Archbishop) on Oct 24, 2001 at 07:12 UTC
    Certainly LWP::* is what you want.

    However as you read up on the documentation, do not neglect the stuff about how to follow robots.txt. What you are building is a webspider, and to be a good netizen you need to pay attention to the rules and not slam websites that have asked you not to slam them...

Re: Learning Project
by Kanji (Parson) on Oct 24, 2001 at 06:36 UTC
Re: Learning Project
by 50-50 (Novice) on Oct 25, 2001 at 05:53 UTC
    Thanks for the information. I'll start reading up on the documentation. Thanks for the heads up on the webspider aspect also, I appreciate it.