#!perl -T =head1 NAME wgetas - download many small files by HTTP, saving to filename of your choice =head1 SYNOPSIS B I =head1 DESCRIPTION Downloads a list of small files through HTTP. The URLs to download and their corresponding filenames are read from a text file. If a file to download to already exists, the corresponding download is skipped silently. This allows easy resuming an interrupted list of downloads. =head1 OPTIONS =over =item B<-i> I Gives a file that lists the URLs to download and filenames to save it: each line has a HTTP URL and a filename separated by whitespace. The filename cannot contain whitespace. =item B<-P> I Gives a directory to download files into: filenames in the listfile are relative to this. Default is the the working directory. =item B<-m> I Directory to put temporary files in. These temporary files then have to be renamed to the final location, so this directory must be on the same filesystem as where the downloads go. =item B<-U> I User-Agent header to send in the HTTP requests. =item B<-w> I Approximate number of seconds per requests to wait between requests. =item B<-e> I Gives a file that lists the files we have already tried to download. The list will be read on startup and any filenames found there will be excluded from downloading. On a successful downloads, as well as on download that has failed in a permanent way, the filename were we would download is written out in this file. =item B<-f> Continue downloading after some permanent HTTP errors. These currently include 404 Not Found and 403 Permission Denied. Allowed only if the B<-e> option is also set. =back =head1 NOTES The downloaded files must not be large, because they're read into memory, though this could be changed easily. The files are downloaded in bursts of 5 files, and we're waiting about 5 minutes (about 1 minute for each file) between bursts. This tool is not very general purpose: it does only what I currently need. =head1 BUGS Please report bugs to the author by email: L =cut