Downloading a lot of files

Have a list of RPMS or Debs? Stick them in a file, one on each line, and run this. I use it when upgrading between Fedora distros, if any RPMS have been missed according to upgrade.log

#!/usr/bin/perl -w
#
# Read in upgrade.log and get missing RPMS 
#
# GH - 22.06.05
#
use strict;

use LWP::Simple;

open(RPMS, "<", "./upgrade.log")
    or die "File not found or unreadable: $!\n";

my $url = 'http://download.fedora.redhat.com/pub/fedora/linux/core/4/i
+386/os/Fedora/RPMS/';

while (<RPMS>) {
    chomp;
    print -e $_ ? "Already downloaded $_\n" : "Downloading $_\n";
    next if (-e $_);
    
    getstore ($url . $_, $_);
  
    print -e $_ ? "Got $_\n" : "Download failed for $_\n";

}
close(RPMS);

print "Download complete.\n";
[download]

Comment on Downloading a lot of files Download Code

Replies are listed 'Best First'.
Re: Downloading a lot of files by ihb (Deacon) on Jun 23, 2005 at 23:47 UTC
`&mirror` could be handy here. `use strict; use LWP::Simple; use File::Slurp qw/ read_file /; my $upgrade_file = './upgrade.log'; my $url = 'http://download.fedora.redhat.com/pub/fedora/linux/core/4/i +386/os/Fedora/RPMS/'; for (read_file($upgrade_file)) { chomp; print "Checking $_... "; my $status = mirror($url . $_, $_); print $status == RC_OK ? 'downloaded' : $status == RC_NOT_MODIFIED ? 'already downloaded' : "error: $status"; print "\n"; } print "Download complete.\n";` [download] `ihb` See perltoc if you don't know which perldoc to read!	[reply] [d/l] [select]
Re: Downloading a lot of files by davidrw (Prior) on Jun 24, 2005 at 01:40 UTC
(since TWTOWTDI can include shell utils ;) ) bash syntax (though less error handling, which could partially be fixed with a `test` of the filename and of `$?`; could also tweak wget's cmdline options, or use curl instead of wget, etc): url='http://download.fedora.redhat.com/pub/fedora/linux/core/4/i386/os +/Fedora/RPMS/' for f in `cat upgrade.log`; do wget $url$f; done [download] But obviously LWP::Simple is cooler. :)	[reply] [d/l] [select]
Re^2: Downloading a lot of files by Corion (Patriarch) on Jun 24, 2005 at 06:57 UTC
That way you lose the feature for persistent connections of wget. I use `wget -i upgrade.log` if I have a file or `open WGET, "\| wget -i -"` if I don't know the list of URLs beforehand.	[reply] [d/l] [select]
Re: Downloading a lot of files by fmerges (Chaplain) on Jun 24, 2005 at 12:37 UTC
Hi, I know this is about Perl...but, why you don use tools like rsync (http://www.rsync.org) or apt-get. Apt-get is wonderfull tool, there're also other tools like apt-get. I use rsync for so many tasks, like file syncs, mirroring, etc Another really interesting tool is Unison (http://www.cis.upenn.edu/~bcpierce/unison/), for syncronise host in a two directions way. Hope it can be useful. Regards :)	[reply]