I found this useful to populate a mirror of centos without downloading the entire repository, but only the files I have needed on any machine of my network. Of course you still have to syncronize repodata by yourself often in a while. But once you do this, if there is no local copy of a package, it will be requested to the upstream.
Add
proxy=http://localmirror:8080to your /etc/yum.conf. Create the mirror root
mkdir -p /opt/mirror_roor/centos
ln -s /var/www/html/centos/ /opt/mirror_root/centos
perl automirror.pl -l http://localhost/ -e http://mirror.centos.org/ --root /opt/mirror_root/And you are done.
I had a lot of fun writing this.
Critics very welcome, enjoy it.
#!/usr/bin/perl use warnings; use strict; use HTTP::Proxy qw(:log); use HTTP::Proxy::HeaderFilter::simple; use HTTP::Proxy::BodyFilter::save; use Getopt::Long; use Pod::Usage; use LWP::UserAgent; use URI; my %mirror = (); my $mirror_root; my $verbose = 1; my $port_to_listen; my $interface_address_or_hostname; Getopt::Long::Configure ('bundling'); GetOptions( 'verbose|v+' => \$verbose, 'port|p=i' => \$port_to_listen, 'I=s' => \$interface_address_or_hostname, 'local|l=s' => \$mirror{'local'}, 'external|e=s' => \$mirror{'external'}, 'root|r=s' => \$mirror_root, 'help|h|?' => pod2usage(1), ) or pod2usage(1); pod2usage(1) unless defined $mirror{'local'} and $mirror{'external'} and $mirro +r_root; my $proxy = HTTP::Proxy->new( port => $port_to_listen || 8080, host => $interface_address_or_hostname || 'localhost.localdomain', logmask => $verbose > 2 ? ALL : $verbose > 1 ? STATUS : FILTERS, engine => 'ScoreBoard', #Forking engine. ); my $ua = LWP::UserAgent->new(); $proxy->push_filter( request => HTTP::Proxy::HeaderFilter::simple->new( sub { my ( $self, $headers, $message ) = @_; return "Not a request" unless $message->isa('HTTP::Request'); my $path = $message->uri->path; my $uri = URI->new( $mirror{local} . $path ); my $selected = $ua->head($uri)->is_success ? 'local' : 'ex +ternal'; $self->proxy->log( HTTP::Proxy::FILTERS, "\u$selected mirror for $path" ); $message->uri( $mirror{$selected} . $path ); } ), response => HTTP::Proxy::BodyFilter::save->new( prefix => $mirror_root, template => '%d/%f', multiple => undef, ), ); $proxy->start; __END__ =head1 NAME automirror.pl - A program to mirror files as you download them. =head1 SYNOPSIS automirror.pl [options] Options: -v, --verbose Increase verbosity level. -p, --port Port to listen for connections. -I, --iface Listen only in this interface address or hostname -l, --local Local mirror address. -e, --external External mirror address. -r, --root Root of the mirror. =head1 OPTIONS =over 8 =item B<--verbose> Increase verbosity level. -v - File and mirror selection information. -vv - Request and response information. -vvv - ALL verbosity avaliable. B<Defaults> to -v =item B<--port> Port to listen for connections. B<Defauls> to 8080 =item B<--iface> Listen only in this interface address or hostname. (eg. 192.168.0.1, localhost or centos.intranet.poormen.com) B<Defaults> to localhost.localdomain =back =head1 DESCRIPTION B<This program> will act as a transparent proxy for a http server, mirroring the files to a especified location as you download them. This is useful for mirroring package repositories, but only the files you've already downloaded. Please note that you still need a working local http server configured to serve the files stored in the location you especified on --root and a external mirror from where to download the files you don`t have already. Of course, this program and the local mirror must run on the same mach +ine, or share a filesytem somehow. NFS or SMB shares, for example. =cut
|
|---|
| Replies are listed 'Best First'. | |
|---|---|
|
Re: Mirror a package repository on the fly.
by 5mi11er (Deacon) on May 21, 2008 at 23:34 UTC | |
by Anonymous Monk on May 28, 2008 at 06:23 UTC | |
by 5mi11er (Deacon) on May 29, 2008 at 05:16 UTC | |
|
Re: Mirror a package repository on the fly.
by 5mi11er (Deacon) on May 22, 2008 at 05:48 UTC | |
by 5mi11er (Deacon) on May 23, 2008 at 19:54 UTC | |
by motobói (Beadle) on May 26, 2008 at 20:52 UTC | |
|
Re: Mirror a package repository on the fly.
by 5mi11er (Deacon) on May 27, 2008 at 17:26 UTC | |
by motobói (Beadle) on Jun 05, 2008 at 06:45 UTC |