This script acts as a transparent proxy which pass a request to a upstream server if the file requested don`t exists on a local mirror. As files are downloaded, the mirror tree is populated.

I found this useful to populate a mirror of centos without downloading the entire repository, but only the files I have needed on any machine of my network. Of course you still have to syncronize repodata by yourself often in a while. But once you do this, if there is no local copy of a package, it will be requested to the upstream.

Add

proxy=http://localmirror:8080
to your /etc/yum.conf. Create the mirror root
mkdir -p /opt/mirror_roor/centos

Create a link to this on /var/www/html:
ln -s /var/www/html/centos/ /opt/mirror_root/centos
perl automirror.pl -l http://localhost/ -e http://mirror.centos.org/ --root /opt/mirror_root/
And you are done.

I had a lot of fun writing this.

Critics very welcome, enjoy it.

#!/usr/bin/perl use warnings; use strict; use HTTP::Proxy qw(:log); use HTTP::Proxy::HeaderFilter::simple; use HTTP::Proxy::BodyFilter::save; use Getopt::Long; use Pod::Usage; use LWP::UserAgent; use URI; my %mirror = (); my $mirror_root; my $verbose = 1; my $port_to_listen; my $interface_address_or_hostname; Getopt::Long::Configure ('bundling'); GetOptions( 'verbose|v+' => \$verbose, 'port|p=i' => \$port_to_listen, 'I=s' => \$interface_address_or_hostname, 'local|l=s' => \$mirror{'local'}, 'external|e=s' => \$mirror{'external'}, 'root|r=s' => \$mirror_root, 'help|h|?' => pod2usage(1), ) or pod2usage(1); pod2usage(1) unless defined $mirror{'local'} and $mirror{'external'} and $mirro +r_root; my $proxy = HTTP::Proxy->new( port => $port_to_listen || 8080, host => $interface_address_or_hostname || 'localhost.localdomain', logmask => $verbose > 2 ? ALL : $verbose > 1 ? STATUS : FILTERS, engine => 'ScoreBoard', #Forking engine. ); my $ua = LWP::UserAgent->new(); $proxy->push_filter( request => HTTP::Proxy::HeaderFilter::simple->new( sub { my ( $self, $headers, $message ) = @_; return "Not a request" unless $message->isa('HTTP::Request'); my $path = $message->uri->path; my $uri = URI->new( $mirror{local} . $path ); my $selected = $ua->head($uri)->is_success ? 'local' : 'ex +ternal'; $self->proxy->log( HTTP::Proxy::FILTERS, "\u$selected mirror for $path" ); $message->uri( $mirror{$selected} . $path ); } ), response => HTTP::Proxy::BodyFilter::save->new( prefix => $mirror_root, template => '%d/%f', multiple => undef, ), ); $proxy->start; __END__ =head1 NAME automirror.pl - A program to mirror files as you download them. =head1 SYNOPSIS automirror.pl [options] Options: -v, --verbose Increase verbosity level. -p, --port Port to listen for connections. -I, --iface Listen only in this interface address or hostname -l, --local Local mirror address. -e, --external External mirror address. -r, --root Root of the mirror. =head1 OPTIONS =over 8 =item B<--verbose> Increase verbosity level. -v - File and mirror selection information. -vv - Request and response information. -vvv - ALL verbosity avaliable. B<Defaults> to -v =item B<--port> Port to listen for connections. B<Defauls> to 8080 =item B<--iface> Listen only in this interface address or hostname. (eg. 192.168.0.1, localhost or centos.intranet.poormen.com) B<Defaults> to localhost.localdomain =back =head1 DESCRIPTION B<This program> will act as a transparent proxy for a http server, mirroring the files to a especified location as you download them. This is useful for mirroring package repositories, but only the files you've already downloaded. Please note that you still need a working local http server configured to serve the files stored in the location you especified on --root and a external mirror from where to download the files you don`t have already. Of course, this program and the local mirror must run on the same mach +ine, or share a filesytem somehow. NFS or SMB shares, for example. =cut

In reply to Mirror a package repository on the fly. by motobói

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":



  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.