While the webserver level blocks are probably best, if you don't have access/can't, I would do it like this:

# untested!
# a hash of IPs you want to allow
my %allow = (
     '127.0.0.1' => 1,
     '10.1.1.1'  => 1,
      # etc ...
);

unless($allow{ $ENV{REMOTE_ADDR} }) {
     die "Permission denied for $ENV{REMOTE_ADDR}\n";
}
[download]

Hope that helps