*Doh!*, my bad, $ENV{REMOTE_ADDR} doesn't have a line break, so you might want to chomp the input first:

open(BANNED, "<$banned") || die "Can't open $banned: $!";
while(<BANNED>) {
   chomp;
   if($_ eq $ENV{REMOTE_ADDR}) {
      print "... you are banned ...";
      exit;
   }
}
close BANNED;
[download]

update: If you use Apache, you also might want to use it's blocking capabilities :)