Re: Apache log piped to Perl

I don't think its a good idea to have apache forking and doing crazy things.

I'd consider de-coupling this from the apache log handler. Set up a CustomLog to a file and write/borrow:) a log tail-er(perldoc -f seek), which runs from inittab (or a daemon), that processes log entries without intefering with Apache. This has the added benefit, of serialising the process (no fork required), and you won't hit process/resource limits when your webserver gets slammed.

That aside, I don't see why the above script shouldn't work. Are you defining the CustomLog in the server config, or a VirtualHost section? If you set the CustomLog to a file, does anything get written to it?

Comment on Re: Apache log piped to Perl

Replies are listed 'Best First'.
Re^2: Apache log piped to Perl by Anonymous Monk on Aug 09, 2006 at 08:48 UTC
I'd consider de-coupling this from the apache log handler. That makes so much sense I rewrote it immediately using File::Tail and Proc::Daemon. By the way this piped version works, but probably won't scale so well as you generously pointed out. Thank you. #!/usr/bin/perl -w use strict; use Fcntl qw(:DEFAULT); use File::Tail; use Net::Netmask; use Net::Whois::IANA; use Proc::Daemon; use SDBM_File; Proc::Daemon::Init; my $bye = join '\|', ( 'strings known to occur', 'in custname and orgname', ); my $bad = 'some\.bad\.host\|other\.worse\d+'; my $file = File::Tail->new("/path/to/ip_log"); my $iana = Net::Whois::IANA->new; tie (my %seen, 'SDBM_File', '/some/sdbm', O_RDWR\|O_CREAT, 0666) or die "$!"; while (defined (my $ip = $file->read)) { chomp $ip; next if exists $seen{$ip}; $seen{$ip} = 1; my $problem = 0; my $host = `host $ip` or next; $problem = 1 if $host =~ /$badhosts/i; if ($problem or $host =~ /(not found\|timed out)/) { $iana->whois_query(-ip=>$ip); my $inetnum = $iana->inetnum \|\| ''; my $custname = $iana->{QUERY}->{custname} \|\| ''; my $orgname = $iana->{QUERY}->{orgname} \|\| ''; next unless $inetnum and ($problem or $custname or $orgname); if ($custname =~ /($bye)/i or $orgname =~ /($bye)/i){ my $mask = Net::Netmask->new($inetnum); `iptables -A INPUT -s $mask -j DROP`; } } } untie %seen; [download]	[reply] [d/l]

Replies are listed 'Best First'.

Re^2: Apache log piped to Perl
by Anonymous Monk on Aug 09, 2006 at 08:48 UTC

I'd consider de-coupling this from the apache log handler.

That makes so much sense I rewrote it immediately using File::Tail and Proc::Daemon. By the way this piped version works, but probably won't scale so well as you generously pointed out. Thank you.

#!/usr/bin/perl -w

use strict;
use Fcntl qw(:DEFAULT);
use File::Tail;
use Net::Netmask;
use Net::Whois::IANA;
use Proc::Daemon;
use SDBM_File;

Proc::Daemon::Init;

my $bye  = join '|', (
 'strings known to occur',
 'in custname and orgname',
);
my $bad = 'some\.bad\.host|other\.worse\d+';

my $file = File::Tail->new("/path/to/ip_log");
my $iana = Net::Whois::IANA->new;

tie (my %seen, 'SDBM_File', '/some/sdbm', O_RDWR|O_CREAT, 0666)
or die "$!";

while (defined (my $ip = $file->read)) {
  chomp $ip;
  next if exists $seen{$ip};
  $seen{$ip} = 1;

  my $problem = 0;
  my $host = `host $ip` or next;
  $problem = 1 if $host =~ /$badhosts/i;

  if ($problem or $host =~ /(not found|timed out)/) {
    $iana->whois_query(-ip=>$ip);

    my $inetnum  = $iana->inetnum || '';
    my $custname = $iana->{QUERY}->{custname} || '';
    my $orgname  = $iana->{QUERY}->{orgname}  || '';

    next unless $inetnum 
    and ($problem or $custname or $orgname);

    if ($custname =~ /($bye)/i or $orgname =~ /($bye)/i){
      my $mask = Net::Netmask->new($inetnum);
      `iptables -A INPUT -s $mask -j DROP`;
    }
  }
}
untie %seen;
[download]

[reply]
[d/l]