Re: Logfile analysis and automatic firewalling

So my first question is, what kind of data structure should I use to store the data collected

Sound like a job for a hash-of-arrays to me. Key is the IP address, and the elements are the number of errors and some way of tracking the time intervals between each error (perhaps the average interval is good enough?)

----
I wanted to explore how Perl's closures can be manipulated, and ended up creating an object system by accident.
-- Schemer

Note: All code is untested, unless otherwise stated

Comment on Re: Logfile analysis and automatic firewalling

Replies are listed 'Best First'.
Re: Re: Logfile analysis and automatic firewalling by pzbagel (Chaplain) on Apr 23, 2003 at 21:00 UTC
I tend to agree with hardburn, actively tracking the IPs in a hash-of-arrays is probably much more efficient than storing every single error in a DB and then calculating error-rates after the fact. Perhaps a sampling mechanism could be employed. Pick a period (5 minutes for instance). When you see an error on a given IP, you store the IP, the time you saw it, and increment the error counter. Now as long as that IP's error counter is less than 5 minutes old, you continue adding subsequent errors to that counter. When the counter is 5 minutes old, you check the error count, decide if the error-rate for 5 minutes is exceeded and decide to allow or ban that IP. Then you remove that hash entry and start again with that IP. When you decide to ban IPs, you can create another hash with IPs and the time they were banned. This hash is used to generate the list of IPs to ban. Once their ban is done (30 minutes) you can drop them from this hash and the program can remove the iptables rules. You can dump these hashes to files at a given interval (or via a signal handler so your iptables generating script can log on, issue a kill with a given signal and expect your script to spit out the list of banned IPs).	[reply]

Replies are listed 'Best First'.

Re: Re: Logfile analysis and automatic firewalling
by pzbagel (Chaplain) on Apr 23, 2003 at 21:00 UTC

I tend to agree with hardburn, actively tracking the IPs in a hash-of-arrays is probably much more efficient than storing every single error in a DB and then calculating error-rates after the fact.

Perhaps a sampling mechanism could be employed. Pick a period (5 minutes for instance). When you see an error on a given IP, you store the IP, the time you saw it, and increment the error counter. Now as long as that IP's error counter is less than 5 minutes old, you continue adding subsequent errors to that counter. When the counter is 5 minutes old, you check the error count, decide if the error-rate for 5 minutes is exceeded and decide to allow or ban that IP. Then you remove that hash entry and start again with that IP.

When you decide to ban IPs, you can create another hash with IPs and the time they were banned. This hash is used to generate the list of IPs to ban. Once their ban is done (30 minutes) you can drop them from this hash and the program can remove the iptables rules.

You can dump these hashes to files at a given interval (or via a signal handler so your iptables generating script can log on, issue a kill with a given signal and expect your script to spit out the list of banned IPs).

[reply]