data mine a firewall log

alexlearn has asked for the wisdom of the Perl Monks concerning the following question:

I have been set the task of analysing a firewall log and then creating a visualisation of this log. My first task is to work out the most important data in the firewall log here is one line of the log

Date/time    Syslog priority    Operation    Message code    Protocol 
+   Source IP    Destination IP    Source hostname    Destination host
+name    Source port    Destination port    Destination service    Dir
+ection    Connections built    Connections torn down
[download]

There are the column names. What sort of information am I looking forward to find attacks in the firewall log and also how can I use Perl to flag certain rows for certain attacks. If you require more information about the scenario I will include it.

Comment on data mine a firewall log Download Code

Replies are listed 'Best First'.
Re: data mine a firewall log by marto (Cardinal) on May 31, 2011 at 15:15 UTC
It seems to me like you have two questions here: 1. How to identify attacks. 2. How to achieve 1 using Perl. For part 1, you need to research Network security, learn what you should be looking for (searching may help). For part two you need to learn Perl: http://learn.perl.org http://perldoc.perl.org The Tutorials section of this site. Update: fixed typo	[reply]
Re: data mine a firewall log by dwhite20899 (Friar) on May 31, 2011 at 15:10 UTC
You are asking a very general question about an extremely complicated subject. I'd point you toward a reference like "Perl for System Administration" which has a chapter dedicated to log analysis. And loook in CPAN.	[reply]
Re: data mine a firewall log by zek152 (Pilgrim) on May 31, 2011 at 15:11 UTC
This is a very vague question so my response might not be useful at all. Hopefully this is not the case. I don't know how much experience you have with network security. Try to talk to someone at your company regarding the specific details of the network's security. Depending on the information provided in the log you might not be able to gather any useful information from the log. If the log is just showing rejected or dropped packets then looking for attacks in the log is pointless. If an attack can be found in the log it clearly failed (because the packet did not get through). And obviously you can not know about things that aren't in the log just from looking at the log. Regardless of the amount of information the log provides you can do a reverse dns lookup on the `source ip` and see if it matches the `source hostname`. If the resolved hostname and source hostname do not match that should raise some flags. It might not be an attack but it is suspicious behavior. If the log shows packets that are passed throught the filter then there are some things that you can look for. If there are windows machines behind the firewall then look for activity on port 445. Many worms use vulnerabilites that windows has traditionally had in services monitoring that port. There are other things that might indicate attacks however a network security expert at the company would be the person to ask. Hope this helps a little.	[reply] [d/l] [select]