Sorting log files with IP addresses (duplicates)

Secode has asked for the wisdom of the Perl Monks concerning the following question:

Greetings to all at Perlmonks,

I have a log file from a Pix firewall the record of the syslog looks like this:

2006-03-20 00:02:48,Local4.Warning,1.1.1.10,Mar 19 2006 23:56:32: %PIX-4-106023: Deny udp src inside:1.1.1.1/1161 dst outside:2.2.2.2/53 by access-group "inside_access_in".

In the syslog file there are mutiple instances of the same src ip address with different or the same dst address's. I am not interested in the src port.

I need to sort the file so that I can have a report based on src IP address with all the associated dst IP address's,dst port,src interface and dst interface.

I prsume(not too sure)the best structure would be an array of arrays - I can get the data from the syslog file into the array and strip the(/ : , ") out of it but have no idea how to sort into the format I need. Any help with this would be really apprciated as once I can understand how to do this I have a number of other projects I would be able to do based on the same kind of thing. I can mail the syslog file if it would help.

Thanks

Pierre

Formatting added by GrandFather

Comment on Sorting log files with IP addresses (duplicates) Download Code

Replies are listed 'Best First'.
Re: Sorting log files with IP addresses (duplicates) by strat (Canon) on Apr 21, 2006 at 09:13 UTC
if you have the file content in an array, you could try something like: `my @sorted = map { $_->[4] } sort { $a->[0] <=> $b->[0] or $a->[1] <=> $b->[1] or $a->[2] <=> $b->[2] or $a->[3] <=> $b->[3] } map { if( my (@ipParts) = $_ =~ m/dst outside:(\d+)\.(\d+)\.(\d+)\.(\d+)/ ) { [ @ipParts, $_ ] } else { # what shell I do if pattern not found? use 0.0.0.0? [ 0, 0, 0, 0, $_ ]; } # else } @input;` [download] Perhaps you have to change the pattern match... See also: Schwartzian Transform Best regards, perl -e "s>>F>e=>y)\martinF)stronat)=>print,print v8.8.8.32.11.32"	[reply] [d/l]
Re^2: Sorting log files with IP addresses (duplicates) by salva (Canon) on Apr 21, 2006 at 09:37 UTC
it gets quite easier (and faster) using Sort::Key::Natural: `use Sort::Key::Natural 'natkeysort'; my @sorted = natkeysort { /src inside:(\d+\.\d+\.\d+\.\d+)/ ? $1 : '0.0.0.0' } @input;` [download]	[reply] [d/l]
Re: Sorting log files with IP addresses (duplicates) by jwkrahn (Abbot) on Apr 21, 2006 at 22:46 UTC
You can sort them using the Socket module and a GRT: `use Socket; print map substr( $_, 4 ), sort map { my ( $src ) = /src\D+([\d.]+)/; my $ip = inet_aton( $src ) \|\| "\0\0\0\0"; "$ip$_"; } @syslog;` [download]	[reply] [d/l]
Re^2: Sorting log files with IP addresses (duplicates) by salva (Canon) on Apr 22, 2006 at 10:52 UTC
From the `inet_aton` docs: `For portability do not assume that the result of inet_aton() is 32 bits wide, in other words, that it would contain only the IPv4 address in network order.` [download] Using `pack` is probably a better option: `my $ip = pack(C4 => split /\./, $src);` [download]	[reply] [d/l] [select]
Re^3: Sorting log files with IP addresses (duplicates) by jwkrahn (Abbot) on Apr 23, 2006 at 01:40 UTC
True, it depends on the characteristics of inet_aton(3) however using pack() and split() only works if the IP address is in the dotted quad format while inet_aton() will work with any valid IP address and return undef if the address is not valid.	[reply]
Re^2: Sorting log files with IP addresses (duplicates) by Secode (Novice) on Apr 24, 2006 at 09:52 UTC
Thanks to everyone for their help I now have the file sorted by src address using the "socket" method and the record looks like this: 2006-03-16 06 50 08 Local4.Warning 155.236.56.102 Mar 16 2006 06 44 11 %PIX-4-106023 Deny udp src inside 10.200.20.71 1047 dst outside 155.236.60.129 161 by access-group inside_access_in 2006-03-16 07 01 20 Local4.Warning 155.236.56.102 Mar 16 2006 06 55 23 %PIX-4-106023 Deny udp src inside 10.200.20.71 1047 dst outside 155.236.60.106 161 by access-group inside_access_in 2006-03-16 07 01 20 Local4.Warning 155.236.56.102 Mar 16 2006 06 55 23 %PIX-4-106023 Deny udp src inside 10.200.20.71 1047 dst outside 155.236.60.106 161 by access-group inside_access_in 2006-03-29 07 11 17 Local4.Warning 155.236.56.102 Mar 29 2006 07 04 12 %PIX-4-106023 Deny udp src inside 10.200.20.71 1046 dst outside 155.236.60.181 161 by access-group inside_access_in 2006-03-14 12 09 52 Local4.Warning 155.236.56.102 Mar 14 2006 12 04 05 %PIX-4-106023 Deny tcp src inside 10.200.21.72 2519 dst outside 207.46.253.188 80 by access-group inside_access_in I now need to get all the dst IP address's sorted together for each src IP - I will need to do more but once I see how you sort dst IP address per src IP address I will be able to do this.	[reply]