Secode has asked for the wisdom of the Perl Monks concerning the following question:

Greetings (Monks){

Thanks very much to all those who helped me with my first question.

I have progressed somewhat with my Pix syslog project and learning a lot but I am up against a new challenge : how do I get my results to look like the following:

src ipaddress - dest ipaddress - port - number of times of occurance

from the results below

%PIX-4-106023 Deny tcp src inside 1.1.1.1 1637 dst outside 4.4.4.4 80 
+by access-group inside_access_in
%PIX-4-106023 Deny tcp src inside 1.1.1.1 2519 dst outside 4.4.4.4 80 
+by access-group inside_access_in

[download]

would look like:

1.1.1.1 4.4.4.4 80 2

 My code looks like the following so far (excuse newbie hacking I am learning) I am using Activestate perl.

#c:\perl\bin\perl
use Socket;

open(INFILE, "<syslog.txt");
open(OUTFILE, ">sorted4.txt");

while (<INFILE>) {
my($line2) = $_;
$line2 =~ s/\"//g;
$line2 =~ s/\:/ /g;
$line2 =~ s/\,/ /g;
$line2 =~ s/\// /g;
$line = substr $line2, index($line2, '%PIX');
push @input, $line;
}

@in = 
map substr( $_, 8 ),
sort
map {
my ( $src ) = /src\D+([\d.]+)/;
my $srcip = inet_aton( $src ) || "\0\0\0\0";
my ( $dst ) = /dst\D+([\d.]+)/;
my $dstip = inet_aton( $dst ) || "\0\0\0\0";
my ( $port ) = /$dst\D+([\d.]+)/;
"$srcip$dstip$_";
}
@input;

print @in; 

close(INFILE);
close(OUTFILE);

[download]

the output looks like this currently and is sorted by source IP and then dst IP per source.

%PIX-4-106023 Deny udp src inside 1.1.1.1 1161 dst outside 3.3.3.3 53 
+by access-group inside_access_in
%PIX-4-106023 Deny tcp src inside 1.1.1.1 1637 dst outside 4.4.4.4 80 
+by access-group inside_access_in
%PIX-4-106023 Deny tcp src inside 1.1.1.1 2519 dst outside 4.4.4.4 80 
+by access-group inside_access_in
%PIX-4-106023 Deny udp src inside 1.1.1.1 1161 dst outside 7.7.7.7 53 
+by access-group inside_access_in
%PIX-4-106023 Deny tcp src inside 1.1.1.1 2519 dst outside 9.9.9.9 80 
+by access-group inside_access_in
%PIX-4-106023 Deny udp src inside 1.1.1.1 1161 dst outside 9.9.9.9 443
+ by access-group inside_access_in
%PIX-4-106023 Deny udp src inside 1.1.1.1 1161 dst outside 9.9.9.9 53 
+by access-group inside_access_in
%PIX-4-106023 Deny tcp src inside 2.2.2.2 2605 dst outside 3.3.3.3 80 
+by access-group inside_access_in
%PIX-4-106023 Deny tcp src inside 2.2.2.2 3944 dst outside 3.3.3.3 80 
+by access-group inside_access_in
%PIX-4-106023 Deny tcp src inside 2.2.2.2 1263 dst outside 4.4.4.4 80 
+by access-group inside_access_in
%PIX-4-106023 Deny tcp src inside 2.2.2.2 2605 dst outside 7.7.7.7 80 
+by access-group inside_access_in
%PIX-4-106023 Deny tcp src inside 2.2.2.2 3944 dst outside 7.7.7.7 80 
+by access-group inside_access_in
%PIX-4-106023 Deny udp src inside 5.5.5.5 1047 dst outside 3.3.3.3 143
+3 by access-group inside_access_in
%PIX-4-106023 Deny tcp src inside 5.5.5.5 1379 dst outside 6.6.6.6 443
+ by access-group inside_access_in
%PIX-4-106023 Deny tcp src inside 5.5.5.5 1379 dst outside 6.6.6.6 80 
+by access-group inside_access_in
%PIX-4-106023 Deny udp src inside 5.5.5.5 1047 dst outside 6.6.6.6 161
+ by access-group inside_access_in
%PIX-4-106023 Deny tcp src inside 8.8.8.8 1263 dst outside 4.4.4.4 80 
+by access-group inside_access_in
%PIX-4-106023 Deny tcp src inside 8.8.8.8 2677 dst outside 6.6.6.6 80 
+by access-group inside_access_in
%PIX-4-106023 Deny tcp src inside 10.10.10.10 1637 dst outside 4.4.4.4
+ 80 by access-group inside_access_in
%PIX-4-106023 Deny tcp src inside 10.10.10.10 2677 dst outside 6.6.6.6
+ 80 by access-group inside_access_in

[download]

 thanks

Secode

Replies are listed 'Best First'.
Re: looping through an array
by McDarren (Abbot) on Apr 25, 2006 at 13:07 UTC
    Let me apologise straight out for ignoring your existing code, but here is one way to get your desired output, from the data you supply... 
    #!/usr/bin/perl -w
use strict;

my %data;
my ($src, $dest, $dst_port);

while (<DATA>) {
    chomp;
    my ($src, $dest, $dst_port) = (split, $_)[5,9,10];
    $data{$src}{$dest}{$dst_port}++;
}

for $src (sort by_ip keys %data) {
    for $dest (sort by_ip keys %{$data{$src}}) {
        for $dst_port (sort keys %{$data{$src}{$dest}}) {
            print "$src $dest $dst_port $data{$src}{$dest}{$dst_port}\
+n";
        }
    }
}

sub ipto32bit {
    my ($ip, $c, $d, $e, $f);
    $ip = shift;
    ($c,$d,$e,$f) = split(/\./,$ip);
    return ($c << 24) + ($d << 16) + ($e << 8) + $f;
}

sub by_ip {
    ipto32bit($a) <=> ipto32bit($b);
}

__DATA__
%PIX-4-106023 Deny udp src inside 1.1.1.1 1161 dst outside 3.3.3.3 53 
+by access-group inside_access_in
%PIX-4-106023 Deny tcp src inside 1.1.1.1 1637 dst outside 4.4.4.4 80 
+by access-group inside_access_in
# etc... rest omitted for brevity

    [download]
    Which prints.. 
    1.1.1.1 3.3.3.3 53 1
1.1.1.1 4.4.4.4 80 2
1.1.1.1 7.7.7.7 53 1
1.1.1.1 9.9.9.9 443 1
1.1.1.1 9.9.9.9 53 1
1.1.1.1 9.9.9.9 80 1
2.2.2.2 3.3.3.3 80 2
2.2.2.2 4.4.4.4 80 1
2.2.2.2 7.7.7.7 80 2
5.5.5.5 3.3.3.3 1433 1
5.5.5.5 6.6.6.6 161 1
5.5.5.5 6.6.6.6 443 1
5.5.5.5 6.6.6.6 80 1
8.8.8.8 4.4.4.4 80 1
8.8.8.8 6.6.6.6 80 1
10.10.10.10 4.4.4.4 80 1
10.10.10.10 6.6.6.6 80 1

    [download]

    Note that I have used split, because your data seems to be well formed. It may or may not be appropriate in your case. The above is not necessarily a serious suggestion, but is given more in the spirit of TIMTOWTDI :)

    Cheers,
    Darren :)

[reply]
[d/l]
[select]

      The following is probably faster (especially for large amounts of data), and it sorts the ports numerically. Your solution oddly sorted them alphabetically. 

      sub sort_by_ip {
   return
   map { substr($_, 4) }
   sort
   map { pack("C4a*", split(/\./), $_) }
   @_;
}

my %data;

while (<DATA>) {
    chomp;
    my ($src, $dest, $dst_port) = (split)[5,9,10];
    $data{$src}{$dest}{$dst_port}++;
}

foreach my $src      (sort_by_ip         keys %data                ) {
foreach my $dest     (sort_by_ip         keys %{$data{$src}}       ) {
foreach my $dst_port (sort { $a <=> $b } keys %{$data{$src}{$dest}}) {
   print "$src $dest $dst_port $data{$src}{$dest}{$dst_port}\n";
}}}

      [download]
[reply]
[d/l]
Re: looping through an array
by reasonablekeith (Deacon) on Apr 25, 2006 at 13:00 UTC
    Well, this is how I'd do it, but it's a bit lazy really. I normally like to self document, and joining the three extracted variables into one string isn't too descriptive. That said it works like this.

    There are two stages, firstly going through the data and building a hash, the keys of which are all the unique combination of src_ip, dest_ip, and port. This key's value is incremented each time it's found, and so the value of the hash key forms our count value.

    Then it's just a case of going through the keys, spliting them apart so we can join them in the correct format, and printing the count value.

    If you were going to do anything more complicated than this then you should probably store your data in a hash of hash format, rather than my mangled string example, but I had to leave you with something to do :)

    anyway, here's the code... 

    my $delim = "DELIM";
my %access_count;
while (<DATA>) {
    my $access_string = join($delim, ($_ =~ m/(\d+\.\d+\.\d+\.\d+).*?(
+\d+\.\d+\.\d+\.\d+)\s(\d+)/));
    print "$access_string\n";
    $access_count{$access_string}++;
}

foreach my $access_string (sort keys %access_count) {
    print join(' - ',
            (split /$delim/, $access_string),
            "[$access_count{$access_string}]"
    ) . "\n";
}


__DATA__
%PIX-4-106023 Deny udp src inside 1.1.1.1 1161 dst outside 3.3.3.3 53 
+by access-group inside_access_in
%PIX-4-106023 Deny tcp src inside 1.1.1.1 1637 dst outside 4.4.4.4 80 
+by access-group inside_access_in
%PIX-4-106023 Deny tcp src inside 1.1.1.1 2519 dst outside 4.4.4.4 80 
+by access-group inside_access_in
%PIX-4-106023 Deny udp src inside 1.1.1.1 1161 dst outside 7.7.7.7 53 
+by access-group inside_access_in
%PIX-4-106023 Deny tcp src inside 1.1.1.1 2519 dst outside 9.9.9.9 80 
+by access-group inside_access_in
%PIX-4-106023 Deny udp src inside 1.1.1.1 1161 dst outside 9.9.9.9 443
+ by access-group inside_access_in

    [download]
    ---
    my name's not Keith, and I'm not reasonable.
[reply]
[d/l]
      You could get golfy about it too*. (using McDarren's split) 
      #!/usr/bin/perl
use warnings;
use strict;

while(<DATA>){$a->{join' ',(split)[5,9,10]}++}print join' - ',split,$a
+->{$_}."\n"for sort keys%$a;

      [download]

      * sorry, I know this isn't particularly useful, but I'm a bit bored here.

      ---
      my name's not Keith, and I'm not reasonable.
[reply]
[d/l]
Re: looping through an array
by mantadin (Beadle) on Apr 25, 2006 at 13:05 UTC
    This does the job ... 
    my ($src, $tgt, $prt, $r);

while(<>){
        ($src, $tgt, $prt) = $_ =~ 
              m/\s+(\S+)\s*\d+ dst outside (.*?) (\d+) by access-gr/;
        die "notgood: $_\n" unless $prt;
        $r->{$src}->{$tgt}->{$prt}++;
}
foreach $src (keys %{$r}) {     # foreach source ip
        foreach $tgt (keys %{$r->{$src}}) {     # foreach target ip
                foreach $prt (keys %{$r->{$src}->{$tgt}}) {
                        printf "%-16s %-16s %-6s counted %d times\n",
                                $src, $tgt, $prt,
                                $r->{$src}->{$tgt}->{$prt};
                }
        }
}

    [download]
    yielding 
    10.10.10.10      4.4.4.4          80     counted 1 times
10.10.10.10      6.6.6.6          80     counted 1 times
1.1.1.1          7.7.7.7          53     counted 1 times
1.1.1.1          3.3.3.3          53     counted 1 times
1.1.1.1          4.4.4.4          80     counted 2 times
1.1.1.1          9.9.9.9          53     counted 1 times
1.1.1.1          9.9.9.9          443    counted 1 times
1.1.1.1          9.9.9.9          80     counted 1 times
5.5.5.5          3.3.3.3          1433   counted 1 times
5.5.5.5          6.6.6.6          161    counted 1 times
5.5.5.5          6.6.6.6          443    counted 1 times
5.5.5.5          6.6.6.6          80     counted 1 times
8.8.8.8          4.4.4.4          80     counted 1 times
8.8.8.8          6.6.6.6          80     counted 1 times
2.2.2.2          7.7.7.7          80     counted 2 times
2.2.2.2          3.3.3.3          80     counted 2 times
2.2.2.2          4.4.4.4          80     counted 1 times

    [download]
    greets
[reply]
[d/l]
[select]
Re: looping through an array
by johngg (Canon) on Apr 25, 2006 at 14:02 UTC
    I put this script together but I will have to learn to type faster as other Monks have beaten me to the punch as usual. In particular, ++ to McDarren's sub by_ip and sub ipto32bit which are much neater than my sort. Anyway, better late than never.

    use strict;
use warnings;

my $rcSortAll = sub
   {
       my @partsA = split /[.:]/, $a;
       my @partsB = split /[.:]/, $b;
       return
          $partsA[0] <=> $partsB[0] ||
          $partsA[1] <=> $partsB[1] ||
          $partsA[2] <=> $partsB[2] ||
          $partsA[3] <=> $partsB[3] ||
          $partsA[4] <=> $partsB[4] ||
          $partsA[5] <=> $partsB[5] ||
          $partsA[6] <=> $partsB[6] ||
          $partsA[7] <=> $partsB[7] ||
          $partsA[8] <=> $partsB[8];
   };

my $rxIP = qr{((?:\d+\.){3}\d+)};
my $rxExtract = qr{(?x)
   inside\s+
   $rxIP
   .*outside\s+
   $rxIP
   \s+(\d+)};

my $rhLogInfo = {};
while(<DATA>)
{
    next unless /$rxExtract/;
    $rhLogInfo->{join ':', $1, $2, $3} ++;
}

foreach my $key (sort $rcSortAll keys %$rhLogInfo)
{
    (my $spaced = $key) =~ s/:/ /g;
    print "$spaced $rhLogInfo->{$key}\n";
}

__END__
%PIX-4-106023 Deny udp src inside 1.1.1.1 1161 dst outside 3.3.3.3 53 
+by access-group inside_access_in
%PIX-4-106023 Deny tcp src inside 1.1.1.1 1637 dst outside 4.4.4.4 80 
+by access-group inside_access_in
%PIX-4-106023 Deny tcp src inside 1.1.1.1 2519 dst outside 4.4.4.4 80 
+by access-group inside_access_in
%PIX-4-106023 Deny udp src inside 1.1.1.1 1161 dst outside 7.7.7.7 53 
+by access-group inside_access_in
%PIX-4-106023 Deny tcp src inside 1.1.1.1 2519 dst outside 9.9.9.9 80 
+by access-group inside_access_in
%PIX-4-106023 Deny udp src inside 1.1.1.1 1161 dst outside 9.9.9.9 443
+ by access-group inside_access_in
%PIX-4-106023 Deny udp src inside 1.1.1.1 1161 dst outside 9.9.9.9 53 
+by access-group inside_access_in
%PIX-4-106023 Deny tcp src inside 2.2.2.2 2605 dst outside 3.3.3.3 80 
+by access-group inside_access_in
%PIX-4-106023 Deny tcp src inside 2.2.2.2 3944 dst outside 3.3.3.3 80 
+by access-group inside_access_in
%PIX-4-106023 Deny tcp src inside 2.2.2.2 1263 dst outside 4.4.4.4 80 
+by access-group inside_access_in
%PIX-4-106023 Deny tcp src inside 2.2.2.2 2605 dst outside 7.7.7.7 80 
+by access-group inside_access_in
%PIX-4-106023 Deny tcp src inside 2.2.2.2 3944 dst outside 7.7.7.7 80 
+by access-group inside_access_in
%PIX-4-106023 Deny udp src inside 5.5.5.5 1047 dst outside 3.3.3.3 143
+3 by access-group inside_access_in
%PIX-4-106023 Deny tcp src inside 5.5.5.5 1379 dst outside 6.6.6.6 443
+ by access-group inside_access_in
%PIX-4-106023 Deny tcp src inside 5.5.5.5 1379 dst outside 6.6.6.6 80 
+by access-group inside_access_in
%PIX-4-106023 Deny udp src inside 5.5.5.5 1047 dst outside 6.6.6.6 161
+ by access-group inside_access_in
%PIX-4-106023 Deny tcp src inside 8.8.8.8 1263 dst outside 4.4.4.4 80 
+by access-group inside_access_in
%PIX-4-106023 Deny tcp src inside 8.8.8.8 2677 dst outside 6.6.6.6 80 
+by access-group inside_access_in
%PIX-4-106023 Deny tcp src inside 10.10.10.10 1637 dst outside 4.4.4.4
+ 80 by access-group inside_access_in
%PIX-4-106023 Deny tcp src inside 10.10.10.10 2677 dst outside 6.6.6.6
+ 80 by access-group inside_access_in

    [download]

    which produces

    1.1.1.1 3.3.3.3 53 1
1.1.1.1 4.4.4.4 80 2
1.1.1.1 7.7.7.7 53 1
1.1.1.1 9.9.9.9 53 1
1.1.1.1 9.9.9.9 80 1
1.1.1.1 9.9.9.9 443 1
2.2.2.2 3.3.3.3 80 2
2.2.2.2 4.4.4.4 80 1
2.2.2.2 7.7.7.7 80 2
5.5.5.5 3.3.3.3 1433 1
5.5.5.5 6.6.6.6 80 1
5.5.5.5 6.6.6.6 161 1
5.5.5.5 6.6.6.6 443 1
8.8.8.8 4.4.4.4 80 1
8.8.8.8 6.6.6.6 80 1
10.10.10.10 4.4.4.4 80 1
10.10.10.10 6.6.6.6 80 1

    [download]

    Cheers,

    JohnGG

[reply]
[d/l]
[select]

Back to Seekers of Perl Wisdom