firepro20 has asked for the wisdom of the Perl Monks concerning the following question:

I have the following logfile

2016-04-29 15:56:48+0000 [cowrie.ssh.transport.HoneyPotSSHFactory] New
+ connection: 89.248.167.131:46055 (172.17.0.2:2222) [session: 3b8d22b
+5]
2016-04-29 15:56:49+0000 [SSHService ssh-userauth on HoneyPotTransport
+,28,89.248.167.131] login attempt [root/root] succeeded
2016-04-29 16:11:14+0000 [cowrie.ssh.transport.HoneyPotSSHFactory] New
+ connection: 52.28.89.99:53059 (172.17.0.2:2222) [session: a6c0fac1]
2016-04-29 16:17:42+0000 [cowrie.ssh.transport.HoneyPotSSHFactory] New
+ connection: 13.92.114.157:1032 (172.17.0.2:2222) [session: d33e1566]
2016-04-29 19:07:10+0000 [cowrie.ssh.transport.HoneyPotSSHFactory] New
+ connection: 89.248.167.131:45178 (172.17.0.6:2222) [session: fafec37
+d]
2016-04-29 19:07:10+0000 [SSHService ssh-userauth on HoneyPotTransport
+,0,89.248.167.131] login attempt [root/root] succeeded
2016-04-29 19:42:58+0000 [cowrie.ssh.transport.HoneyPotSSHFactory] New
+ connection: 89.248.167.131:56925 (172.17.0.6:2222) [session: 539960a
+3]
2016-04-29 19:42:58+0000 [SSHService ssh-userauth on HoneyPotTransport
+,1,89.248.167.131] login attempt [root/root] succeeded
2016-04-29 20:39:03+0000 [cowrie.ssh.transport.HoneyPotSSHFactory] New
+ connection: 89.248.167.131:54138 (172.17.0.6:2222) [session: b9f550d
+f]
2016-04-29 20:39:03+0000 [SSHService ssh-userauth on HoneyPotTransport
+,2,89.248.167.131] login attempt [root/root] succeeded
2016-04-29 21:13:41+0000 [cowrie.ssh.transport.HoneyPotSSHFactory] New
+ connection: 141.8.83.213:64400 (172.17.0.6:2222) [session: e696835c]
2016-04-29 21:13:59+0000 [SSHService ssh-userauth on HoneyPotTransport
+,3,141.8.83.213] login attempt [user1/test123] failed
2016-04-29 21:14:10+0000 [SSHService ssh-userauth on HoneyPotTransport
+,3,141.8.83.213] login attempt [user1/test1234] failed
2016-04-29 21:14:13+0000 [SSHService ssh-userauth on HoneyPotTransport
+,3,141.8.83.213] login attempt [user1/test123] failed

[download]

I want to increment a value called frequency whenever the IP is found and the same username and password are used to login. For that I want to output the SourcePort, the statusfailed/succedded and the total number of occurrences of the IP, and user/pass combination. So for instance, taking the last three entries, the result would be 

Port,Status,Occurrences
64400,failed,2
64400,failed,1

[download]

This is the code I have so far

$ip = "";
        $port = "";
        $usr = "";
        $pass = "";
        $status = "";
        $frequency = 0;

        #Given this is a user/pass attempt honeypot logger, I will use
+ a wide character to reduce the possibility of stopping
        #the WEKA CSV loader from functioning by using smileyface as s
+eperators.


        while(my $lines = <LOG2>){

                if($lines =~ /New connection/){

                ($ip, $port) = (split /[\[\]\s:()]+/, $lines)[7,8];

                }
                if($lines =~ /login attempt/){#and the ip of the new c
+onnection

                if($lines =~ /$ip/){
                ($usr, $pass, $status) = (split /[\s:\[\]\/]+/, $lines
+)[-3,-2,-1];

                        if($lines =~/$ip/ && $lines =~/$usr/ && $lines
+ =~ /$pass/){

                        $frequency++

                        }

                        if($ip && $port && $usr && $pass && $status ne
+ ""){
                                print FILE2 join "$seperator",($port, 
+$status, $frequency, $end);
                                print FILE2 "\n";
                        }
                        }


                }

[download]

This was also cross posted on Stack Overflow

Replies are listed 'Best First'.
Re: Increment frequency of attempts based on IP and login details combination
by FreeBeerReekingMonk (Deacon) on Apr 30, 2016 at 18:04 UTC
    This should be easy to modify to get what you want. There are some more succinct methods to query deep hashes, cant find the module right now.

    while(<DATA>){
        if (/New connection: ([\d\.]+):(\d+)/){
                ($ip,$port) = ($1,$2);
                next;
        }
        chomp;
        if (/login attempt\s+\[(.*)\]\s+(\w+)$/){
                ($user_pass,$status) = ($1,$2);
                $HONEY{$ip}{$port}{$status}{$user_pass} +=1;
                print "DEBUG: Add ip=$ip:$port $status $user_pass\n";
        }
}

for my $ip (keys %HONEY){
        for my $port (keys %{$HONEY{$ip}}){
                for my $user (keys %{$HONEY{$ip}{$port}}){
                        for my $status (keys %{$HONEY{$ip}{$port}}){
                                for my $user_pass (keys %{$HONEY{$ip}{
+$port}{$status}}){
                                        $freq = $HONEY{$ip}{$port}{$st
+atus}{$user_pass};
                                        push(@DATA, "$port,$status,$fr
+eq") ;
                                }
                        }
                }
        }
}

for my $data (sort @DATA){
        print $data . "\n";
}

__DATA__
2016-04-29 15:56:48+0000 [cowrie.ssh.transport.HoneyPotSSHFactory] New
+ connection: 89.248.167.131:46055 (172.17.0.2:2222) [session: 3b8d22b
+5]
2016-04-29 15:56:49+0000 [SSHService ssh-userauth on HoneyPotTransport
+,28,89.248.167.131] login attempt [root/root] succeeded
2016-04-29 16:11:14+0000 [cowrie.ssh.transport.HoneyPotSSHFactory] New
+ connection: 52.28.89.99:53059 (172.17.0.2:2222) [session: a6c0fac1]
2016-04-29 16:17:42+0000 [cowrie.ssh.transport.HoneyPotSSHFactory] New
+ connection: 13.92.114.157:1032 (172.17.0.2:2222) [session: d33e1566]
2016-04-29 19:07:10+0000 [cowrie.ssh.transport.HoneyPotSSHFactory] New
+ connection: 89.248.167.131:45178 (172.17.0.6:2222) [session: fafec37
+d]
2016-04-29 19:07:10+0000 [SSHService ssh-userauth on HoneyPotTransport
+,0,89.248.167.131] login attempt [root/root] succeeded
2016-04-29 19:42:58+0000 [cowrie.ssh.transport.HoneyPotSSHFactory] New
+ connection: 89.248.167.131:56925 (172.17.0.6:2222) [session: 539960a
+3]
2016-04-29 19:42:58+0000 [SSHService ssh-userauth on HoneyPotTransport
+,1,89.248.167.131] login attempt [root/root] succeeded
2016-04-29 20:39:03+0000 [cowrie.ssh.transport.HoneyPotSSHFactory] New
+ connection: 89.248.167.131:54138 (172.17.0.6:2222) [session: b9f550d
+f]
2016-04-29 20:39:03+0000 [SSHService ssh-userauth on HoneyPotTransport
+,2,89.248.167.131] login attempt [root/root] succeeded
2016-04-29 21:13:41+0000 [cowrie.ssh.transport.HoneyPotSSHFactory] New
+ connection: 141.8.83.213:64400 (172.17.0.6:2222) [session: e696835c]
2016-04-29 21:13:59+0000 [SSHService ssh-userauth on HoneyPotTransport
+,3,141.8.83.213] login attempt [user1/test123] failed
2016-04-29 21:14:10+0000 [SSHService ssh-userauth on HoneyPotTransport
+,3,141.8.83.213] login attempt [user1/test1234] failed
2016-04-29 21:14:13+0000 [SSHService ssh-userauth on HoneyPotTransport
+,3,141.8.83.213] login attempt [user1/test123] failed

    [download]
[reply]
[d/l]
      Depending on how is is going to be used, it may not be necessary to build deeply nested hashes, as a composite key can often render the same service and make things easier.

      For example, 

      $HONEY{$ip}{$port}{$status}{$user_pass} +=1;

      [download]
      might be replaced by: 
      $HONEY{"$ip;$port;$status;$user_pass"} ++;

      [download]
      but it really depends on how it's gonna be used.
[reply]
[d/l]
[select]
      Just another way of iterating on a deeply nested hash which does not need to go back to the original hash each time (a made up example) 
      use strict;
use warnings;
use Data::Dumper;


my %hash;
my $count = 0;

# populating a deeply nested hash
for my $ip (qw <ip1 ip2 ip3>) {
    for my $port ( qw <port1 port2 port3>) {
        for my $status (qw <stat1 stat2>) {
            for my $user (qw <user1 user2 user3>) {
                $hash{$ip}{$port}{$status}{$user} = $count++;  # (just
+ a dummy value for testing)
            }
        }
    }
}

# print Dumper \%hash;

# iterating through the nested hash
for my $nest0 (values %hash) {
    while (my ($port, $nest1) = each %$nest0) {        
        while (my ($status, $nest2) = each %$nest1) {
            for my $freq (values %$nest2) {
                print "$port\t$status\t$freq\n";
            }
        }
    }
}

      [download]
      Note: I use each when I need both the key and the sub-hash ref (because I need to print the key) and values when I only need to dive one step further into the nested structure. But, for some reason, a for loop with each seems to behave inconsistently (incomplete results) on my Perl version, so I used a while loop for the cases with each. I think to remember from the past there was a bug at some point on each, maybe my version of Perl is one of them. Well, anyway, replacing the for loop by a while loop solves the issue.

      This prints in part: 

      port2   stat2   9
port2   stat2   11
port2   stat2   10
port2   stat1   6
port2   stat1   8
port2   stat1   7
port3   stat2   15
port3   stat2   17
port3   stat2   16
port3   stat1   12
port3   stat1   14
port3   stat1   13
port1   stat2   3
port1   stat2   5
port1   stat2   4
port1   stat1   0
port1   stat1   2
.....

      [download]
[reply]
[d/l]
[select]

      I tried what you suggested but the argument for the loop was read from a file like so

      #!/usr/bin/perl

$log = "/home/tsec/prototype/logs/extractedlogs/cowrieresult.log";

open(DATA, $log) or die "Can't open '$log': $!";

sub tester(){

while(<DATA>){
        if (/New connection: ([\d\.]+):(\d+)/){
                ($ip,$port) = ($1,$2);
                next;
        }
        chomp;
        if (/login attempt\s+\[(.*)\]\s+(\w+)$/){
                ($user_pass,$status) = ($1,$2);
                $HONEY{$ip}{$port}{$status}{$user_pass} +=1;
                print "DEBUG: Add ip=$ip:$port $status $user_pass\n";
        }
}

for my $ip (keys %HONEY){
        for my $port (keys %{$HONEY{$ip}}){
                for my $user (keys %{$HONEY{$ip}{$port}}){
                        for my $status (keys %{$HONEY{$ip}{$port}}){
                                for my $user_pass (keys %{$HONEY{$ip}{
+$port}{$status}}){
                                        $freq = $HONEY{$ip}{$port}{$st
+atus}{$user_pass};
                                        push(@DATA, "$port,$status,$fr
+eq") ;
                                }
                        }
                }
        }
}

for my $data (sort @DATA){
        print $data . "\n";
}


}

      [download]

      Unfortunately nothing was outputted on the screen. Also if I use stict and warnings it says that variables requires explicit package name and other errors as well as shown below

      Variable "@DATA" is not imported at ./test2.prg line 31.
Variable "@DATA" is not imported at ./test2.prg line 38.
Global symbol "$log" requires explicit package name at ./test2.prg lin
+e 6.
Global symbol "$log" requires explicit package name at ./test2.prg lin
+e 8.
Global symbol "$log" requires explicit package name at ./test2.prg lin
+e 8.
Global symbol "$ip" requires explicit package name at ./test2.prg line
+ 14.
Global symbol "$port" requires explicit package name at ./test2.prg li
+ne 14.
Global symbol "$user_pass" requires explicit package name at ./test2.p
+rg line 19.
Global symbol "$status" requires explicit package name at ./test2.prg 
+line 19.
Global symbol "%HONEY" requires explicit package name at ./test2.prg l
+ine 20.
Global symbol "$ip" requires explicit package name at ./test2.prg line
+ 20.
Global symbol "$port" requires explicit package name at ./test2.prg li
+ne 20.
Global symbol "$status" requires explicit package name at ./test2.prg 
+line 20.
Global symbol "$user_pass" requires explicit package name at ./test2.p
+rg line 20.
Global symbol "$ip" requires explicit package name at ./test2.prg line
+ 21.
Global symbol "$port" requires explicit package name at ./test2.prg li
+ne 21.
Global symbol "$status" requires explicit package name at ./test2.prg 
+line 21.
Global symbol "$user_pass" requires explicit package name at ./test2.p
+rg line 21.
Global symbol "%HONEY" requires explicit package name at ./test2.prg l
+ine 25.
Global symbol "%HONEY" requires explicit package name at ./test2.prg l
+ine 26.
Global symbol "%HONEY" requires explicit package name at ./test2.prg l
+ine 27.
Global symbol "%HONEY" requires explicit package name at ./test2.prg l
+ine 28.
Global symbol "%HONEY" requires explicit package name at ./test2.prg l
+ine 29.
Global symbol "$freq" requires explicit package name at ./test2.prg li
+ne 30.
Global symbol "%HONEY" requires explicit package name at ./test2.prg l
+ine 30.
Global symbol "@DATA" requires explicit package name at ./test2.prg li
+ne 31.
Global symbol "$freq" requires explicit package name at ./test2.prg li
+ne 31.
Global symbol "@DATA" requires explicit package name at ./test2.prg li
+ne 38.
Execution of ./test2.prg aborted due to compilation errors.

      [download]
[reply]
[d/l]
[select]

        Hello firepro20,

        Unfortunately nothing was outputted on the screen.

        A big part of the problem here is simply the layout, which makes it hard to see the structure of the code. With a bit of indentation:

        $log = "/home/tsec/prototype/logs/extractedlogs/cowrieresult.log";

open(DATA, $log) or die "Can't open '$log': $!";

sub tester()
{
    while(<DATA>){
        ...
    }

    for my $ip (keys %HONEY){
        ...
    }

    for my $data (sort @DATA){
        ...
    }
}

        [download]

        See the problem now? sub tester is defined, but never called! You have to invoke the sub:

        my $log = '/home/tsec/prototype/logs/extractedlogs/cowrieresult.log';
open(my $data, '<', $log) or die "Can't open '$log' for reading: $!";
tester($data);
close  $data              or die "Can't close '$log': $!";

sub tester
{
    my ($fh) = @_;

    while (<$fh>) {
        ...
    }
    ...
}

        [download]

        and then you should see the output you were expecting. Some notes:

        • Prefer the 3-argument form of open (it’s clearer and safer).
        • For filehandles, prefer lexical variables to barewords. Also, avoid using DATA for files, because this will confuse readers of your code: they will expect it to have its pre-defined meaning.
        • Pass arguments to subroutines explictly via @_.
        • Don’t use subroutine prototypes unless you have a good reason. In sub tester() the () is a prototype specifying that the sub takes no arguments.

        Hope that helps,

        Athanasius <°(((>< contra mundum Iustus alius egestas vitae, eros Piratica,

[reply]
[d/l]
[select]

        So, what did you try to fix those errors? Hint: my.

        Alexander

        --
        Today I will gladly share my knowledge and experience, for there are no sweeter words than "I told you so". ;-)
[reply]
Re: Increment frequency of attempts based on IP and login details combination
by Discipulus (Canon) on Apr 30, 2016 at 21:39 UTC
    As a side note (because i do understand the usefulness of your desired output) some general suggestions: 
    $ip = "";
        $port = "";
        $usr = "";
        $port = "";
        $status = "";
        $frequency = 0;

    [download]
    is unneeded and error prone because reveals the absence of use strict; use warnings; In fact in a more concise, more robust, less error prone way you can say: 
    my ($ip,$port,$port,$status,$frequency);

    [download]

    Read this post to understand why. In Perl, given your variable declaration, you can freely do the following:

            #string?       #number?  #add to string?  #concatenate to numb
+ers?
 my $ip=""; my $freq = 0;        $ip++;          $freq.="still no data
+";

    [download]

    L*

    There are no rules, there are no thumbs..
    Reinvent the wheel, then learn The Wheel; may be one day you reinvent one of THE WHEELS.
[reply]
[d/l]
[select]
Re: Increment frequency of attempts based on IP and login details combination
by Laurent_R (Canon) on Apr 30, 2016 at 14:10 UTC
    Hi firepro20,

    I would use a hash or HoH (keys: IP, port, user/pwd and status) to record the number of connections, and output the result either when finished with one given IP, or completely at the end, depending on whether you want to count once or several times the same IP coming with some interval.

[reply]

Back to Seekers of Perl Wisdom