print "\t=> Bad user!\n" if  @{$searches{$conn}} > 3;
[download]

In your dataset, there is no one with more than 3 searches. If you do not like the other output, remove the print statements. If you want to check for the total number of searches, you need to sum up across connections like this:

for my $user (keys %users) {
  my $totalsearches=0;
  for my $conn (@{$users{$user}}) {
    if( exists $searches{$conn} ) {
      print "User $user had ".scalar( @{$searches{$conn}} )." searches
+ on connection $conn\n";
      $totalsearches +=  @{$searches{$conn}};
    }
  }
  print "User $user had $totalsearches searches in total\n";
  print "\t=> Bad user!\n" if $totalsearches > 3;
}
[download]

If you only want a total count the code can be simplified ;

use strict;
use warnings;

my %users;
my %conn;

while (<DATA>) { # I use DATA handle instead of $fh for convenience
  if( /BIND/ ) {
    my( $conn, $uid ) = /conn=(\d+).*uid=(.*?),/;
    $conn{$conn} = $uid;
  }
  if( /SRCH=Q/ ) {
    my ($timestamp, $conn) = /\[(.*?)\] conn=(\d+)/;
    my $uid = $conn{$conn}; 
    ++$users{$uid};
  }
}

for my $uid (keys %users) {
  my $count = $users{$uid};
  print "\t=> Bad user $uid ! count = $count\n" if  $count>3;
}
[download]

poj

Actually, there is some issue with regex, where I am putting /^BIND$/ to match exact word, as there is another string "UNBIND".

is this not the right way to match exact string /^BIND$/? Please let me know if it is not.

/^BIND$/ matches if that is the only word on the line.


Try adding word boundaries \b like this ;

while (<DATA>){
  if (/\bBIND\b/){
    print $_;
  }
}
__DATA__
this is a BIND sentence
I have UNBIND in this one
BIND as first word
last word is BIND
[download]


poj

BIND seems to be surrounded by spaces, so /\sBIND\s/ should work. Your suggestion would only fit a string (or line) containing BIND and nothing else as the caret is signifying the start of the string (or line) and dollar the end of string (or line).

Thank You, both works ( /b and /s). One last query related to this post, I would like to add time search, where the time has following format :

[04/Jun/2013:15:06:13
[download]

I would like to scan the log for last one hour only whenever I run the script. Thanks for all your effort.

Remove the extra print statements if you are happy it works as expected.

use strict;
use warnings;

## add this
use DateTime::Format::Strptime qw(strptime);
my $NOW = DateTime->now();
my $MAX_AGE = 60;
print "Time now is $NOW\n";
##

my %users;
my %conn;

while (<DATA>) { # I use DATA handle instead of $fh for convenience
 if( /\bBIND\b/ ) {
  my( $conn, $uid ) = /conn=(\d+).*uid=(.*?),/;
    $conn{$conn} = $uid;
  }
  if( /SRCH=Q/ ) {
    my ($timestamp, $conn) = /\[(.*?)\] conn=(\d+)/;
    
    ## add this
    my $dt = strptime('%d/%B/%Y:%T',substr($timestamp,0,20));
    my $age = $dt->delta_ms( $NOW )->in_units('minutes');
    print $timestamp." = " .$dt->datetime." ; $age mins\n";
    ##
    
    ## add condition
    if ($age >= $MAX_AGE){
      my $uid = $conn{$conn}; 
      ++$users{$uid};
    }
  }
}


for my $uid (keys %users) {
  my $count = $users{$uid};
  print "\t=> Bad user $uid ! count = $count\n" if  $count > 3;
}
[download]

poj

Not ideal (not exactly 60 mins 0 seconds and won't work either side of midnight) but try this

#!perl
use strict;
use warnings;

## add this
my $MAX_AGE = 60; # minutes
# calc cut off date time
my @mth = qw(Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec);
my @f = localtime();
my $TODAY = sprintf "%02d/%s/%4d",$f[3],$mth[$f[4]],$f[5]+1900; 
my $START_MINUTE = $f[2]*60+$f[1] - $MAX_AGE;
print "Now is $TODAY $START_MINUTE mins\n";
##

my %users;
my %conn;

while (<DATA>) { # I use DATA handle instead of $fh for convenience
 if( /\bBIND\b/ ) {
  my( $conn, $uid ) = /conn=(\d+).*uid=(.*?),/;
    $conn{$conn} = $uid;
  }
  if( /SRCH=Q/ ) {
    my ($timestamp, $conn) = /\[(.*?)\] conn=(\d+)/;
    
    ## add this
    my ($date,$h,$m,undef) = split ':',$timestamp,4;
    next unless ($date eq $TODAY);
    my $minutes = $h*60 + $m;
    print $timestamp." = $date $h $m ; $minutes <=> $START_MINUTE\n";
    ## 
    
    # add condition
    if ($minutes >= $START_MINUTE){
      my $uid = $conn{$conn}; 
      ++$users{$uid};
    }
  }
}


for my $uid (keys %users) {
  my $count = $users{$uid};
  print "\t=> Bad user $uid ! count = $count\n" if  $count > 3;
}
[download]

poj

I see that you've changed the logic, can you tell me know how to just disable the time part, so that I can run the script to verify the main logic.

OK, comment out this line ;

## next unless ($date eq $TODAY);
[download]

and the if

 # add condition
 ##   if ($minutes >= $START_MINUTE){
      my $uid = $conn{$conn}; 
      ++$users{$uid};
 ##   }
[download]

poj

Hi Poj,

I tried running the code by commenting the rows as suggested, however I do not get the result ( ie bad user) Here is the log, can you try from your side, you must see bad user getting printed.

[04/Jun/2013:13:06:13 -0600] conn=13570 op=14 msgId=13 - BIND dn="uid=
+xyz123,ou=People,o=xyz.com" method=128 version=3
[04/Jun/2013:15:06:13 -0600] conn=13570 op=14 msgId=15 - RESULT err=0 
+tag=101 nentries=48030 etime=139 SRCH=Q
[04/Jun/2013:13:06:13 -0600] conn=13572 op=14 msgId=13 - BIND dn="uid=
+xyz123,ou=People,o=xyz.com" method=128 version=3
[04/Jun/2013:15:06:13 -0600] conn=13572 op=14 msgId=15 - RESULT err=0 
+tag=101 nentries=48030 etime=139 SRCH=Q
[04/Jun/2013:17:06:13 -0600] conn=13571 op=14 msgId=13 - BIND dn="uid=
+someoneelse,ou=People,o=xyz.com" method=128 version=3
[04/Jun/2013:18:06:17 -0600] conn=13571 op=14 msgId=15 - RESULT err=0 
+tag=101 nentries=48030 etime=139 SRCH=Q
[04/Jun/2013:13:06:13 -0600] conn=13573 op=14 msgId=13 - BIND dn="uid=
+xyz123,ou=People,o=xyz.com" method=128 version=3
[04/Jun/2013:15:06:13 -0600] conn=13573 op=14 msgId=15 - RESULT err=0 
+tag=101 nentries=48030 etime=139 SRCH=Q
[04/Jun/2013:13:06:13 -0600] conn=13574 op=14 msgId=13 - BIND dn="uid=
+xyz123,ou=People,o=xyz.com" method=128 version=3
[04/Jun/2013:15:06:13 -0600] conn=13574 op=14 msgId=15 - RESULT err=0 
+tag=101 nentries=48030 etime=139 SRCH=Q
[04/Jun/2013:13:06:13 -0600] conn=13575 op=14 msgId=13 - BIND dn="uid=
+someone,ou=People,o=xyz.com" method=128 version=3
[04/Jun/2013:15:06:13 -0600] conn=13575 op=14 msgId=15 - RESULT err=0 
+tag=101 nentries=48030 etime=139
[download]

I am only getting print statement for time, here is the output, i should get number of counts, and print statement as bad user.

Now is 17/Jun/2013 404 mins
17/Jun/2013:06:12:37 -0600 = 17/Jun/2013 06 12 ; 372 <=> 404
17/Jun/2013:06:32:34 -0600 = 17/Jun/2013 06 32 ; 392 <=> 404
17/Jun/2013:06:32:46 -0600 = 17/Jun/2013 06 32 ; 392 <=> 404
17/Jun/2013:06:32:55 -0600 = 17/Jun/2013 06 32 ; 392 <=> 404
17/Jun/2013:06:33:28 -0600 = 17/Jun/2013 06 33 ; 393 <=> 404
[download]

See what you get with the count condition commented out like this ;

print "\t=> Bad user $uid ! count = $count\n";# if  $count > 3;

Or change the value from 3 to 0

poj

I regret, the script does print count, however the issue is that there are lot of logs, is there any way to put the hashes within file db, so that it is not consuming memory?

I doubt if memory usage is your problem. How many is "a lot" and how big are the logs, i.e. how many lines in the biggest ?

poj

I took the prod log, an hour log is of 200 MBs. Is that low/high?

To be on safer side, I would like to dump the search onto the disk, and then perform the searches ( ie bind etc)

Not quite sure what you mean but assuming you have a number of logs all in one folder, then this script will scan them and extract only the records you are interested in into a single file which you can process with the 'bad user' script.

#!perl
use strict;

# start time
my $t0 = time();

# input - create list of files
my @files = grep { /.log/ } glob("*") ;

# output
my $count_out = 0;
my $outfile  = 'output.log';
open OUT,'>',$outfile or die "Could not open $outfile : $!";

# process
for my $infile (@files){
  next if ($infile eq $outfile);
  open IN,'<',$infile or die "Could not open $infile : $!";
  print "Reading $infile .. ";
  my $count_in = 0;
  while (<IN>){
    ++$count_in;
    if (/BIND|SRCH=Q/){
      print OUT;
      ++$count_out;
    }
  }
  close IN;
  print "$count_in records read\n";
}

# finish
close OUT;
my $dur = time() - $t0;
print "Finished in $dur secs \n $count_out written to $outfile\n";
[download]

poj

Thanks Poj.

Actually, my intent is to dump the content of hash into disk instead of memory. Anyway, I will try to utilize the provide script. However, can you put hourly logic into this as you had before?