Match a Log Entry Only If a Certain Keyword is not Present

Dru has asked for the wisdom of the Perl Monks concerning the following question:

Hello Monks,

I am trying to match a log file when it contains an entry with the "sent" action but not the "created" action regardless if the "sent" action is also logged. In my below example 192.168.7.16 would match because there is a log entry for "sent" but 192.168.66.176 would not because even though there is a log entry for "sent" there is also an entry for "created."

Does someone know how I could go about doing this?

#!/usr/bin/perl

use strict;
use warnings;

my %hash;

while(<DATA>){
  my ($srcip, $action) = (split /\s+/)[1,4];
  $action = $1 if ($action =~ /\[\d+\](\w+)/);
  $hash{$srcip}{action} = $action;
}

for (keys %hash){
  print "$_ == $hash{$_}{action}\n";
}

__DATA__
ex100525.log:09:42:26 192.168.66.176 webcountry 192.168.0.166 [5933]cr
+eated /140NOE77111_V460_+IE38/FTP+script/put771.ftp 226 0
ex100525.log:09:42:27 192.168.66.176 webcountry 192.168.0.166 [5933]cr
+eated /140NOE77111_V460_+IE38/FTP+script/update_noe77111_module.doc 2
+26 0
ex100525.log:09:42:27 192.168.66.176 webcountry 192.168.0.166 [5933]cr
+eated /140NOE77111_V460_+IE38/FTP+script/upfwnoe.bat 226 0
ex100525.log:09:42:27 192.168.66.176 webcountry 192.168.0.166 [5933]CW
+D /140NOE77111_V460_+IE38/Release+Note 550 2
ex100525.log:09:42:27 192.168.66.176 webcountry 192.168.0.166 [5933]CW
+D /140NOE77111_V460_+IE38/Release+Note 250 0
ex100525.log:09:42:27 192.168.66.176 webcountry 192.168.0.166 [5933]se
+nt /140NOE77111_V460_+IE38/Release+Note/RN_140NOE77111_V46.doc 226 0
ex100525.log:09:42:27 192.168.7.16 webcountry 192.168.0.166 [5933]sent
+ /140NOE77111+V4.6/140NOE77111_V460_+IE38 250 0
ex100525.log:09:42:27 192.168.7.16 webcountry 192.168.0.166 [5933]CWD 
+/140NOE77111+V4.6/140NOE77111_V460_+IE38 250 0
[download]

Thanks,
Dru

Perl, the Leatherman of Programming languages. - qazwart

Comment on Match a Log Entry Only If a Certain Keyword is not Present Download Code

Replies are listed 'Best First'.
Re: Match a Log Entry Only If a Certain Keyword is not Present by eff_i_g (Curate) on Dec 08, 2010 at 04:07 UTC
#!/usr/bin/perl use strict; use warnings; use feature 'switch'; use Data::Dumper; my %ok; my %not_ok; while (<DATA>) { my ($src_ip, $action) = (split /\s+/)[1,4]; if ($action =~ /\[\d+\](\w+)/) { given ($1) { when ('sent') { ++$ok{$src_ip} unless exists $not_ok{$src_ip}; } when ('created') { delete $ok{$src_ip} if exists $ok{$src_ip}; ++$not_ok{$src_ip}; } } } } print Dumper(\%ok, \%not_ok); __DATA__ ex100525.log:09:42:26 192.168.66.176 webcountry 192.168.0.166 [5933]cr +eated /140NOE77111_V460_+IE38/FTP+script/put771.ftp 226 0 ex100525.log:09:42:27 192.168.66.176 webcountry 192.168.0.166 [5933]cr +eated /140NOE77111_V460_+IE38/FTP+script/update_noe77111_module.doc 2 +26 0 ex100525.log:09:42:27 192.168.66.176 webcountry 192.168.0.166 [5933]cr +eated /140NOE77111_V460_+IE38/FTP+script/upfwnoe.bat 226 0 ex100525.log:09:42:27 192.168.66.176 webcountry 192.168.0.166 [5933]CW +D /140NOE77111_V460_+IE38/Release+Note 550 2 ex100525.log:09:42:27 192.168.66.176 webcountry 192.168.0.166 [5933]CW +D /140NOE77111_V460_+IE38/Release+Note 250 0 ex100525.log:09:42:27 192.168.66.176 webcountry 192.168.0.166 [5933]se +nt /140NOE77111_V460_+IE38/Release+Note/RN_140NOE77111_V46.doc 226 0 ex100525.log:09:42:27 192.168.7.16 webcountry 192.168.0.166 [5933]sent + /140NOE77111+V4.6/140NOE77111_V460_+IE38 250 0 ex100525.log:09:42:27 192.168.7.16 webcountry 192.168.0.166 [5933]CWD +/140NOE77111+V4.6/140NOE77111_V460_+IE38 250 0 [download]	[reply] [d/l]
Re: Match a Log Entry Only If a Certain Keyword is not Present by Marshall (Canon) on Dec 08, 2010 at 04:22 UTC
You want to know all ip's for which the sent status occured, but not the created status. Here is one easy way. Use a bit for "created" and a bit for "sent" as values in a hash keyed on ip address. Print that key if only the "sent status" bit is set ("created" never occurred). The scheme below can be expanded to any combination of "sent" and "created" that you need although the logic needs to change a bit if you want ip's that never had either "sent" or "created". #!/usr/bin/perl use strict; use warnings; my %hash; my $sent_bits = 1; # binary 0001 my $created_bits = 2; # binary 0010 while(<DATA>) { next if /^\s*$/; #skip blank lines my ($srcip, $action_field) = (split /\s+/,$_)[1,4]; (my $action) = $action_field =~ m/^\[\d+\](\w+)/; $hash{$srcip} \|= $sent_bits if ($action eq "sent"); $hash{$srcip} \|= $created_bits if ($action eq "created"); } foreach my $sent_or_created (keys %hash) { print "$sent_or_created\n" if ($hash{$sent_or_created} == $sent_bits); } #prints: 192.168.7.16 __DATA__ ex100525.log:09:42:26 192.168.66.176 webcountry 192.168.0.166 [5933]cr +eated /140NOE77111_V460_+IE38/FTP+script/put771.ftp 226 0 ex100525.log:09:42:27 192.168.66.176 webcountry 192.168.0.166 [5933]cr +eated /140NOE77111_V460_+IE38/FTP+script/update_noe77111_module.doc 2 +26 0 ex100525.log:09:42:27 192.168.66.176 webcountry 192.168.0.166 [5933]cr +eated /140NOE77111_V460_+IE38/FTP+script/upfwnoe.bat 226 0 ex100525.log:09:42:27 192.168.66.176 webcountry 192.168.0.166 [5933]CW +D /140NOE77111_V460_+IE38/Release+Note 550 2 ex100525.log:09:42:27 192.168.66.176 webcountry 192.168.0.166 [5933]CW +D /140NOE77111_V460_+IE38/Release+Note 250 0 ex100525.log:09:42:27 192.168.66.176 webcountry 192.168.0.166 [5933]se +nt /140NOE77111_V460_+IE38/Release+Note/RN_140NOE77111_V46.doc 226 0 ex100525.log:09:42:27 192.168.7.16 webcountry 192.168.0.166 [5933]sent + /140NOE77111+V4.6/140NOE77111_V460_+IE38 250 0 ex100525.log:09:42:27 192.168.7.16 webcountry 192.168.0.166 [5933]CWD +/140NOE77111+V4.6/140NOE77111_V460_+IE38 250 0 [download]	[reply] [d/l]
Re: Match a Log Entry Only If a Certain Keyword is not Present by anonymized user 468275 (Curate) on Dec 08, 2010 at 22:51 UTC
to amend as little as possible, you could make grep ignore both entries if created is found `for my $ok(grep !defined( $hash{ $_ }{ created }), keys( %hash )){ print "$ok == $hash{$ok}{action}\n"; }` [download] One world, one people	[reply] [d/l]
Re: Match a Log Entry Only If a Certain Keyword is not Present by Marshall (Canon) on Dec 10, 2010 at 09:12 UTC
Ok, another solution that more closely follows the OP's code is below: The action field is like: "`[5933]created`", so the regex can be simplified to just capture the first sequence of A-Za-z in the $action_field. In the version below, the %hash now counts the number of occurrences of each action related to each ip address. Printing an IP address that was sent, but not created is a straightforward "if (..)" statement. #!/usr/bin/perl use strict; use warnings; my %hash; while(<DATA>) { next if /^\s*$/; #skip blank lines (optional) my ($srcip, $action_field) = (split /\s+/,$_)[1,4]; (my $action) = $action_field =~ m/([A-Za-z]+)/; $hash{$srcip}{$action}++; } foreach my $ip (keys %hash) { print "$ip\n" if ( $hash{$ip}{'sent'} and !$hash{$ip}{'created'} ); } #prints: 192.168.7.16 __DATA__ ex100525.log:09:42:26 192.168.66.176 webcountry 192.168.0.166 [5933]cr +eated /140NOE77111_V460_+IE38/FTP+script/put771.ftp 226 0 ex100525.log:09:42:27 192.168.66.176 webcountry 192.168.0.166 [5933]cr +eated /140NOE77111_V460_+IE38/FTP+script/update_noe77111_module.doc 2 +26 0 ex100525.log:09:42:27 192.168.66.176 webcountry 192.168.0.166 [5933]cr +eated /140NOE77111_V460_+IE38/FTP+script/upfwnoe.bat 226 0 ex100525.log:09:42:27 192.168.66.176 webcountry 192.168.0.166 [5933]CW +D /140NOE77111_V460_+IE38/Release+Note 550 2 ex100525.log:09:42:27 192.168.66.176 webcountry 192.168.0.166 [5933]CW +D /140NOE77111_V460_+IE38/Release+Note 250 0 ex100525.log:09:42:27 192.168.66.176 webcountry 192.168.0.166 [5933]se +nt /140NOE77111_V460_+IE38/Release+Note/RN_140NOE77111_V46.doc 226 0 ex100525.log:09:42:27 192.168.7.16 webcountry 192.168.0.166 [5933]sent + /140NOE77111+V4.6/140NOE77111_V460_+IE38 250 0 ex100525.log:09:42:27 192.168.7.16 webcountry 192.168.0.166 [5933]CWD +/140NOE77111+V4.6/140NOE77111_V460_+IE38 250 0 [download]	[reply] [d/l] [select]