Compare two log files line by line containing a keyword

rahu_6697 has asked for the wisdom of the Perl Monks concerning the following question:

Replies are listed 'Best First'.
Re: Compare two log files line by line containing a keyword by Discipulus (Canon) on Feb 19, 2019 at 08:27 UTC
Hello rahu_6697, I'm not sure I've understood your question, but time will tell.. First of all I think you need some advice on syntax: in order of importance `@arr & @arr2` `perl -E "@a1=qw(1 2 3); @a2=qw(a b c d e); say for (@a1 & @a2)" 1 perl -E "@a1=qw(1 2 3); @a2=qw(a b c d e); say for (@a1 , @a2)" 1 2 3 a b c d e` [download] Infact to concatenate two list in list context the comma is used not `&` see `perlop` Then your way to open filehandle, while running, is oldish and error prone: `open(FILE, "<x.log"); close (FILE); # has to be open my $file_handle, '<', 'x.log' or die "unable to open 'x.log' $!"; .. close $file_handle;` [download] Third, you are reassigning twice to `@array2 = grep ..` and, as long as you print to the destination file immediately, can also be what your intended, but is not clear and error prone in long distance: `grep` in list context will returns a list and you can use `push` to extend the array: `my @array; ... #push 1 push @array, $_ for grep{.. #push 2 push @array, $_ for grep{..` [download] Also please use `<code> ..code tags.. </code>` also for your data input/output PS After reading twice, please clearify the following: > Now I want to compare the data present after the keyword following colon(:) and in output file I have to print the mismatched lines from first data-set on comparing it with second data-set and number of lines in second data set that are not present in first data-set. If I understand what you are asking for I'd go with a datastructure like a hash of arrays, which keys will be all keywords occurences and array element populed by the evntual presence in file 1 and file 2: # pseudocode my %report; foreach my $line( $fh_one ){ if... matching.. extract the part interesting $report{ $interesting_part }[0] = 1; # first file populates the +element 0 foreach my $line( $fh_two ){ if... matching.. extract the part interesting $report{ $interesting_part }[1] = 1; # first file populates the +element 1 # result like %report = ( interesting1 = [ undef, 1 ], # is present only in the second file interesting2 = [ 1, undef ], # is present only in the first file interesting3 = [ 1, 1 ], # is present in both files ); [download] L* There are no rules, there are no thumbs.. Reinvent the wheel, then learn The Wheel; may be one day you reinvent one of THE WHEELS.	[reply] [d/l] [select]
Re^2: Compare two log files line by line containing a keyword by rahu_6697 (Novice) on Feb 19, 2019 at 09:45 UTC
I think I failed to explain my problem. The task is I have two log files (x.log and y.log) I have to search through them line by line and have to compare only those lines which contains a string ("mti_clk_chk") in between. lets take an example If my data set is like this. LOG 1 (x.log) contains INFO @1102266 PHResourceLayer_Z4: mti_clk_chk:################ start of test ################ ; T=1102266 INFO @1102334 PHResourceLayer_Z4: mti_clk_chk:Checking the period of MTI, MTI10 clk from SV; T=1102334 LOG 2 (y.log) contains UVM_INFO @1092507 reporter Z4_COREA: mti_clk_chk: ################ start of test ################ ; T=1092507 UVM_INFO @1092563 reporter Z4_COREA: mti_clk_chk: Checking the period of MTI, MTI10 clk from SV; T=1092563 Then for first line I have to check "################ start of test ################ ; T=1102266" and "################ start of test ################ ; T=1092507" as value of T are not same so it should give these details in output file stating the details are not matching. Similiarly for line 2 we have to match "Checking the period of MTI, MTI10 clk from SV; T=1102334" and "Checking the period of MTI, MTI10 clk from SV; T=1092563". Here also values of T are not matching so pass it to output file. (For each line in the logs, every details after "mti_clk_chk:" needs to be checked )	[reply]
Re^3: Compare two log files line by line containing a keyword by thanos1983 (Parson) on Feb 19, 2019 at 11:50 UTC
Hello rahu_6697, Based on your updated description something like that should work: #!/usr/bin/perl use strict; use warnings; use IO::All; use Data::Dumper; use List::Compare; my @x = io('x.log')->chomp->slurp; my @x_start = grep /mti_clk_chk/, @x && grep /start of test/, @x; my @x_period = grep /mti_clk_chk/, @x && grep /Checking the period/, @ +x; print Dumper \@x_start; # print Dumper \@x_period; my @y = io('y.log')->chomp->slurp; my @y_start = grep /mti_clk_chk/, @x && grep /start of test/, @y; my @y_period = grep /mti_clk_chk/, @x && grep /Checking the period/, @ +y; print Dumper \@y_start; # print Dumper \@y_period; my $lcStart = List::Compare->new('-u', \@x_start, \@y_start); my @LorRonlyStart = $lcStart->get_symmetric_difference; print Dumper \@LorRonlyStart; io('start.log')->appendln(@LorRonlyStart); my $lcPeriod = List::Compare->new('-u', \@x_period, \@y_period); my @LorRonlyPeriod = $lcPeriod->get_symmetric_difference; print Dumper \@LorRonlyPeriod; io('period.log')->appendln(@LorRonlyPeriod); [download] But I am not really sure if this is what you want as from my point of view you are comparing wrong data. I mean the lines that you provide us as sample there are not the same not just by T=<TIME> but also as syntax. For example you are comparing: `INFO @1102266 PHResourceLayer_Z4: mti_clk_chk:################ start o +f test ################ ; T=1102266 UVM_INFO @1092507 reporter Z4_COREA: mti_clk_chk: ################ sta +rt of test ################ ; T=1092507` [download] The lines they do not have the same "@1102266" and "@1092507" so how you are going to be sure you are comparing the correct lines? On the sample of code that I provided you above I am grepping lines with two parameters (I assume this should be enough). The problem appears when you are going to compare the arrays. Based on your description the T=<TIME> can be different but from my point of view this is not enough, you might need to add more parameters to be sure you are comparing the correct lines. If so update your answer so we can try to help you further. Hope this helps, BR Seeking for Perl wisdom...on the process of learning...not there...yet!	[reply] [d/l] [select]
Re^3: Compare two log files line by line containing a keyword by BillKSmith (Monsignor) on Feb 19, 2019 at 14:51 UTC
Your second example suggests that the log files are the same except for "details". Perhaps you want to examine the first line of each file. If they contain your "keyword", report the difference between them, otherwise do nothing. Repeat for the second line of each file, then the third line of each file, etc. Even if this simplification is true, you still have not explained what kind of differences to expect or how to format them. Bill	[reply]
Re^3: Compare two log files line by line containing a keyword by poj (Abbot) on Feb 19, 2019 at 11:12 UTC
Do the strings `'################ start of test ################'` and `'Checking the period of MTI, MTI10 clk from SV'` appear only once in the files ? poj	[reply] [d/l] [select]
Re^4: Compare two log files line by line containing a keyword by rahu_6697 (Novice) on Feb 20, 2019 at 09:28 UTC
Re: Compare two log files line by line containing a keyword by Marshall (Canon) on Feb 19, 2019 at 18:45 UTC
I am very confused as to what the desired result is? Your question is not clear, at least to me. BTW, when you present data, please use `<code>..</code>` tags. At this point, I didn't spend time worrying about how to improve your code that generates the 2 arrays. Before doing that, I would like to understand the end objective. To try to understand the requirements, I wrote some simple code to replicate your @array and @array1. Then I reduced those arrays down to just the last part of the line that includes the 'multi_clk_chk:' stuff. I take it that only the stuff after 'multi_clk_chk:' matters? See below and please explain what a comparison between Data Set A and Data Set B would be? I suppose that these log files contain many records? Of which you are just showing one of each? A first step might be to figure how to figure out how to select a record from X and a record from Y? The difference in the T values seems to be what you really want? It could be that a hash table is the right kind of data structure for each record? text before the T value as the key and T number as the value? Would order matter in the report? `stuff before the T value => T value A: MTI Clock is being generated correctly ; => T=1102377 B: MTI Clock is being generated correctly ; => T=1092605 perhaps result: MTI Clock is being generated correctly ; T=1102377 T=1092605` [download] I'm sure you will get lots of interesting implementation ideas if you can explain the requirements better. Update: Please think about how you want the results to be presented to the user. General file compares can generate some confusing looking stuff. A good UI can be important. For example I am working on a feature request on a different project.. I understood instantly how to calculate the new desired result (the math was easy)...but, I spent an hour thinking about how to present this new data to the user in way that was not confusing and so that they'd understand what this information meant. Another Update: I went ahead and hacked in some additional code for the "use a hash for comparison" idea. Still not sure what the OP wants, but this might be useful. No consideration for efficiency was made - that can be optimized later once the problem is understood. #!/usr/bin/perl; use strict; use warnings; use Data::Dump qw(pp); # initialize @array and @array1 to what the OP describes as his curren +t "output"..... my @array =<<'EOF' =~ m/(^.\n)/mg; INFO @576892 mti_clk_chk: run_stimulus called; T=576892 INFO @1102266 PHResourceLayer_Z4: mti_clk_chk:################ start o +f test ################ ; T=1102266 INFO @1102334 PHResourceLayer_Z4: mti_clk_chk:Checking the period of M +TI, MTI10 clk from SV; T=1102334 INFO @1102372 mti_clk_chk: Checking period of MTI CLk; T=1102372 INFO @1102377 mti_clk_chk: Period value of MTI Clock: 3.125000 ns; T=1 +102377 INFO @1102377 mti_clk_chk: MTI Clock is being generated correctly ; T= +1102377 INFO @1102377 mti_clk_chk: Checking period of MTI10 CLk; T=1102377 INFO @1102418 mti_clk_chk: Period value of MTI10 Clock: 31.250000 ns; +T=1102418 INFO @1102418 mti_clk_chk: MTI10 Clock is being generated correctly ; +T=1102418 INFO @1102717 PHResourceLayer_Z4: mti_clk_chk: All clock period Checki +ng done; T=1102717 INFO @1148661 mti_clk_chk: C-Code exit execution. code=<aa>; T=1148661 INFO @1148661 mti_clk_chk: ********************* SV END******** +****** ; T=1148661 EOF my @array1 =<<'EOF2' =~ m/(^.\n)/mg; UVM_INFO @0 reporter testbench.top_level_module.\mti_clk_chk::main : M +TI_CLK_CHK_STIM Started .....; T=0 UVM_INFO @0 reporter testbench.top_level_module.\mti_clk_chk::main : r +un_stimulus called; T=0 UVM_INFO @1092507 reporter Z4_COREA: mti_clk_chk: ################ sta +rt of test ################ ; T=1092507 UVM_INFO @1092563 reporter Z4_COREA: mti_clk_chk: Checking the period +of MTI, MTI10 clk from SV; T=1092563 UVM_INFO @1092598 reporter testbench.top_level_module.\mti_clk_chk::ma +in : Checking period of MTI CLk; T=1092598 UVM_INFO @1092605 /proj/rru2_verif/usr/Tilak/SV_UVM/testbench/data_ipd +ss/v_ms_mti_stim_vip/testbench/classes_v/mti_clk_chk.sv(147) uvm_test +_top.default_env.default_sequencer100@@mti_clk_chk mti_clk_chk:INFO: +Period value of MTI Clock: 3.125000 ns; T=1092605 UVM_INFO @1092605 reporter testbench.top_level_module.\mti_clk_chk::ma +in : MTI Clock is being generated correctly ; T=1092605 UVM_INFO @1092605 reporter testbench.top_level_module.\mti_clk_chk::ma +in : Checking period of MTI10 CLk; T=1092605 UVM_INFO @1092655 /proj/rru2_verif/usr/Tilak/SV_UVM/testbench/data_ipd +ss/v_ms_mti_stim_vip/testbench/classes_v/mti_clk_chk.sv(165) uvm_test +_top.default_env.default_sequencer100@@mti_clk_chk mti_clk_chk:INFO: +Period value of MTI10 Clock: 31.250000 ns; T=1092655 UVM_INFO @1092655 reporter testbench.top_level_module.\mti_clk_chk::ma +in : MTI10 Clock is being generated correctly ; T=1092655 UVM_INFO @1092850 reporter Z4_COREA: mti_clk_chk: All clock period Che +cking done; T=1092850 UVM_INFO @1092886 /proj/rru2_verif/usr/Tilak/SV_UVM/testbench/data_ipd +ss/v_ms_mti_stim_vip/testbench/classes_v/mti_clk_chk.sv(186) uvm_test +_top.default_env.default_sequencer100@@mti_clk_chk mti_clk_chk:INFO: +********************* SV END****************** ; T=1092886 EOF2 # evidently all that matters is stuff after the 'mti_clk_chk:' string # so remove all before that critical stuff @array = map{/^.(mti_clk_chk:.\n)/;$1;}@array; @array1 = map{/^.(mti_clk_chk:.\n)/;$1;}@array1; ## just print these arrays out so that we can see how to ## compare them? my $n=1; foreach (@array) {print "A$n) ",$_; $n++} print "===============================\n"; $n=1; foreach (@array1) {print "B$n) ",$_; $n++} ## Make hash tables of the 2 records ## regex could be better - not the point here.. just a quick demo of a +n approach.. my %A = map{/mti_clk_chk:(?::main :\|INFO:)?\s(.).\sT=(\d+)$/;$1 => + $2}@array; my %B = map{/mti_clk_chk:(?::main :\|INFO:)?\s(.).\sT=(\d+)$/;$1 => + $2}@array1; #print only common values in sequential order of file A print "\n******************************************\n"; print "Common Text descriptions with A and B T values:\n"; print "*****************************************\n"; foreach my $mti (map{/mti_clk_chk:(?::main :\|INFO:)?\s(.).\sT=(\d+ +)$/;$1}@array) { if (exists $B{$mti}) { print "$mti T(a)=$A{$mti} T(b)=$B{$mti}\n"; } } __END__ A1) mti_clk_chk: run_stimulus called; T=576892 A2) mti_clk_chk:################ start of test ################ ; T=11 +02266 A3) mti_clk_chk:Checking the period of MTI, MTI10 clk from SV; T=11023 +34 A4) mti_clk_chk: Checking period of MTI CLk; T=1102372 A5) mti_clk_chk: Period value of MTI Clock: 3.125000 ns; T=1102377 A6) mti_clk_chk: MTI Clock is being generated correctly ; T=1102377 A7) mti_clk_chk: Checking period of MTI10 CLk; T=1102377 A8) mti_clk_chk: Period value of MTI10 Clock: 31.250000 ns; T=1102418 A9) mti_clk_chk: MTI10 Clock is being generated correctly ; T=1102418 A10) mti_clk_chk: All clock period Checking done; T=1102717 A11) mti_clk_chk: C-Code exit execution. code=<aa>; T=1148661 A12) mti_clk_chk: ********************* SV END**************** +; T=1148661 =============================== B1) mti_clk_chk::main : MTI_CLK_CHK_STIM Started .....; T=0 B2) mti_clk_chk::main : run_stimulus called; T=0 B3) mti_clk_chk: ################ start of test ################ ; T=1 +092507 B4) mti_clk_chk: Checking the period of MTI, MTI10 clk from SV; T=1092 +563 B5) mti_clk_chk::main : Checking period of MTI CLk; T=1092598 B6) mti_clk_chk:INFO: Period value of MTI Clock: 3.125000 ns; T=109260 +5 B7) mti_clk_chk::main : MTI Clock is being generated correctly ; T=109 +2605 B8) mti_clk_chk::main : Checking period of MTI10 CLk; T=1092605 B9) mti_clk_chk:INFO: Period value of MTI10 Clock: 31.250000 ns; T=109 +2655 B10) mti_clk_chk::main : MTI10 Clock is being generated correctly ; T= +1092655 B11) mti_clk_chk: All clock period Checking done; T=1092850 B12) mti_clk_chk:INFO: ******************** SV END************ + ; T=1092886 # Some of the issues involved in the comparison between sets: # Line B1 doesn't exist in set A # line A11 doesn't exist in set B # one plausible way to present the output? **************************************** Common Text descriptions with A and B T values: **************************************** run_stimulus called; T(a)=576892 T(b)=0 ################ start of test ################ ; T(a)=1102266 T(b)=1 +092507 Checking the period of MTI, MTI10 clk from SV; T(a)=1102334 T(b)=1092 +563 Checking period of MTI CLk; T(a)=1102372 T(b)=1092598 Period value of MTI Clock: 3.125000 ns; T(a)=1102377 T(b)=1092605 MTI Clock is being generated correctly ; T(a)=1102377 T(b)=1092605 Checking period of MTI10 CLk; T(a)=1102377 T(b)=1092605 Period value of MTI10 Clock: 31.250000 ns; T(a)=1102418 T(b)=1092655 MTI10 Clock is being generated correctly ; T(a)=1102418 T(b)=1092655 All clock period Checking done; T(a)=1102717 T(b)=1092850 ******************** SV END****************** ; T(a)=1148661 T( +b)=1092886 [download]	[reply] [d/l] [select]
Re^2: Compare two log files line by line containing a keyword by rahu_6697 (Novice) on Feb 20, 2019 at 06:10 UTC
You understood the question correctly and your output is quite good also, but I'm facing this issue while run. One thing more I want this output in a separate output file, please change the code accordingly. `Can't locate Data/Dump.pm in @INC (@INC contains: /usr/local/lib64/per +l5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/per +l5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .) at script3.pl lin +e 4. BEGIN failed--compilation aborted at script3.pl line 4` [download] Code I am trying to run is #!/usr/bin/perl; use strict; use warnings; use Data::Dump qw(pp); open(FILE, "<x.log"); my @array = <FILE>; close(FILE); open(FILE, "<y.log"); my @array1 = <FILE>; close(FILE); my @array =<<'EOF' =~ m/(^.\n)/mg; EOF my @array1 =<<'EOF2' =~ m/(^.\n)/mg; EOF2 @array = map{/^.(mti_clk_chk:.\n)/;$1;}@array; @array1 = map{/^.(mti_clk_chk:.\n)/;$1;}@array1; ## just print these arrays out so that we can see how to compare them? my $n=1; foreach (@array) {print "A$n) ",$_; $n++} print "===============================\n"; $n=1; foreach (@array1) {print "B$n) ",$_; $n++} ## Make hash tables of the 2 records ## regex could be better - not the point here.. just a quick demo of a +n approach.. my %A = map{/mti_clk_chk:(?::main :\|INFO:)?\s(.).\sT=(\d+)$/;$1 => + $2}@array; my %B = map{/mti_clk_chk:(?::main :\|INFO:)?\s(.).\sT=(\d+)$/;$1 => + $2}@array1; #print only common values in sequential order of file A print "\n******************************************\n"; print "Common Text descriptions with A and B T values:\n"; print "*****************************************\n"; foreach my $mti (map{/mti_clk_chk:(?::main :\|INFO:)?\s(.).\s*T=(\d+ +)$/;$1}@array) { if (exists $B{$mti}) { print "$mti T(a)=$A{$mti} T(b)=$B{$mti}\n"; } } [download]	[reply] [d/l] [select]
Re^3: Compare two log files line by line containing a keyword by thanos1983 (Parson) on Feb 20, 2019 at 09:28 UTC
Hello again rahu_6697, Although fellow Monk Marshall spend for free his time to implement a solution for your problem you do not even show the minimum amount of time and effort. You are saying that the script is not compiling because of: `Can't locate Data/Dump.pm in @INC (@INC contains: /usr/local/lib64/per +l5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/per +l5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .) at script3.pl lin +e 4. BEGIN failed--compilation aborted at script3.pl line 4` [download] Did you spend even 5 seconds to search on the web regarding this error? Can't locate ... in @INC Even this was not enough you ask him to modify his code so you can get the desired output into a file. One thing more I want this output in a separate output file, please change the code accordingly. Seriously if you want to learn even the basics regarding Perl and in generally scripting I would recommend start trying and failing until you get it right. Asking for other to resolve your problem will not teach you anything. Seeking for Perl wisdom...on the process of learning...not there...yet!	[reply] [d/l] [select]
Re^3: Compare two log files line by line containing a keyword by Marshall (Canon) on Feb 22, 2019 at 01:16 UTC
As other Monks have pointed out, the error you are getting is about Data::Dump not being installed. I used one of the subroutines in that module (pp) while fiddling with your problem. If you had investigated what pp() does and carefully read my code, you would have discovered that I don't even use that subroutine in the code version that I posted. So `#use Data::Dump qw(pp);` and my code still runs! Please continue to spend effort on your own code. Do not expect final polished code from this forum. My objective was to demo an approach that probably hadn't occurred to you. I believe that I was successful in that endeavour. The Monks do expect that you spend some serious effort trying to understand the advice that you have been given. Update: Homework for the OP: 1) Open and read log file X, line by line and create a hash %X like I show as %A. Hashes have no order to them so we need an array to maintain the input order. Save each key to hash %X as a new element of an array (i.e push that key onto another array). 2) Open and read log file Y, line by line and create a hash %Y like I show as %B. There is no need for an array here. 3) Now loop over the array from step (1) and print stuff that is common between hash %X and %Y. This is not what you need: `open(FILE, "<x.log"); my @array = <FILE>; # no need to use memory that way # a lot of the input line will be # thrown away close(FILE);` [download] Better: `open(FILE, "<", "x.log") or die "unable to open x.log $!"; while my $line (<FILE>) { ... process each $line... }` [download]	[reply] [d/l] [select]
Re^3: Compare two log files line by line containing a keyword by karlgoethebier (Abbot) on Feb 21, 2019 at 15:35 UTC
See here for help. �The Crux of the Biscuit is the Apostrophe� `perl -MCrypt::CBC -E 'say Crypt::CBC->new(-key=>'kgb',-cipher=>"Blowfish")->decrypt_hex($ENV{KARL});'`Help	[reply] [d/l]