Re: Compare two log files line by line containing a keyword

I am very confused as to what the desired result is? Your question is not clear, at least to me.

BTW, when you present data, please use <code>..</code> tags.

At this point, I didn't spend time worrying about how to improve your code that generates the 2 arrays. Before doing that, I would like to understand the end objective.

To try to understand the requirements, I wrote some simple code to replicate your @array and @array1. Then I reduced those arrays down to just the last part of the line that includes the 'multi_clk_chk:' stuff. I take it that only the stuff after 'multi_clk_chk:' matters?

See below and please explain what a comparison between Data Set A and Data Set B would be?

I suppose that these log files contain many records? Of which you are just showing one of each?
A first step might be to figure how to figure out how to select a record from X and a record from Y?
The difference in the T values seems to be what you really want?

It could be that a hash table is the right kind of data structure for each record? text before the T value as the key and T number as the value? Would order matter in the report?

stuff before the T value => T value
A: MTI Clock is being generated correctly ; => T=1102377
B: MTI Clock is being generated correctly ; => T=1092605
perhaps result:
MTI Clock is being generated correctly ;  T=1102377  T=1092605
[download]

I'm sure you will get lots of interesting implementation ideas if you can explain the requirements better.

Update: Please think about how you want the results to be presented to the user. General file compares can generate some confusing looking stuff. A good UI can be important. For example I am working on a feature request on a different project.. I understood instantly how to calculate the new desired result (the math was easy)...but, I spent an hour thinking about how to present this new data to the user in way that was not confusing and so that they'd understand what this information meant.

Another Update: I went ahead and hacked in some additional code for the "use a hash for comparison" idea. Still not sure what the OP wants, but this might be useful. No consideration for efficiency was made - that can be optimized later once the problem is understood.

#!/usr/bin/perl;
use strict;
use warnings;
use Data::Dump qw(pp);

# initialize @array and @array1 to what the OP describes as his curren
+t "output".....

my @array =<<'EOF' =~  m/(^.*\n)/mg;
INFO @576892 mti_clk_chk: run_stimulus called; T=576892
INFO @1102266 PHResourceLayer_Z4: mti_clk_chk:################ start o
+f test ################ ; T=1102266
INFO @1102334 PHResourceLayer_Z4: mti_clk_chk:Checking the period of M
+TI, MTI10 clk from SV; T=1102334
INFO @1102372 mti_clk_chk: Checking period of MTI CLk; T=1102372
INFO @1102377 mti_clk_chk: Period value of MTI Clock: 3.125000 ns; T=1
+102377
INFO @1102377 mti_clk_chk: MTI Clock is being generated correctly ; T=
+1102377
INFO @1102377 mti_clk_chk: Checking period of MTI10 CLk; T=1102377
INFO @1102418 mti_clk_chk: Period value of MTI10 Clock: 31.250000 ns; 
+T=1102418
INFO @1102418 mti_clk_chk: MTI10 Clock is being generated correctly ; 
+T=1102418
INFO @1102717 PHResourceLayer_Z4: mti_clk_chk: All clock period Checki
+ng done; T=1102717
INFO @1148661 mti_clk_chk: C-Code exit execution. code=<aa>; T=1148661
INFO @1148661 mti_clk_chk: ************************ SV END************
+******** ; T=1148661
EOF

my @array1 =<<'EOF2' =~  m/(^.*\n)/mg;
UVM_INFO @0 reporter testbench.top_level_module.\mti_clk_chk::main : M
+TI_CLK_CHK_STIM Started .....; T=0
UVM_INFO @0 reporter testbench.top_level_module.\mti_clk_chk::main : r
+un_stimulus called; T=0
UVM_INFO @1092507 reporter Z4_COREA: mti_clk_chk: ################ sta
+rt of test ################ ; T=1092507
UVM_INFO @1092563 reporter Z4_COREA: mti_clk_chk: Checking the period 
+of MTI, MTI10 clk from SV; T=1092563
UVM_INFO @1092598 reporter testbench.top_level_module.\mti_clk_chk::ma
+in : Checking period of MTI CLk; T=1092598
UVM_INFO @1092605 /proj/rru2_verif/usr/Tilak/SV_UVM/testbench/data_ipd
+ss/v_ms_mti_stim_vip/testbench/classes_v/mti_clk_chk.sv(147) uvm_test
+_top.default_env.default_sequencer100@@mti_clk_chk mti_clk_chk:INFO: 
+Period value of MTI Clock: 3.125000 ns; T=1092605
UVM_INFO @1092605 reporter testbench.top_level_module.\mti_clk_chk::ma
+in : MTI Clock is being generated correctly ; T=1092605
UVM_INFO @1092605 reporter testbench.top_level_module.\mti_clk_chk::ma
+in : Checking period of MTI10 CLk; T=1092605
UVM_INFO @1092655 /proj/rru2_verif/usr/Tilak/SV_UVM/testbench/data_ipd
+ss/v_ms_mti_stim_vip/testbench/classes_v/mti_clk_chk.sv(165) uvm_test
+_top.default_env.default_sequencer100@@mti_clk_chk mti_clk_chk:INFO: 
+Period value of MTI10 Clock: 31.250000 ns; T=1092655
UVM_INFO @1092655 reporter testbench.top_level_module.\mti_clk_chk::ma
+in : MTI10 Clock is being generated correctly ; T=1092655
UVM_INFO @1092850 reporter Z4_COREA: mti_clk_chk: All clock period Che
+cking done; T=1092850
UVM_INFO @1092886 /proj/rru2_verif/usr/Tilak/SV_UVM/testbench/data_ipd
+ss/v_ms_mti_stim_vip/testbench/classes_v/mti_clk_chk.sv(186) uvm_test
+_top.default_env.default_sequencer100@@mti_clk_chk mti_clk_chk:INFO: 
+************************ SV END******************** ; T=1092886
EOF2

# evidently all that matters is stuff after the 'mti_clk_chk:' string
# so remove all before that critical stuff

@array  = map{/^.*(mti_clk_chk:.*\n)/;$1;}@array;
@array1 = map{/^.*(mti_clk_chk:.*\n)/;$1;}@array1;

## just print these arrays out so that we can see how to 
## compare them?

my $n=1;
foreach (@array) {print "A$n) ",$_; $n++}

print "===============================\n";

$n=1;
foreach (@array1) {print "B$n) ",$_; $n++}

## Make hash tables of the 2 records
## regex could be better - not the point here.. just a quick demo of a
+n approach..

my %A = map{/mti_clk_chk:(?::main :|INFO:)?\s*(.*).*\s*T=(\d+)$/;$1 =>
+ $2}@array;
my %B = map{/mti_clk_chk:(?::main :|INFO:)?\s*(.*).*\s*T=(\d+)$/;$1 =>
+ $2}@array1;

#print only common values in sequential order of file A
print "\n********************************************\n";
print "Common Text descriptions with A and B T values:\n";
print "********************************************\n";

foreach my $mti (map{/mti_clk_chk:(?::main :|INFO:)?\s*(.*).*\s*T=(\d+
+)$/;$1}@array)
{
    if (exists $B{$mti})
    {
        print "$mti T(a)=$A{$mti} T(b)=$B{$mti}\n";
    }
}


__END__
A1) mti_clk_chk: run_stimulus called; T=576892
A2) mti_clk_chk:################ start of test ################ ; T=11
+02266
A3) mti_clk_chk:Checking the period of MTI, MTI10 clk from SV; T=11023
+34
A4) mti_clk_chk: Checking period of MTI CLk; T=1102372
A5) mti_clk_chk: Period value of MTI Clock: 3.125000 ns; T=1102377
A6) mti_clk_chk: MTI Clock is being generated correctly ; T=1102377
A7) mti_clk_chk: Checking period of MTI10 CLk; T=1102377
A8) mti_clk_chk: Period value of MTI10 Clock: 31.250000 ns; T=1102418
A9) mti_clk_chk: MTI10 Clock is being generated correctly ; T=1102418
A10) mti_clk_chk: All clock period Checking done; T=1102717
A11) mti_clk_chk: C-Code exit execution. code=<aa>; T=1148661
A12) mti_clk_chk: ************************ SV END******************** 
+; T=1148661
===============================
B1) mti_clk_chk::main : MTI_CLK_CHK_STIM Started .....; T=0
B2) mti_clk_chk::main : run_stimulus called; T=0
B3) mti_clk_chk: ################ start of test ################ ; T=1
+092507
B4) mti_clk_chk: Checking the period of MTI, MTI10 clk from SV; T=1092
+563
B5) mti_clk_chk::main : Checking period of MTI CLk; T=1092598
B6) mti_clk_chk:INFO: Period value of MTI Clock: 3.125000 ns; T=109260
+5
B7) mti_clk_chk::main : MTI Clock is being generated correctly ; T=109
+2605
B8) mti_clk_chk::main : Checking period of MTI10 CLk; T=1092605
B9) mti_clk_chk:INFO: Period value of MTI10 Clock: 31.250000 ns; T=109
+2655
B10) mti_clk_chk::main : MTI10 Clock is being generated correctly ; T=
+1092655
B11) mti_clk_chk: All clock period Checking done; T=1092850
B12) mti_clk_chk:INFO: ************************ SV END****************
+**** ; T=1092886


# Some of the issues involved in the comparison between sets:
#   Line B1 doesn't exist in set A
#   line A11 doesn't exist in set B
# one plausible way to present the output?

********************************************
Common Text descriptions with A and B T values:
********************************************
run_stimulus called;  T(a)=576892 T(b)=0
################ start of test ################ ;  T(a)=1102266 T(b)=1
+092507
Checking the period of MTI, MTI10 clk from SV;  T(a)=1102334 T(b)=1092
+563
Checking period of MTI CLk;  T(a)=1102372 T(b)=1092598
Period value of MTI Clock: 3.125000 ns;  T(a)=1102377 T(b)=1092605
MTI Clock is being generated correctly ;  T(a)=1102377 T(b)=1092605
Checking period of MTI10 CLk;  T(a)=1102377 T(b)=1092605
Period value of MTI10 Clock: 31.250000 ns;  T(a)=1102418 T(b)=1092655
MTI10 Clock is being generated correctly ;  T(a)=1102418 T(b)=1092655
All clock period Checking done;  T(a)=1102717 T(b)=1092850
************************ SV END******************** ;  T(a)=1148661 T(
+b)=1092886
[download]

Comment on Re: Compare two log files line by line containing a keyword Select or Download Code

Replies are listed 'Best First'.
Re^2: Compare two log files line by line containing a keyword by rahu_6697 (Novice) on Feb 20, 2019 at 06:10 UTC
You understood the question correctly and your output is quite good also, but I'm facing this issue while run. One thing more I want this output in a separate output file, please change the code accordingly. `Can't locate Data/Dump.pm in @INC (@INC contains: /usr/local/lib64/per +l5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/per +l5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .) at script3.pl lin +e 4. BEGIN failed--compilation aborted at script3.pl line 4` [download] Code I am trying to run is #!/usr/bin/perl; use strict; use warnings; use Data::Dump qw(pp); open(FILE, "<x.log"); my @array = <FILE>; close(FILE); open(FILE, "<y.log"); my @array1 = <FILE>; close(FILE); my @array =<<'EOF' =~ m/(^.\n)/mg; EOF my @array1 =<<'EOF2' =~ m/(^.\n)/mg; EOF2 @array = map{/^.(mti_clk_chk:.\n)/;$1;}@array; @array1 = map{/^.(mti_clk_chk:.\n)/;$1;}@array1; ## just print these arrays out so that we can see how to compare them? my $n=1; foreach (@array) {print "A$n) ",$_; $n++} print "===============================\n"; $n=1; foreach (@array1) {print "B$n) ",$_; $n++} ## Make hash tables of the 2 records ## regex could be better - not the point here.. just a quick demo of a +n approach.. my %A = map{/mti_clk_chk:(?::main :\|INFO:)?\s(.).\sT=(\d+)$/;$1 => + $2}@array; my %B = map{/mti_clk_chk:(?::main :\|INFO:)?\s(.).\sT=(\d+)$/;$1 => + $2}@array1; #print only common values in sequential order of file A print "\n******************************************\n"; print "Common Text descriptions with A and B T values:\n"; print "*****************************************\n"; foreach my $mti (map{/mti_clk_chk:(?::main :\|INFO:)?\s(.).\s*T=(\d+ +)$/;$1}@array) { if (exists $B{$mti}) { print "$mti T(a)=$A{$mti} T(b)=$B{$mti}\n"; } } [download]	[reply] [d/l] [select]
Re^3: Compare two log files line by line containing a keyword by thanos1983 (Parson) on Feb 20, 2019 at 09:28 UTC
Hello again rahu_6697, Although fellow Monk Marshall spend for free his time to implement a solution for your problem you do not even show the minimum amount of time and effort. You are saying that the script is not compiling because of: `Can't locate Data/Dump.pm in @INC (@INC contains: /usr/local/lib64/per +l5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/per +l5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .) at script3.pl lin +e 4. BEGIN failed--compilation aborted at script3.pl line 4` [download] Did you spend even 5 seconds to search on the web regarding this error? Can't locate ... in @INC Even this was not enough you ask him to modify his code so you can get the desired output into a file. One thing more I want this output in a separate output file, please change the code accordingly. Seriously if you want to learn even the basics regarding Perl and in generally scripting I would recommend start trying and failing until you get it right. Asking for other to resolve your problem will not teach you anything. Seeking for Perl wisdom...on the process of learning...not there...yet!	[reply] [d/l] [select]
Re^3: Compare two log files line by line containing a keyword by Marshall (Canon) on Feb 22, 2019 at 01:16 UTC
As other Monks have pointed out, the error you are getting is about Data::Dump not being installed. I used one of the subroutines in that module (pp) while fiddling with your problem. If you had investigated what pp() does and carefully read my code, you would have discovered that I don't even use that subroutine in the code version that I posted. So `#use Data::Dump qw(pp);` and my code still runs! Please continue to spend effort on your own code. Do not expect final polished code from this forum. My objective was to demo an approach that probably hadn't occurred to you. I believe that I was successful in that endeavour. The Monks do expect that you spend some serious effort trying to understand the advice that you have been given. Update: Homework for the OP: 1) Open and read log file X, line by line and create a hash %X like I show as %A. Hashes have no order to them so we need an array to maintain the input order. Save each key to hash %X as a new element of an array (i.e push that key onto another array). 2) Open and read log file Y, line by line and create a hash %Y like I show as %B. There is no need for an array here. 3) Now loop over the array from step (1) and print stuff that is common between hash %X and %Y. This is not what you need: `open(FILE, "<x.log"); my @array = <FILE>; # no need to use memory that way # a lot of the input line will be # thrown away close(FILE);` [download] Better: `open(FILE, "<", "x.log") or die "unable to open x.log $!"; while my $line (<FILE>) { ... process each $line... }` [download]	[reply] [d/l] [select]
Re^3: Compare two log files line by line containing a keyword by karlgoethebier (Abbot) on Feb 21, 2019 at 15:35 UTC
See here for help. �The Crux of the Biscuit is the Apostrophe� `perl -MCrypt::CBC -E 'say Crypt::CBC->new(-key=>'kgb',-cipher=>"Blowfish")->decrypt_hex($ENV{KARL});'`Help	[reply] [d/l]