in reply to Re: Parsing log file with regular-expression match nested brackets
in thread Parsing log file with regular-expression match nested brackets [RESOLVED]

Hello 1nickt,

Well my first step is to read the log file into a string as I am doing. Then as a second step is to split complete entities from log file into separate array elements that start with ([     11] 25/2/2017-19:02:06.980 proces_name thanos-Rx: UDT{) and end with (}).

As a third step I want them to spit each array element into separate lines based on new line characters (\n) into separate element on sub array. The reason that I want to do that is to have concrete element reference so when I will be using loops with regular expressions rules to search through the array elements it will match.

My ultimate goal is to match entities "first level array elements" with each other based on similarities with in the lines.

It sounds a bit of complicated so I will try to simplify it.

Let say that I have a log file:

[     11] 25/2/2017-19:02:06.980 proces_name thanos-Rx: UDT{
[     11] 25/2/2017-19:02:06.980 proces_name thanos-Rx:              l
+ine2: data
[     11] 25/2/2017-19:02:06.980 proces_name thanos-Rx:              l
+ine3: data
[     11] 25/2/2017-19:02:06.980 proces_name thanos-Rx:              l
+ine4: data
[     11] 25/2/2017-19:02:06.980 proces_name thanos-Rx:              l
+ine5: data
[     11] 25/2/2017-19:02:06.980 proces_name thanos-Rx:              l
+ine6: data
[     11] 25/2/2017-19:02:06.980 proces_name thanos-Rx:              l
+ine7: data
}
[     11] 25/2/2017-19:02:06.980 proces_name thanos-Tx: UDT{
[     11] 25/2/2017-19:02:06.980 proces_name thanos-Tx:              l
+ine10: data
[     11] 25/2/2017-19:02:06.980 proces_name thanos-Tx:              l
+ine11: data
[     11] 25/2/2017-19:02:06.980 proces_name thanos-Tx:              l
+ine12: data
[     11] 25/2/2017-19:02:06.980 proces_name thanos-Tx:              l
+ine13: data
[     11] 25/2/2017-19:02:06.980 proces_name thanos-Tx:              l
+ine14: data
[     11] 25/2/2017-19:02:06.980 proces_name thanos-Tx:              l
+ine15: data
}

[download]

For simplicity reasons I kept the sample of log file without any complexities.

So lets assume that: [     11] 25/2/2017-19:02:06.980 proces_name thanos-Rx:              line7: data contains some data that I want to fine exactly the same match on the log file e.g.: [     11] 25/2/2017-19:02:06.980 proces_name thanos-Tx:              line11: data

.

it would be very simple if we wanted to catch the single line but I need to return the whole array containing all the lines that starts [     11] 25/2/2017-19:02:06.980 proces_name thanos-Rx: UDT{ and ends with }.

So the expected and desired result would be a hash:

$VAR1 = {
          'Test1' => {
                          '[     11] 25/2/2017-19:02:06.980 proces_nam
+e thanos-Rx: UDT{
[     11] 25/2/2017-19:02:06.980 proces_name thanos-Rx:              l
+ine2: data
[     11] 25/2/2017-19:02:06.980 proces_name thanos-Rx:              l
+ine3: data
[     11] 25/2/2017-19:02:06.980 proces_name thanos-Rx:              l
+ine4: data
[     11] 25/2/2017-19:02:06.980 proces_name thanos-Rx:              l
+ine5: data
[     11] 25/2/2017-19:02:06.980 proces_name thanos-Rx:              l
+ine6: data
[     11] 25/2/2017-19:02:06.980 proces_name thanos-Rx:              l
+ine7: data
}'
                        },
          'Test2' => {
                         '[     11] 25/2/2017-19:02:06.980 proces_name
+ thanos-Tx: UDT{
[     11] 25/2/2017-19:02:06.980 proces_name thanos-Tx:              l
+ine10: data
[     11] 25/2/2017-19:02:06.980 proces_name thanos-Tx:              l
+ine11: data
[     11] 25/2/2017-19:02:06.980 proces_name thanos-Tx:              l
+ine12: data
[     11] 25/2/2017-19:02:06.980 proces_name thanos-Tx:              l
+ine13: data
[     11] 25/2/2017-19:02:06.980 proces_name thanos-Tx:              l
+ine14: data
[     11] 25/2/2017-19:02:06.980 proces_name thanos-Tx:              l
+ine15: data
}'
                       }
        };

[download]

I hope my explanation did not confused more it helped you to understand my approach to my problem.

If you have any further questions please do not hesitate to ask.

Seeking for Perl wisdom...on the process of learning...not there...yet!

Replies are listed 'Best First'.
Re^3: Parsing log file with regular-expression match nested brackets
by 1nickt (Canon) on Feb 28, 2017 at 16:43 UTC

    Well, hm, did you read the node I linked to? You basically restated the content of your first post. What I was suggesting was to state instead a higher-level overview of your task. Because from what I can see, for example, you would be much better off logging into a database, whereupon you select groups of records using an SQL query. But I am guessing, because you haven't yet said something like (guessing again):

    "I'm working on a logging system for my application, and I want to collate the debug output from the client and the server for the same transaction. I'm thinking of parsing the text log file of each using a regexp, but I'm wondering if there's a better way?"

    That's the difference between discussing "X" and discussing "Y".

    Hope this helps!


    The way forward always starts with a minimal test.
[reply]

      Hello again,

      I have read before the XY Problem and this is the reason that I tried to go even into further details on my previous reply.

      But let me try again. I am login into a servers through a client script that has an application that produces this kind of log files.

      No I can not change the log file output, no I am not processing the stdout in real time. I am collecting the log files for further processing in the background, in order to detect miss matched errors.

      It would help me personally a lot to match entities from the log files through a script. I can do the whole process manually using "less and /keyword search" through the file as I am currently doing. But it would be much faster and easier for me to simply write a script to automate the process instead of doing it manually.

      I can not use a database on the system.

      I do not know if this helps but this is the reason that I am doing it.

      Hope this helps, as a higher level reply instead of a low level. :D

      Seeking for Perl wisdom...on the process of learning...not there...yet!
[reply]
[d/l]
[select]

        Is this close to want you want ?

        #!/usr/bin/perl
use strict;
use warnings;
use Data::Dump 'pp';

my %hash  = ();
my $level = 0; 
my $key; 
while (<DATA>){
  next unless /\S/;
  my ($proc,$data) = split ':\s+',$_,2;
  if ($level == 0){
    $key = $proc;
  } else {
    push @{$hash{$key}},$data if (defined $data);    
  } 
  ++$level if /{/;
  --$level if /}/;
}
pp \%hash;

__DATA__
[     11] 25/2/2017-19:02:06.980 proces_name thanos-Rx: UDT{
[     11] 25/2/2017-19:02:06.980 proces_name thanos-Rx:              l
+ine2: data
[     11] 25/2/2017-19:02:06.980 proces_name thanos-Rx:              l
+ine3: data
[     11] 25/2/2017-19:02:06.980 proces_name thanos-Rx:              l
+ine4: data
[     11] 25/2/2017-19:02:06.980 proces_name thanos-Rx:              l
+ine5: data
[     11] 25/2/2017-19:02:06.980 proces_name thanos-Rx:              l
+ine6: data
[     11] 25/2/2017-19:02:06.980 proces_name thanos-Rx:              l
+ine7: data
}
[     11] 25/2/2017-19:02:06.980 proces_name thanos-Tx: UDT{
[     11] 25/2/2017-19:02:06.980 proces_name thanos-Tx:              l
+ine10: data
[     11] 25/2/2017-19:02:06.980 proces_name thanos-Tx:              l
+ine11: data
[     11] 25/2/2017-19:02:06.980 proces_name thanos-Tx:              l
+ine12: data
[     11] 25/2/2017-19:02:06.980 proces_name thanos-Tx:              l
+ine13: data
[     11] 25/2/2017-19:02:06.980 proces_name thanos-Tx:              l
+ine14: data
[     11] 25/2/2017-19:02:06.980 proces_name thanos-Tx:              l
+ine15: data
}

        [download]
        poj
[reply]
[d/l]

In Section Seekers of Perl Wisdom