itsmeman2 has asked for the wisdom of the Perl Monks concerning the following question:

i have a log file that is generated from an application that I'd like to mine. the output log is different than reports generated, but the log file is always accurate.

what i would like to know how to do is navigate text data. for example I find string xyz and if I find that string, I want to go four rows previously and grab that value, then four rows before that and grab that value based on string xyz.

any ideas how I can accomplish that?

is it possible to store each row in an array and give it a value for each row and navigate back to that?

Replies are listed 'Best First'.
Re: managing log files
by zek152 (Pilgrim) on Aug 06, 2010 at 20:40 UTC

    Well there are quite a few ways. One method would be to store the log in a hash (or an array):

    my %log_file_hash = (); my @log_file_array; my $linenum = 0; my $line; open FILE, "logfile.txt"; while ($line = <FILE>) { chomp $line; $log_file_hash{$linenum} = $line; push @log_file_array, $line; $linenum++; } #now the whole file is in the hash and array. print "The first line is " . $log_file_hash{"0"} . "\n"; print "From the array... " . $log_file_array[0] . "\n";

    Another way to do this is to just store the last 4 lines and sort of move a frame over your data:

    $line_0 = ""; $line_1 = ""; $line_2 = ""; $line_3 = ""; while ($line = <FILE>) { chomp $line; $line_3 = $line_2; $line_2 = $line_1; $line_1 = $line_0; $line_0 = $line; }

    Hopefully this is useful. If I misunderstood your question just let me know.

    Update: minor inplace grammar edits.

Re: managing log files
by dasgar (Priest) on Aug 07, 2010 at 04:52 UTC

    Based on your described algorithm, I'd suggest using Tie::File. If you're using large log files, be sure to read the section on memory.

    Basically, this module allows you to "tie" a file to an array where each line in the file is an element in the array. This allows you to use array control functions (such as push and pop) to manipulate the file (such as adding a new line of text and deleting a line.)

    The (untested) code below should help you get started with using Tie::File with what you're trying to do.

    use strict; use Tie::File; my $filename = "some.log"; my @file; tie @file, 'Tie::File', $filename || die "Unable to open file\n"; my $index = 0; foreach my $line (@file) { if ($line =~ m/xyz/i) { my $var1 = $file[$index-4]; my $var2 = $file[$index+4]; print "Found values '$var1' and '$var2'\n"; } $index++; } untie @file;
Re: managing log files
by ahmad (Hermit) on Aug 06, 2010 at 23:41 UTC

    a not so perfect solution , but might work

    #!/usr/bin/perl use strict; use warnings; my @array; my $PreviousLines = 4; while (<DATA>) { chomp; if (/xyz/) { print join "\n", @array; print "\n\n"; }else{ push @array, $_; if (@array > $PreviousLines ) { # delete first element splice @array, 0,1; } } }
Re: managing log files
by apl (Monsignor) on Aug 07, 2010 at 12:11 UTC
    Don't worry about how to do it in Perl. Describe how you'd do in English. ("Read a line. Store it in an array. Is 'xyz' in the line?" etc.)

    Once you have it in English, it's easy to translate to Perl, C++or anything else.

    Note:English, in the above, is a placeholder for human language you're most comfortable writing in. 8-)

Re: managing log files
by itsmeman2 (Initiate) on Aug 09, 2010 at 15:29 UTC
    thanks for all the help - here is what I came up with. this allows me to navigate a text file and put together a minimized version of it. Instead of 27,008 rows I can get it down to 50 rows of data I am focused on. Thanks a lot!
    #!perl use tie::file my $count = 0; tie @array, 'Tie::File', 'log.txt' or die; my $index = 0; foreach my $line (@array) { if ($line =~ m/Age: 3/i) { my $age = @array[$index]; my $var1 = @array[$index-4]; my $var2 = @array[$index+6]; my $var3 = @array[$index+7]; if ($var2 =~ m/Added to non/i) { print "$var1\n"; print "$age\n"; print "$var2\n"; print "-----------------\n"; } if ($var3 =~ m/Added to non/i) { print "$var1\n"; print "$age\n"; print "$var3\n"; print "-----------------\n"; } } $index++; } untie @array;