familyofcrowes has asked for the wisdom of the Perl Monks concerning the following question:

I am trying to find a way to read a log file looking for a key word and return text from the log associated with that key word. Big issue is that the log file updates dynamically and is cleared every 24 hours. So If I look for keyword "101", I will get all of them since last night, where what I need is just the ones since my last check/poll.

Example log: 101 QE300800 0AAAAAAAAAAABEU5 TAERP060_600 TESTCOS 589854 5 TAERP060_600 maestro TAERP060\040PRIORITY(5)\040CPDEPR(N) -32768 1 2011110306000000 0 +++ 879 0 1320300000 1320300000 180 0 2 0AAAAAAAAAAABEU5 879 DET_M-S_1 2011110306000000 v.4 NONE NONE 4 NONE NONE NONE 0 0 0 0 0 NONE

I need the "TAERP060_600" and the next one too. How do you grab the data, then, how do you make sure you get them all but no duplicates? Any help is MEGA appreciated!!!!! So far I have the keyword search and thats all.... I know... lame....

perl ${SCRIPT} /tws/dv/event.log "101 " #!/usr/bin/perl -s if (@ ARGV[0] =~ /\bhelp\b/) { print "Usage: KeywordSearch.pl <path>\/filename> <pattern>\n"; exit 1; } chomp($file = @ ARGV[0]); chomp($pattern = @ ARGV[1]); if (length($file)==0||length($pattern)==0){ print "Message: Bad argument! Please see help\n"; print "Statistic: 0\n"; exit 1; } else { if (!-e $file){ print "Message: \"$file\" (File NOT FOUND!)\n"; print "Statistic: 0\n"; exit 1; } } open (F, '<', $file) || die $!; my @lines = <F>; foreach my $line (@lines) { if ($line =~ /\Q$pattern\E/) { print "Message: \"$pattern\" (Pattern FOUND)\n"; print "Statistic: 1\n"; exit 0; } } print "Message: \"$pattern\" (Pattern NOT FOUND)\n"; print "Statistic: 0\n"; exit 0

Replies are listed 'Best First'.
Re: Returning data from a log file
by cavac (Prior) on Nov 03, 2011 at 20:24 UTC

    Is the logfile cleared or moved and a new one created? If it's just cleared, you *will* miss some data eventually.

    Since i don''t know what the data fields mean, it is rather hard to give advice? Is there some unique identifier for each line? If not, then it's rather hard to find a working solution - it can be done, but it depends on a number of small implementation details.

    A general way would be to either have the above mentioned unique identifier or to bluntly count lines and hope for the best. If the first line in the file doesn't match the first line of the file you "remembered" from the last run, open an older file and try again. Then, when you found the correct file, work on that. When finished, check for newer files and work on them in correct order. As Last step, "remember" the first line of the last file you worked on.

    Don't use '#ff0000':
    use Acme::AutoColor; my $redcolor = RED();
    All colors subject to change without notice.
Re: Returning data from a log file
by keszler (Priest) on Nov 04, 2011 at 01:26 UTC

    In 1998, before SMTP AUTH standards were created, Curt Sampson <cjs@cynic.net> wrote poprelayd to implement "POP before SMTP" authentication. It works by watching a log file for new lines (like `tail -f`), and it also re-opens the log file when the inode changes. I've used his getlogline subroutine for similar file-watching purposes several times.

    Code is at http://poprelay.sourceforge.net/poprelayd

Re: Returning data from a log file
by pvaldes (Chaplain) on Nov 03, 2011 at 21:05 UTC

    When updating the new data is written on the first or the last line of your log?

    Do you have a my_log.old.gz file or my_log.1.gz file?

    Any of the fields of your log is in any way a timestamp?


    how do you're sure that you get no duplicates?

    With this? are the lines unique? maybe you need more data:

    while (@lines) { chomp; print if /\Q$pattern\E/; }