Using Seek to read and parse growing log files

Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

Dear monks,

I desperately need divine guidance on accomplishing the following:
1. Read and parse through a log file
2. Remember and store the offset to the last line read
3. Next time in, start reading the log file at the previous offset
I believe I am part way there as i've gained some great snippets from the 'hall of records' at the monestery, however i am getting some unexpected results. I can parse through and start at what i believe to be the appropriate offset but I get an extraline or something when it writes to my datafile. Here is what my code looks like:

# open the log file
open(REGLOG, $regLog) or die "Can't open logfile: $regLog";

#open data file
open(REGDATA, ">> $regDataFile") or die "can't open the Registration d
+ata file";

# seek to last offset
seek(REGLOG, $offset[0], 1);

while (<REGLOG>) 
{
    chomp; # removing trailing \n
    # code truncated here for brevity.....
        # fields defined through delimiters....

    print REGDATA "$regDate, $regUserName, $regEmail, $regCountry, $re
+gIp, $regRefer, \n";
    
    
}
[download]

The file begins reading from the offset but adds an extra line of ,,,,,, It is as if it is reading an extra line which I thought chomp would remove. Any ideas what I am missing?? Thanks.

Comment on Using Seek to read and parse growing log files Download Code

Replies are listed 'Best First'.
Re: Using Seek to read and parse growing log files by cleverett (Friar) on Nov 18, 2003 at 20:54 UTC
First turn on warnings with a "use warnings;" right below the "use strict;" in your program. I'm willing to be that $regDate, $regUserName, $regEmail, $regCountry, $regIp, $regRefer remain undefined the first time through. I you find a benign reason for the undefined variables, then just skip printing the line: `next unless defined $regDate;` [download] Otherwise dig through your code to find the reason. I wonder what would happen if you increment the offset you're seeking on by 1. `seek(REGLOG, $offset[0] + 1, 1);` [download]	[reply] [d/l] [select]
Re: Re: Using Seek to read and parse growing log files by Anonymous Monk on Nov 18, 2003 at 21:44 UTC
Yep, that is it, my variable were not defined from the offset although I'm a bit confused why they wouldn't be defined. Anyway, I check for variables being defined before printing them to my datafile and that solves the issue. Thanks to everyone who replied...	[reply]
Re: Using Seek to read and parse growing log files by Roger (Parson) on Nov 18, 2003 at 22:51 UTC
Just to point out that there was a similar discussion thread here - Opening a file at a designated offset based on closing of the file, posted on 11th November 2003. It might be worthwhile to go through its responses. Your solution works, provided that you never trim, rotate, reset your log file. I have posted a more reliable solution in the discussion mentioned above.	[reply]
Re: Using Seek to read and parse growing log files by holo (Monk) on Nov 18, 2003 at 20:57 UTC
Does any log line get skipped where the ",,,," appears? Since I can't see the parsing code, I'm not sure about this but I think the problem is solved by searching one byte beyond the last end of file (and skip the newline) `seek(REGLOG, $offset[0] + 1, 1);` As a side note, you could use File::Tail and do the parsing on the fly as the file grows. That's a pain over reboots though! Update: Hail cleverett for it is he who preceded me ;)	[reply] [d/l]
Re: Using Seek to read and parse growing log files by exussum0 (Vicar) on Nov 18, 2003 at 21:30 UTC
I do exactly the same thing you do, but instead of running the process every now and then, put a signal handler in front of it, and ... `. . . while(1) { while(my $line=<$fh>) { # lot's o processing } sleep 5; }` [download] When I don't need or want to process anymore, just kill the process via the prepped signal handler, so i can clean up db handles or whatever i'm doing, gracefully. Doesn't solve your exact problem, but gives you a new possibility. -s Play that funky music white boy..	[reply] [d/l]
Re: Using Seek to read and parse growing log files by ysth (Canon) on Nov 18, 2003 at 21:07 UTC
Your problem is in the "code truncated here for brevity" part. Are you using a regexp match but not checking if it succeeds?.	[reply]
Re: Using Seek to read and parse growing log files by talexb (Chancellor) on Nov 19, 2003 at 19:20 UTC
It may be more efficient to parse chunks of log file as they get handed to you by `logrotate`. That way you won't have to store the last location of where you looked in the file and worry about invalidating that when you really to rotate your logs. --t. alex Life is short: get busy!	[reply] [d/l]