Tail a rapidly changing file

rah has asked for the wisdom of the Perl Monks concerning the following question:

I am trying to extract data from a very rapidly changing log file. I would like to sample this file every 5 minutes and get the most recent 5 minutes worth of data. I don't really want to write a tail program myself. I am a firm believer of not re-inventing the wheel.

I had hoped to used File::Tail, or better yet File::Tail::App to perform the tail so I could focus on the parsing and reporting. I really like the lastrun capability of File::Tail::App and thought I could exploit that to help control the data sampling. However, I think that my data is changing too rapdily for this. Not long after I start the script, the lastrun file becomes zero length and the tail behaves like a unix tail -f. I think that because the existence of an EOF is so fleeting, that from the program's perspective there isn't one and so it continues to read.

What I'm wondering is if there is a way for me to tell File::Tail:App or the underlying File::Tail where I want it to stop. I could easily tell it the file size in bytes and therefore a place where it could end. But as far as I can tell there is no provision for a second seek value to denote the end of the data.

Am I just trying to ask these tail modules to do something they can't? Do I need to write my own tail functionality using seek or sysseek?. My test code at this point is nothing more that the example provided with the File::Tail::App module documentation but used on one of my log files. I haven't even started on the parsing logic.

#!/usr/bin/perl
use strict;
use warnings;
use Unix::PID '/var/run/tail.pid';
use File::Tail::App qw(tail_app);

   tail_app({
      'new'          => ['my.log'],
      'line_handler' => \&_wag_tail,
      'lastrun_file' => 'x.lastrun',
   });

   sub _wag_tail {
      my($line) = @_;
     print "$line \n";
   }
[download]

Comment on Tail a rapidly changing file Download Code

Replies are listed 'Best First'.
Re: Tail a rapidly changing file by sflitman (Hermit) on Jan 16, 2010 at 06:07 UTC
I would just write my own code, it's pretty easy: `#!/usr/bin/perl use strict; use warnings; use Fcntl qw/:seek/; my $file='my.log'; # position to end open(FILE,$file) \|\| die "$file: $!"; seek(FILE,0,SEEK_END) \|\| die "Seek: $!"; my $pos=tell(FILE); close FILE; while(1) { sleep(5*60); # every 5 minutes open(FILE,$file) \|\| die "$file: $!"; seek(FILE,$pos,SEEK_SET) \|\| die "Seek: $!"; while (<FILE>) { # do something with $_ lines } $pos=tell(FILE); # update close FILE; }` [download] Send it the kill signal when you're done with it, or you could get fancy by adding code to abort when it sees something or other. HTH, SSF	[reply] [d/l]
Re^2: Tail a rapidly changing file by matze77 (Friar) on Jan 16, 2010 at 15:09 UTC
Hello! I dont know what `use Fcntl qw/:seek/;` accomplishes if someone got the time maybe he/she could explain? Thanks MH	[reply] [d/l]
Re^3: Tail a rapidly changing file by BrowserUk (Patriarch) on Jan 16, 2010 at 15:28 UTC
I dont know what use Fcntl qw/:seek/; accomplishes ... It imports the constants `SEEK_SET SEEK_CUR SEEK_END`. Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice. "I'd rather go naked than blow up my ass"	[reply] [d/l]
Re: Tail a rapidly changing file by Khen1950fx (Canon) on Jan 15, 2010 at 23:51 UTC
Try File::ReadBackwards and read grinder's Performing a tail(1) in Perl (reading the last N lines of a file).	[reply]
Re^2: Tail a rapidly changing file by rah (Monk) on Jan 16, 2010 at 01:01 UTC
I looked at both earlier, but I don't think either will apply. I have the problem Grinder describes about not being able to stop File::Tail. But I don't think any of the suggestions in the linked node will work. I don't think ReadBackwards will work because I don't know how many lines I need. It will be some very large number most times and highly variable at all times.	[reply]


Don't ask to ask, just ask
	PerlMonks