Re: Re: nonblocking I/O - testing whether file has more data to read

Hi matija, thanks for the comments.

In this case, we're talking about a webserver application in which the logs are split out to separate files on a per-script and per-site basis, and I imagine typical usage would be to tail all logs for a site, for example just after an upgrade, to monitor for new problems that may have arisen.

So for most of those files, I'd expect no output at all - and therefore also no data on which to base predictions about the next time the file may be updated.

However I can imagine another use would be to turn on verbose debugging for all scripts, then run a filtering monitor over the log files until a particular problem shows itself - in that case I can imagine large quantities of data appearing in all the files.

I do need to cope with the files moving from under me, but the best strategy to cope with that is likely to be quite dependent on the strategy for reading from the file. In this case it is also likely to be dependent on the tools doing the moving - it is possible that the files will temporarily be replaced with a pipe, in which case it is important that I don't end up reading from that. In fact that's the main reason I'm wary of simply opening a pipe from tail(1), which otherwise has all the options I need in its GNU incarnation.

I think I shall try next with a simple fstat/sleep loop, and see what that does for performance under light and heavy loads.

Hugo

Comment on Re: Re: nonblocking I/O - testing whether file has more data to read Download Code

Replies are listed 'Best First'.
Re: Re: Re: nonblocking I/O - testing whether file has more data to read by matija (Priest) on Apr 09, 2004 at 05:43 UTC
If File::Tail detects no activity in the file, it will check it rarely. You can configure any interval, but I think the default is once per minute. In my opinion, for something like a whole bunch of rarely updated files, File::Tail is the right thing. The code would look somewhat like this: `foreach (keys %files) { push(@tails,File::Tail->new(name=>$_,tail=>0,reset_tail=>0)); } while (1) { ($nfound,$timeleft,@pending)= File::Tail::select(undef,undef,undef,$timeleft,@tails); foreach (@pending) { my $line=$_->read; process($line); } }` [download] This is actualy extracted from a script I use to monitor a whole bunch of logs on fairly active servers - the script calls one or more handlers which parse each particular type of log file, and the aggregate results are sent to a central monitoring machine. I ripped out the complications that deal with monitoring no files (because I colllect other data, too), and the multiple handler dispatch.	[reply] [d/l]

Replies are listed 'Best First'.

Re: Re: Re: nonblocking I/O - testing whether file has more data to read
by matija (Priest) on Apr 09, 2004 at 05:43 UTC

File::Tail

In my opinion, for something like a whole bunch of rarely updated files, File::Tail is the right thing. The code would look somewhat like this:

foreach (keys %files) {
    push(@tails,File::Tail->new(name=>$_,tail=>0,reset_tail=>0));
}
while (1) {
  ($nfound,$timeleft,@pending)=
          File::Tail::select(undef,undef,undef,$timeleft,@tails);
  foreach (@pending) {
      my $line=$_->read;
      process($line);
  }
}
[download]

[reply]
[d/l]