Hmm, interesting. An fstat/sleep loop might well be sufficient. Part of the purpose of this script is to provide similar levels of support to the standard sysadmin tools at the point we replace a single error log for each webserver with lots of little logs, one for each script, so I'd hope to achieve responsiveness similar to tail -f error_log. But a second or two's delay is unlikely to be critical, and could certainly help to make the process more cooperative.

There are situations in which I'd expect large amounts of input to be filtered down to small amounts of output, but if I calculate the time to sleep from the start of the fstat/read cycle (rather than doing a fixed sleep each time) I can minimise the danger of falling behind.

Hugo

I went for this approach, and it seems to work fine, giving good responsiveness without causing a detectable load on the system.

The basic loop is something like this:

  my @closed = list_of_files();
  my @open = ();
  my $count = 0;
  while (1) {
    my $time = Time::HiRes::time();
    check_moved(\@open) if ($count % $CHECKMOVED) == 0;
    check_new(\@closed) if ($count % $CHECKNEW) == 0;
    check_read(\@open) if ($count % $CHECKREAD) == 0;
    ++$count;
    my $delay = $TICK - (Time::HiRes::time() - $time);
    Time::HiRes::sleep($delay) if $delay > 0;
  }
[download]

I'm currently using constants $TICK=0.1, $CHECKREAD=1, $CHECKNEW=20, $CHECKMOVED=50, but later I plan to make these more dynamic based on the size of the file list and other details.

Checking for new files involves trying to open each file in @closed, and if the open succeeds moving it from @closed to @open and recording its filehandle, the current read position, the file id (device and inode) and the blocksize.

Checking for moved files involves doing a stat by name for each file in @open, and marking the file as close_pending if either the stat fails or the device/inode has changed.

Checking for read involves doing a stat by filehandle for each file in @open, and reading new data if the filesize is greater than my current file position. Additionally if this file is marked as close_pending, I try to lock the file, and if it succeeds (which in this context means all writers to the file have finished) I close the file and move it from @open to @closed.

Thanks to everyone for your help.

Hugo