I went for this approach, and it seems to work fine, giving good responsiveness without causing a detectable load on the system.

The basic loop is something like this:

  my @closed = list_of_files();
  my @open = ();
  my $count = 0;
  while (1) {
    my $time = Time::HiRes::time();
    check_moved(\@open) if ($count % $CHECKMOVED) == 0;
    check_new(\@closed) if ($count % $CHECKNEW) == 0;
    check_read(\@open) if ($count % $CHECKREAD) == 0;
    ++$count;
    my $delay = $TICK - (Time::HiRes::time() - $time);
    Time::HiRes::sleep($delay) if $delay > 0;
  }
[download]

I'm currently using constants $TICK=0.1, $CHECKREAD=1, $CHECKNEW=20, $CHECKMOVED=50, but later I plan to make these more dynamic based on the size of the file list and other details.

Checking for new files involves trying to open each file in @closed, and if the open succeeds moving it from @closed to @open and recording its filehandle, the current read position, the file id (device and inode) and the blocksize.

Checking for moved files involves doing a stat by name for each file in @open, and marking the file as close_pending if either the stat fails or the device/inode has changed.

Checking for read involves doing a stat by filehandle for each file in @open, and reading new data if the filesize is greater than my current file position. Additionally if this file is marked as close_pending, I try to lock the file, and if it succeeds (which in this context means all writers to the file have finished) I close the file and move it from @open to @closed.

Thanks to everyone for your help.

Hugo