Then just check for any files with -M >= -M last-run
I think this is not a sure way to check that a file is not being written to though, say, an FTP process might have stalled, or a long running process that hasn't flushed its file handle yet...
To reliably check for the completion of a file, I think it is necessary to introduce some trigger files, or to have the process that creates the file to write to a different extension first, and then rename it to the matching extension.
| [reply] |
It's no worse than what he's already doing. Is there a way to ask the OS if any process has a file open for writing? I don't know of a way to do it off the top of my head. I think you'd need cooperation from the process doing the writing, which this person doesn't have.
| [reply] |
Hi perrin, on the unix system, use fuser (system V), fstat (BSD), pff (public domain) to find out various things about processes using a particular file (or files).
I responded to your post earlier simply because I thought just checking a file's size to find out if it is still being written to is not a reliable method, not really related to the original post.
| [reply] |
Won't this pound your CPU and disk by running in a tight loop?
Yes it does. Unfortunately the files have to get processed *immediately* so a cron job won't work here.
A bit more info would help here. A legacy third party app (of which I have no control over) drops a file into a directory and waits for the file to be processed by my Perl program. The Perl program processes the file and appends results to it before renaming the file to something the legacy app is expecting (the file extension is changed to .done). Once the legacy app sees the .done extension it knows processing is complete and it reads the processing results from the file. Behind that legacy app is a patient user waiting for their results hence the reason why the files must be processed immediately.
Seeing as I'm kinda stuck in legacy app hell, is there anything else my Perl daemon could do to aleviate the CPU & disk pounding here?
| [reply]
[d/l]
[select] |
That's pretty nasty. I can't think of anything simple that will solve the essential problem, i.e. polling the directory, but you could probably help things out just by adding a sleep(1) to the end of your loop.
| [reply] |
This reminded me of something I've been meaning to play with in the linux kernel:
CONFIG_IMON:
This enables support for imon, the inode monitor. Through imon and fam, the File Alteration Monitor, programs can express interest in individual files and directories, and the kernel will notify those programs when the files change. This enables desktops, mail readers, administration tools, etc. to respond to changes in the system immediately. If you don't enable imon, such programs will have to poll files (by checking them every few seconds) to determine whether they've changed.
See http://oss.sgi.com/projects/fam/ for more information on imon.
If you don't know whether you want imon, it doesn't do any harm to say N here. (If you want imon, say M, not Y.)
This option lives under 'File Systems' and is called 'Inode Monitor Support'. If this file processing is important enough this may help you - assuming you are on linux...
--tidiness is the memory loss of environmental mnemonics
| [reply]
[d/l] |