|
insensate has asked for the wisdom of the Perl Monks concerning the following question:
I have a process that wakes up occasionally and processes new files in a directory. The files are large and I want to know that the transfer of the file from the client is complete before I try to read and process the file. I tried flock(FH,2) to no avail. I'd like to avoid watching the file for several seconds for modified time change for performance reasons but could default to that if nothing better comes up. Any thoughts??
Thanks!!
Jason |
Re: Testing file transfer completeness
by mojotoad (Monsignor) on Dec 02, 2003 at 19:46 UTC
|
How are the files being transfered? The most reliable solution is to transfer to a temporary filename and have the client rename it to the proper filename upon completion. rsync, for example, does this automatically during transfers.
If you cannot control the file names adequately, (say people are FTP'ing the files in) and you are on a linux box, you could check the output of lsof and see if the file is still open by any process.
Matt
| [reply]
[d/l]
[select] |
Re: Testing file transfer completeness
by dragonchild (Archbishop) on Dec 03, 2003 at 04:29 UTC
|
I've had to deal with that very issue here. I gave a pretty decent solution here, in the same thread. The solution given is the solution currently used in a very large bank for processing a ton of various files coming in from various sources (some controlled, some not) onto various OS's. Give it a whirl!
------
We are the carpenters and bricklayers of the Information Age.
Please remember that I'm crufty and crochety. All opinions are purely mine and all code is untested, unless otherwise specified.
| [reply] |
Re: Testing file transfer completeness
by duff (Parson) on Dec 02, 2003 at 20:21 UTC
|
Just touch a file at the end of transfer and only process those files that are older than the touched file (or all of them if the file isn't there). Or if you can't touch a file on the remote system (or a reasonable equivalent), perhaps you could send a manifest (with file sizes) before sending anything else.
| [reply] |
|
