Re: Monitoring an FTP upload directory

If you can monitor the FTP server's xferlog log file, that is simply the best way to ensure you don't process incomplete files. The FTP server only makes an entry in that log file when the file transfer is 'finished'. In addition the entry notes whether the transfer was c=complete or i=incomplete. In my hacked-together monitor program I can check and process only finished transfers, which were complete, and whose path matches one of several possible directories being acted on. And react so fast that remote users sometimes re-transfer the file two and three times wanting to be sure it really did disappear "on purpose". (sigh)

I've not had to use the features for renaming files that wjw mentions. Though if you can't see the xferlog file that would be a very good way to do it. And, yes, we use ProFTPD also for its extreme flexibility.

Comment on Re: Monitoring an FTP upload directory

Replies are listed 'Best First'.
Re^2: Monitoring an FTP upload directory by wjw (Priest) on Nov 07, 2006 at 21:25 UTC
We had also had the problem of cutomers uploading more than once. The files they uploaded were very large (50-60 meg) so they would start a transfer, go away for a while and come back noting that the file was not there and disregarding the fact that their client showed a successful transfer, then start the whole thing over again. We addressed that problem by actually placing a text file in the upload directory which had a name like (file_name)xfer_complete.txt. The contents of the file were always the same; something like "your file has been recieved and moved to a secure place on our network" if I recall correctly. Most customers caught on right away, the others we would 'coach' if we noticed they needed it. We removed these files after a few hours, and our scp script was coded to ignore them. Saved us a lot of those Sighs! :-)Just an idea to save some bandwidth for you and your customers. :-) ...the majority is always wrong, and always the last to know about it...	[reply]
Re^3: Monitoring an FTP upload directory by friedo (Prior) on Nov 08, 2006 at 00:29 UTC
I think this is a great idea, thanks. I'm going to go with tailing the xferlog, too. (Why didn't I think of that before? :) ).	[reply]
Re^3: Monitoring an FTP upload directory by shenme (Priest) on Nov 08, 2006 at 02:48 UTC
I've really wanted to do this, like creating a <your_filename>.received_correctly_and_processed.txt file. Automatically removing the file after a while makes it sound even better. The problem we often saw was that we were _so_ fast at picking and moving the file, that the user wouldn't ever see it, even when directly following the PUT with a DIR. It is one thing to see the transition from 'there' to "not there", but it kinda freaked them out that they never saw it at all. Like it had gone into the bit bucket. And our applications are named _somewhat_ differently...	[reply]