If you control, or can influence the process that uploads the file, have it append the filesize to the name. Then testing for upload completion before download becomes trivial.

Ie. mydatafile.tar.gz becomes mydatafile.tar.gz.123456789

If you have control of the writes on the remote server, create it as a temp file and rename/move it to the correct name once the file writes are complete.

Can you have your remote server deal with the issue of indicating when it is done? One solution that could work is a maildir structure. Create tmp file, rename to include the inode number, do the writing, then move into the "done" directory when complete. Assumption being that the temp and done directories are on the same file system.

If the file is on a remote ftp server, call size on it, sleep a bit and call size again. If the sizes are the same, it's probably done.

I don't think there's a definitive way of knowing whether some remote process is done writing.

mmittiga17:

At my previous job, we had the same issue. We solved it by having our data provider add a "summary" report to the end of the upload, so we wouldn't upload any data files until that one appeared. We actually didn't even use the contents of the summary file, though we could have. (It just contained the names and lengths of the data files uploaded, along with the record counts, batch date and batch totals.)

...roboticus

When your only tool is a hammer, all problems look like your thumb.