in reply to Testing file transfer completeness

Just touch a file at the end of transfer and only process those files that are older than the touched file (or all of them if the file isn't there). Or if you can't touch a file on the remote system (or a reasonable equivalent), perhaps you could send a manifest (with file sizes) before sending anything else.