You can then loop for some reasonable amount of time, and if the mtime doesn't change, you can guess that the ftp server isn't writing to them anymore.

Something like:

if ( -f $control_file && -f $data_file ) {
    rename($data_file,$data_file. $$);
    rename($control_file,$control_file . $$);
} else {
    exit;
}
while (1) {
    $mtime=(stat($data_file . $$))[9];
    # drop out of the loop if the file hasn't
    # changed in 5 minutes (perhaps longer
    # for a wan connection ?)
    last if (time > $mtime + 300);
    sleep(30);
}
#process the files
[download]

You mentioned you didn't have control over the process, but if you ever do get control, here's two ideas:

• Have the ftp client send the files as file.tmp, then use the "rename file.tmp file" ftp command when the file is sent.
• Lacking that, install an ftp server that does something similar. proftpd has a configuration parameter called HiddenStor that does this.

Update: yep..I missed a sleep in the lower loop, which would burn lots of resources...fixed

Says kschwab:

Something like:

while (1) {
      $mtime=(stat($data_file . $$))[9];
      # drop out of the loop if the file hasn't
      # changed in 5 minutes (perhaps longer
      # for a wan connection ?)
      last if (time > $mtime + 300);
  }
[download]

That is a busy-wait loop. Even if the loop runs only once, you have called stat one million times and run up the load on the system for five minutes.

Try it like this:

my $old_mtime = (stat($file))[9];
if (time <= $old_mtime + 300) {
  while (1) {
      sleep 300;
      my $new_mtime=(stat($file))[9];
      last if $old_mtime == $new_mtime;
      $old_mtime = $new_mtime;
  }
}
[download]

Right, I should have slept() in the loop. I'm not sure how one loop runs stat() a million times though :)

Fixed my code and re-posted. Thanks for the catch.

Use stat to compare the inode access/modify/change times for the data and control files. Also check that the control file is big enough to be finished. If all the inode times are near equal, and the files are resonable sizes, you may be able to assume the file is okay.

my $name = 'foo.txt';
my $cnrl_name = $name . '.ctrl'; 
if ( ((stat($cnrl_name))[9] > (stat($name))[9]) and ((-s($cnrl_name))>
+10) )
{
  copy $name, "dir/$name";
  unlink $name, $cnrl_name;
}
[download]

Thanks repson - I'll give that a go and see how I get on :-)

Pete

I'd strongly recomment that you follow kschwab's advice and rename the data file to the same directory before you start to copy or process it. Copying takes time, and you don't want the remote process to start updating the file while your process in in the middle of copying it. Renaming it first will help prevent that.

Then, after you rename it, check to see if it's still changing, because it's possible that the remote process opened the file just before you renamed it.

There's still a race condition, but it's much smaller. Proper design would have been for the remote program to never send the same data twice and never use the same filename twice. I also recommend that you find the person who designed this broken protocol and kick them in the ass.

An alternative might be to use lock files. Have the FTP server send a lock file before it starts uploading, remove it when it's done, and have your script honor it. If you have a great deal of control over the FTP process on the other end, such a lock file could work both ways.

Thanks guys - with the advice that's been offered, I can now implement a solution with is as safe as it can be, considering the interface.

As regards the design of the interface - I had a little input into it, however, the developer of the software which runs on the remote machine had 'done this sort of thing before', and also claimed that no crc/md5 checking needed to be done because 'in [his] experience, ftp is a totally safe transport medium' - ah well, perhaps we can fix the interface with v2 ...

Pete

'in his experience, ftp is a totally safe transport medium'

Which, of course, explains the existence of SSH and SCP.

For all the recent concern I've seen around the Monastery regarding security, I'm suprised nobody has pointed out the inherent flaws in the concept of using FTP to transfer your files, your username, and your password in plain text across the internet.

SCP, an encrypted drop-in replacement for FTP, is a great alternative, and there is even a Net::SCP module that should allow you to keep using your current scripts, even if they currently rely on Net::FTP.

And for a secure (encrypted) remote shell prompt in Perl, you can't beat the Net::SSH module.

I realize it's probably too late (or perhaps completely unfeasible ;-) to switch to these in the middle of your project, but maybe you or someone else who reads this can use these in future projects. Take a look at the secure alternatives to FTP and Telnet; now that the US encryption export laws have changed (and the RSA patents have expired) you can use them for free on almost any OS.

By 'safe', he meant that the file would arrive at the destination without corruptions.

Currently, because all the network connections are on a private WAN, security is considered an obstruction! This attitude drives me nuts, but my recommendations fall on deaf ears.

However, I get the impression that someone higher up is organizing security audits (not just of the computer networks), so hopefully this attitude will fade away once the right incentives are in place (such as dismissals, etc ;-)

Pete