select, stat, and waiting for files

oakbox has asked for the wisdom of the Perl Monks concerning the following question:

I have a remote process that takes between 10 and 30 seconds to complete. This process creates a 1 to 3 meg file and then scp's that file to my server. The process on my server knows where that scp'd file will show up, but needs to wait until it is completed before sending it on to the user.

Let's assume that the remote process is doing it's thing, and that I am now, sitting on my server, waiting for the result file.

  my $sanity;
while(1){ 
  if(-e $pdffile){ last; } 
  sleep 1; 
  $sanity++; 
  if($sanity > 60){ return(0); }
  }
[download]

When I get through that chunk, I know the file exists. Yeah! Now I need to wait for it to stop growing...

while(1){
select(undef, undef, undef, 0.5); # .5 second wait
my $usizer = (stat($pdffile))[7]; # how big is that file?
if($usizer eq $sizer){ last; } # did it grow at all?
$sizer = $usizer; # set size for next round
}
[download]

Now for the question: Is this the *right* way to do this? Another programmer mentioned that using select/stat isn't reliable due to disk buffering/ network concerns.

- Thank you,
Richard

oakbox

Comment on select, stat, and waiting for files Select or Download Code

Replies are listed 'Best First'.
Re: select, stat, and waiting for files by osunderdog (Deacon) on Sep 12, 2005 at 12:12 UTC
First off, I assume you're on a *nix machine. But figured I should state that in advance. One problem you are facing is the file comes into existence, but isn't quite all there yet. I've seen this handled two ways 1) create the payload file in another location and then use a local system move to put it in the monitor directory and 2) move the payload file then move a trigger file (zero byte file) to the monitor directory. Neither of these techniques are perfect, but they might be good enough™ Another option might be to use something like dnotify or it's ilk to tell you when the file has arrived. Although these will depend on your os. I've tried to use an os dependent directory monitor, however, I ran into problems when the monitor directory went from a local disk to an NFS mount. That's a much harder problem to solve. Just some thoughts Hazah! I'm Employed!	[reply]
Re: select, stat, and waiting for files by gri6507 (Deacon) on Sep 12, 2005 at 12:44 UTC
To go along with the previous idea, before starting to scp'ing your file, touch a filename.size file which only contains the size of the file you are about to scp. That way, your server could look at the size file and figure out if the actual file is completely there.	[reply]
Re^2: select, stat, and waiting for files by gargle (Chaplain) on Sep 12, 2005 at 12:59 UTC
Or even better, if you're going to do it like that: scp the file and at the end ssh and touch a filename.ok. When filename.ok shows up you know that the first file has been send already. Or, scp the file and log the scp session in a seperate file. Then, scp this second file. When the second file arrives you know the first one has arrived. Later on you can check the contents of the seperate file to look for transfer errors. -- if ( 1 ) { $postman->ring() for (1..2); }	[reply]
Re: select, stat, and waiting for files by sasikumar (Monk) on Sep 12, 2005 at 13:23 UTC
Hi Richard, I Would be sending a signal to the server process stating the scp is over. I feel that would be a best method. But i am not sure as i have not dealt with such huge files across network. Thanks SasiKumar	[reply]
Re^2: select, stat, and waiting for files by samizdat (Vicar) on Sep 12, 2005 at 14:12 UTC
The problem with this approach is that the signal will probably appear before the file is actually completely written to the disk, due to the write-caching of most modern 'NIX systems. This makes it dangerous to assume anything based on the scp.	[reply]
Re^3: select, stat, and waiting for files by sgifford (Prior) on Sep 12, 2005 at 15:38 UTC
It doesn't matter whether the file is physically written to disk; the kernel will pretend that it is, and `scp` will use the kernel to access the file so will see it that way. The only time having the file physically on disk would matter would be if the machine shut off unexpectedly, or if something was accessing the disk directly instead of using the kernel's filesystem drivers.	[reply]
Re: select, stat, and waiting for files by samizdat (Vicar) on Sep 12, 2005 at 13:00 UTC
After you complete the send, have your program send a tiny file along. When that file's send is complete, you know you have all of the big one. This should take much less than 1/2 second :D	[reply]