roboticus, no, the "Invalid cross-device link" error is caught by File::Copy::move() and causes it to try to copy the file and then unlink.

runrig, no, "trying to delete the file while it is still being written to" is not in evidence in the traces provided. The trace shows the destination being unlinked (despite it never having been created) after the source could not be opened.

As the original poster correctly noted, the problem is:

open("/mnt­/xfer/3165­0_WRETCHED Radio 04-25_DIST­.wav", O_RDONLY|O­_L
+ARGEFILE­) = -1 ETXTBSY (Text file busy
[download]

So the addition of $! will likely add "Text file busy" to the output.

I've only ever seen that error when trying to delete an executable image when it was in use. On a good Unix platform, "man 2 open" will actually note more details about what "Text file busy" really means in the case of open() failing.

Indeed, open(2) says:

ETXTBSY

The file is a pure procedure (shared text) file that is being executed and the open() system call requests write access.

which doesn't match the name *.wav nor O_RDONLY so the original poster might want to check "man 2 open" on his particular operating system (and hope that it has "good" man pages in this respect).

Let me add a little more information. This script transfers several dozen files per day. The files are either plain text, mp3 or wav audio files. It may run flawlessly for several days, then it might crash 5 or 6 times in one day.

If it crashes during the transfer attempt of a file, when restarted, it transfers the file without problem.

Hence, I believe the problem is somehow timing related rather than the actual command structure. If the problem was really related to "Invalid cross-device link" errors or the like, would that not then cause the script to crash every time? I would think that type of error is more a result of an underlying problem.





BTW, this is my first post on Perl Monks. I am truly impressed by the speed and quality of responses. Thank you all very much. I am anticipating a solution soon, thanks to your collective knowledge and kindness.



Michael

How do you know that the file has finished being written before you start copying it? In such situations, I usually recommend that you have a standardized naming convention for "still writing to the file" like "*.tmp" and then have whatever process is writing the file rename it to remove the ".tmp" after it is finished writing it.

If after you have (already?) solved the above concern, the "text file busy" problem (besides causing me to research what that error really means which might even lead me to reading OS source code) would lead me to detecting move() failing and leaving $! set to ETXTBUSY (from Errno) and then just sleeping and retrying a couple of times.

- tye        

Hence, I believe the problem is somehow timing related rather than the actual command structure. If the problem was really related to "Invalid cross-device link" errors or the like, would that not then cause the script to crash every time

Not all. After all, "Text file busy" means *something else* (another handle? another process?) is keeping the file "busy" (mmapped?).

UPDATE: It's clear that I was out in left field. Thanks to runrig and tye for clarifying and correcting things.

move() is a function in File::Copy

Perhaps a solution might be in the case of a text file busy error (see $!), instead of dying right away, sleep and retry, or skip and wait for the next iteration of your move script. If the file ages too much, then emit a warning or error of some sort.

If it is truly just a temporary error, then retrying at a later time might be fully appropriate.

What does your script do to ensure the file being put into the folder is completely copied there?

Possibly, you are trying to move the file out before the creating process finishes writing the file. Not that the error you are getting, ETXTBSY, hint that this is the case. Neither does ETXBSY hint at a reasonable cause based on your description.

Perhaps moving the file out before it has been fully written is part of your current problem. Even if not, your description hints that this may be an issue. See Check whether a file is open for a discussion of the problem.

Since I am using the move() function from the File::Copy module, I have been relying on the correctness of the module to check the various things mentioned.

In comparing the two strace excerpts, it appears to me that the error is occurring before the move process actually begins. In the strace segment that does work, I see that if the open attempt is successful, then the read/write sequences begin.

I am still quite new to this level of script work. But if I am understanding the strace output correctly, this error is occurring before anything actually touches the file itself. It seems that something is opening the file and not getting it closed in time for the next open attempt. I just don't know what.

May this involve delving into the File::Copy module and possible needing some kind of change or delay there? I have looked at the module, but the majority of it is way over my head.


Thanks,
Michael

I have been relying on the correctness of the module to check the various things mentioned.

Um, File::Copy::move()'s documentation says nothing about "Waits for the source file to be finished being created and then...".

Trying to get the monitor/copier process to correctly determine whether the to-be-moved file is finished being written is likely a hard problem, perhaps an impossible problem (depending).

That is why I outlined the "usual" way to have whatever is dumping these files into the source directory to do it in a way that makes it clear when the file being dumped there has been completely dumped (and is thus ready to be moved by the monitor/copier).

- tye        

# check if the file has stablized for over 30 seconds
if (is_stable_file($file, 30)) {
   process($file);
}

sub is_stable_file {

    my $file                  = shift;
    my $wait_seconds          = shift || 0;

    my $result                = 1;

    if ($wait_seconds > 0) {

       my ($mtime)            = (stat($file))[9];
       my $secs_since_mod     = (time - $mtime);

       if ($secs_since_mod < $wait_seconds) {
          $result             = 0;
       }

    }

    return $result;

}
[download]