gnu@perl has asked for the wisdom of the Perl Monks concerning the following question:

I need some opinions / ideas here. I have a situation where a third party vendor is delivering files via ftp to one of our servers. They do not have the ability to do a 'rename' or any other ftp command outside of 'put'. We need to be able to tell when the file is complete and then move it to a processing directory.

A few methods have been used here before, but I don't really like them. They work fine, I'm just picky I guess.

One method was to watch the size over time. If the file didn't grow for X amount of time then consider it complete and begin processing. This has it's obvious flaws, just one of which being a slow network connection.

The other method was to perform and 'fuser -u' on the file and see if anything came back. Effective, but I prefer not to have to shell out of a perl script if I don't have to(I would like to keep this in perl).

Does anyone have a better way to see if a file is open or currently being written? It is possible to use Proc::ProcessTable, but that would be pretty ineffecient to run every 5 minutes (or sooner).

TIA,
Chad.

Replies are listed 'Best First'.
Re: Detecting An Open File
by valdez (Monsignor) on Jul 21, 2003 at 19:04 UTC

    A possible solution would be to test size of file and if it doesn't change for a time longer than the timeout of the connection, then the file is closed. Of course, you can't tell if the file is complete or the connection has timed out.
    A better solution would be to receive another file with checksum and size of the file to be transferred.
    This topic has been discussed some time ago at looking for perl module which returns if a file is open; the module for Linux will probably change due to a different implementation of /proc fs in Linux v2.6.

    HTH, Valerio

[reply]
Re: Detecting An Open File
by flounder99 (Friar) on Jul 21, 2003 at 18:53 UTC
    Check out this node: fuser in perl it is set up for linux but you might be able to get it to work under solaris. (I'm not at all familiar with where solaris stores this kind of info)

    --

    flounder

[reply]
      Thanks for your input, this idea would work on solaris, I have actually already done something quite similar using Proc::ProcessTable.
[reply]
Re: Detecting An Open File
by Anonymous Monk on Jul 21, 2003 at 21:21 UTC
    How about having the vendor do somwthing like this ...

    $ touch file_to_be_sent.end
$ ftp your.server.com
...
> put file_to_be_sent.dat
> put file_to_be_sent.end
> bye

    [download]
    This way you will know file_to_be_sent.dat has completed uploading when file_to_be_sent.end appears on your file system.
[reply]
[d/l]
Re: Detecting An Open File
by waswas-fng (Curate) on Jul 22, 2003 at 05:04 UTC
    What I have done in the past is used a lock file that is generated on the sending side and that you delete after proccessing the file.
    For instance: 

    Empty ftp dir
Client connects to ftp server and drops active file (testfile.zip)
Client verifys ftp completed and puts another known file (testfile.zip
+.lock)

    [download]

    Meanwhile your application loops and does nothing until it sees testfile.zip.lock exists whne it then grabs that file and does something with it and deletes the file and the lockfile. The other side can be configured to check for the lockfile before it sends the file to verify that proccessing has been completed without clobbering data.

    -Waswas
[reply]
[d/l]
Re: Detecting An Open File
by blue_cowdawg (Monsignor) on Jul 21, 2003 at 18:16 UTC

        Does anyone have a better way to see if a file is open or currently being written?
    What OS? Linux? Solaris?

    Peter L. BergholdBrewer of Belgian Ales
    Peter@Berghold.Netwww.berghold.net
    Unix Professional
[reply]
      Sorry about that, Solaris (Sparc, not that that part matters).
[reply]

        /etc/fuser will tell you what PID has the process open.

        Peter L. BergholdBrewer of Belgian Ales
        Peter@Berghold.Netwww.berghold.net
        Unix Professional
[reply]
Re: Detecting An Open File
by TVSET (Chaplain) on Jul 21, 2003 at 21:53 UTC
    In the TIMTHOWTDI spirit, you could configure your FTP server to allow only 1 simultanous login for username in question. This will work as a very simple locking mechanism, where only one user will be able to access file on the FTP. One of the users will be your vendor, and another - your perl script.

    Leonid Mamtchenkov aka TVSET

[reply]
Re: Detecting An Open File
by rir (Vicar) on Jul 21, 2003 at 20:40 UTC
    It sounds like you don't have other file clobbering issues. So you might find a ftp server that can output helpful debugging info that you could decipher. Don't know, haven't looked.
[reply]

Back to Seekers of Perl Wisdom