Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

hi,
i want to auto detect when a particular file is uploaded to ftp server, which is running on Linux machine, and call a perl script which processes that file. the file is uploaded to ftp server on a daily basis. It can land on ftp server at any time.Are there any utilities in linux or in Perl language that help me to do this thing..

thanks
kamesh
  • Comment on auto detection of file upload on ftp server

Replies are listed 'Best First'.
Re: auto detection of file upload on ftp server
by thospel (Hermit) on Oct 15, 2004 at 09:36 UTC
    On linux you can use the dnotify mechanism, for example using this program. Interesting enough there doesn't seem to be a direct perl module for it, though it shouldn't be very hard to do.

    update A later node teaches me that there is a perl module: SGI::FAM also works on linux. It needs you to have the fam library installed though.

    update Here is some pure perl code interfacing to linux dnotify. Unfortunately there seems to be no way to get to the siginfo data. This example watches directory /tmp, but you can also watch an (existing) file. You'd probably do the real detecting of what changed not in the signal handler but after the pause(). I used a realtime signal to allow for signal queueing, but since you can't get to the siginfo information anyways (to get the fd on which there was a change), it probably makes sense to leave the signal on e.g. SIGIO, and just rescan all places you are watching.

    #!/usr/bin/perl -w use strict; use POSIX qw(pause sigaction SA_SIGINFO); use constant { DN_ACCESS => 0x00000001, DN_MODIFY => 0x00000002, DN_CREATE => 0x00000004, DN_DELETE => 0x00000008, DN_RENAME => 0x00000010, DN_ATTRIB => 0x00000020, DN_MULTISHOT=> 0x80000000, F_SETSIG => 10, F_LINUX_SPECIFIC_BASE => 1024, SIGRTMIN => 32, SIGRTMAX => 64, }; use constant F_NOTIFY => F_LINUX_SPECIFIC_BASE+2; my $signum = SIGRTMIN+1; my $dir = "/tmp"; my $new_action = POSIX::SigAction->new(sub { print STDERR "Change in $dir"}); $new_action->safe(1); sigaction($signum, $new_action) || die "Could not sigaction signal $signum: $!"; open(my $fh, "<", "/tmp") || die "Could not open $dir: $!"; fcntl($fh, F_SETSIG, $signum) || die "Could not set IO signal to $signum for $dir: $!"; fcntl($fh, F_NOTIFY, DN_MODIFY|DN_CREATE|DN_DELETE|DN_MULTISHOT) || die "Could not set F_NOTIFY for $dir: $!"; pause while 1;
    After each change you can now try to detect to see if a new upload has finished. However, you'll need to keep this process running and probably will be doing multiple checks while the file is still in the process of being uploaded. So in the end it's not clear that you gained much over not using dnotify at all and just periodically check for a finished upload.
Re: auto detection of file upload on ftp server
by gellyfish (Monsignor) on Oct 15, 2004 at 09:40 UTC
Re: auto detection of file upload on ftp server
by Roger (Parson) on Oct 15, 2004 at 12:23 UTC
    It is easy to detect the arrival of a file, the hard part is to verify that the file transfer has completed or not. There are several methods to do this. Three methods on top of my head:

    Method 1 - two step process. Step 1 to wait for a file, step 2 to check the size of the file and wait for it to stop growing/changing for, say, 5 minutes to ensure that the transfer is complete. This method is dangerous, because the connection could drop halfway during processing, but your process would not know and carry on with a half completed file. The result is not pleasent.

    Method 2 - let the remote process send a trigger file after the file is transmitted successfully. Your process will wait for the trigger file, say, with the name of the file you are waiting for, plus the .trig extension. The problem is that remote systems may not be able to send the trigger file (3rd party application).

    Method 3 - use a self verifying file format, like a zip file. The zip file carries CRC for file content validation.

A reply falls below the community's threshold of quality. You may see it by logging in.