Bjoern has asked for the wisdom of the Perl Monks concerning the following question:

Hello Perl Folks,

i've got a general question concerning a little program I'm writing at the moment. The main purpose of it is, to be an "event triggered launcher". To give you an example - I would like to have the following job done:

scan a defineable list of directories, say d:\data\company1, d:\data\company2, d:\data\company3 ....
if the program detects a file (or more) in one of these directories it should check wether the file(s) is(are) completely written, something like flock "filehandle", ($LOCK_EX + $LOCK_NB) or return(1).
if yes, it should start a job "on" that particular file, e.g. move the file from that directory to a different one or send that file via ftp to host xyz or.....
That should be defineable for every directory (and/or perhaps for every file)!
And of course the program should do a good job in logging everything, one central log (normal and errors) and one log for each directory.

I've almost finished the tool but I'm not completely satisfied with it, the code looks a bit complex (300 lines, 20% comments -> perl feature or just me, being a beginner?).
My question now is, if there exist some special pieces of code, modules, libraries or functions it would be worth to have a look at, to simplify the task/code. Perhaps someone has already written something similar and I could snaffle some lines.

For example I would like to have the code to be runnable on Unix and Win32 but these back/forward slashes within the path names are driving me crazy........now it only runs on Win32 :-(

This is my first posting, so be forgiving about any newbie mistake (please correct me).

cheers

Bjoern

P.S.
The file-locking really works :-)
  • Comment on General Question: File triggered job launcher

Replies are listed 'Best First'.
Re: General Question: File triggered job launcher
by trantor (Chaplain) on Nov 28, 2001 at 14:00 UTC

    The usual suspect is File::Find!

    And as a general rule, forward slashes will work in Win32 as well. Regarding file name portability, File::Spec is another good one to check out.

    -- TMTOWTDI

Re: General Question: File triggered job launcher
by hakkr (Chaplain) on Nov 28, 2001 at 16:14 UTC
    use Fcntl qw/:flock/; #list of directories to look in my @directoryList=('/dir/dir1','/dir2'); #for each dir foreach my $dir (@directoryList){ #open dir opendir(DIRHANDLE, "$dir ") or die "could't open dir $!"; #create array of files present in dir my @files= grep(!/^\./,readdir(DIRHANDLE)); #for each file test open and test lock foreach my $file (@files){ open FILE , "/$dir/$file" || die "Could not open"; #if lock obtained do your job if (flock (FILE,LOCK_EX)){ #do your job on $file } } }
    Instead of a foreach loop you could use map to map your jobs onto each file.
      I'd have to say that while your code logic is sound, I probably would opt for using File::Find instead of this sort of hand-coded solution. The reason for this is that your code is fairly sensitive to changes in the file structure and expected layout - It doesn't allow for the likelihood of other sub-directories within your directories (and is trying to open them as files), ignores files which may start with a period (grep(!/^\./, readdir(DIRHANDLE))) and while, probably more beyond the scope of the problem, doesn't check file permissions before trying to access the file.

      These are all small(-ish) things I know, but if you take this code and extend it into a recursive search of directories, you could end up with some very nasty results with directory cross-links (especially to higher level directories). This is where widely tested modules offer the greatest advantage whereby you can to a certain extent rely on the testing and ground-work provided by others.

      Just a few random thoughts that this code provoked

       

      perl -e 's&&rob@cowsnet.com.au&&&split/[@.]/&&s&.com.&_&&&print'

Re: General Question: File triggered job launcher
by tachyon (Chancellor) on Nov 28, 2001 at 17:37 UTC

    You can use a / as the path separator on both UNIX and WIN32, give it a try. A useful data structure might be a hash like this:

    # set up a hash that indexes each target dir to an array that # contains references to the actions we want to do on that dir my %dirs = ( dir1 => [ \&log, \&foo ], dir2 => [ \&log, \&bar ], dir3 => [ \&log, \&foo, \&bar ] ); for my $dir ( keys %dirs ) { do{ print "Skipping $dir\n"; next } unless check($dir); for my $action_sub ( @{$dirs{$dir}} ) { &$action_sub($dir); } } sub check { # perform a check that we are ok to go, return true if so return 0 if $_[0] eq 'dir2'; # fake it to skip dir 2 return 1; } sub log { print "Doing log to $_[0]\n"; } sub foo { print "Doing foo to $_[0]\n"; } sub bar { print "Doing bar to $_[0]\n"; } __END__ Doing log to dir1 Doing foo to dir1 Skipping dir2 Doing log to dir3 Doing foo to dir3 Doing bar to dir3
    cheers

    tachyon

    s&&rsenoyhcatreve&&&s&n.+t&"$'$`$\"$\&"&ee&&y&srve&&d&&print

Re: General Question: File triggered job launcher
by grinder (Bishop) on Nov 28, 2001 at 21:01 UTC

    If you are in control of the files coming into the directories in question, there is a much simpler technique available that doesn't rely on file locking. This makes things much more robust.

    On the flip side, you need the cooperation of the file provider.

    Arrange to have things so that the provider sends two files, the data file itself (foobar.txt) and when all is done, files closed, buffers flushed, speakers burst, amps all wet (ah no, that's a Sonic Youth song I'm thinking of)... you then create a second file named 'foobar.txt.check'.

    Creating the check file can be as simple as open F, 'foo.txt.check' and close F. You then just have to look for check files. If one exists, you are guaranteed that the file it refers to has been written in its entirety.

    After you are done with the file, you can at least delete the check file, so that you don't attempt to reprocess it. From there, you can either also delete the corresponding data file, or else the provider can later scan the directory, and delete any data files for which there no longer corresponds a check file.

    --
    g r i n d e r
      Thanks to all who replied,
      I saw some things I already implemented, but I also got some ideas that might help me a bit with the task.

      Bjoern