Batch Processing Files

Cody Pendant has asked for the wisdom of the Perl Monks concerning the following question:

I have a very long, very repetitive set of tasks I'd like to do which is relatively slow, so what I'd like to do is fill a batch folder with files listing the tasks, then add files to it. That way the script can churn through files 1 to 10, while I get files 11 to 15 ready, and just dump them in without the script having to stop.

Here's my first attempt:

$batch_dir = "batch/";
$log_dir = "batchlog/";

while (1) {
    opendir( D, $batch_dir ) or die;
    print "checking $batch_dir for files\n";
    @batchfiles = grep { !/^\./ } readdir(D);

    unless (@batchfiles) {
        print "Done: no more files to process\n";
        exit;
    }

    # batchfiles contains all files at 
    # most recent readdir() 
    foreach $file (@batchfiles) {
        open( F, "$batch_dir$file" ) or die "$!";
        while (<F>) {
            do_stuff_with($_);
            # actual processing omitted
        }
        close(F);
        rename("$batch_dir$file","$log_dir$file") || die "$!";
        print "finished $file\n";
    }
}
[download]

Are there any problem, omissions, Traps for the Unwary in that that fellow monks can see?

($_='kkvvttuubbooppuuiiffssqqffssmmiibbddllffss')
=~y~b-v~a-z~s; print

Comment on Batch Processing Files Download Code

Replies are listed 'Best First'.
Re: Batch Processing Files by davido (Cardinal) on Jul 08, 2004 at 02:28 UTC
You could check for the existance of files, and then sleep for ten seconds if none exist. That's going to be a little more processor and IO-friendly to other processes on the system than repeatedly checking for new files as many times per second as is possible. Your script is also assuming it will never see files of the same name twice. That may be fine, but just something you should keep in mind. Also, I think it would be a good idea for @batchfiles to be lexically scoped (as well as any other variables within the while(){...} loop). This is important because with a script that's going to be idling a long time, and iterating over and overagain, you'll have to remember to either reset all variables within the loop, or let them fall out of scope. The latter is automatic if you use lexicals. Dave	[reply]
Re: Batch Processing Files by ercparker (Hermit) on Jul 08, 2004 at 02:53 UTC
Hopefully this will work for you: in the past i've setup something similiar to run on a crontab entry Then you could just dump the files into the batch directory and they would get processed when the script runs at the set interval. I'm assuming/hoping your'e on a unix or linux system and this is a task that would need to happen on a hourly, daily or weekly etc. occurrence	[reply]