(This solution may not scale well if there are many directories, but you'll be able to adapt to your case)

Create a fifo (mknod -p fifoname) and have a script continuosly read from it. If there's nothing in the FIFO the process won't consume resources.

Programs to run are simply inserted in the FIFO as strings; in the cron job, you substitute the script execution with a write into the FIFO, i.e. you prepend echo and append a redirection to the FIFO. The reader script will extract and execute them, via some eval mechanism, or fork(), or whatever you like.

There are tons of theme variations, but I think this should give you the idea.

Interesting idea. However, I don't have much control over the system and the reader script could be killed accidentally (or intentionally), so that's a little concerning.

What I was hoping for was some sort of IPC... the same script adding to the FIFO and being awoken.

The only problem with the sleep above is that the processes could get out of order... I'd rather keep it all in the same script if possible.
--------------
"But what of all those sweet words you spoke in private?"
"Oh that's just what we call pillow talk, baby, that's all."

You can take into account the accidental kill by putting some monitor script in the crontab itself - but you would have an explosion of scripts :)

My previous solution was clean in the sense that it voids active waits. If you don't bother having slightly active waits, you could keep a queue file in each directory, with write access protected via a lock, in which you append your 'name' (e.g. pid number), then keep controlling until it's your turn. When you're done, you simply remove yourself from the top of the file (using the same locking to prevent multiple writings). In this way, the access to the queue file should be rather quick, then a check every second should suffice without slowing your machine to a crawl.

Note that if you use PIDs, you can also check if the current first-in-the-list is alive, just to prevent infinite locking by a badly dead script.

Flavio

Don't fool yourself.

Obviously you can use a database for this. It might be overkill though, but using transactions and the right isolation level can make for a very nice queue.

I guess an alternative is to have a directory that is used for locking. The directory would contain a mutex lock file that is used to show that a job is already running as well as a file per job queued up (you could use numeric increasing filenames to ensure the jobs are processed in order.) The process would look like this:

1. Process starts, tries to get lock on mutex file.
   1. If the lock is obtained the directory is scanned for job files. If job files exist then the path the process was originally tasked to perform is written to the directory as a job file and the first job in the queue is performed.
   2. If the lock fails then there is already a task running, so simply write a job file and exit.
2. Directory is processed.
3. Job file is deleted.
4. Prior to termination and the relinquishing of the lock the process rescans for pending job files. If any are present it processes them in order until there are no jobs pending.

Anyway, thats probably what i would do if I wansnt going to to use a DB.

Yes, I'd considered using a database, but I wanted to avoid it if possible. Plus it's too easy :) I would imagine the I/O overhead for the files (and shell overhead for execution) would be more than the DB overhead (plus I'm already using a DB, so the connection overhead is already accounted for).

There are certain things in memory of that particular process (command line parameters, time of execution, etc.) that require me to keep the process around (sleeping) - I could write all necessary things out, but... that sounds too complicated and less maintainable... so I'm thinking:
1. Create a queue file w/ PID ($$.queue)
2. Get oldest queue file (ls -rt | head -1)
3. Check to see if that process is running (kill 0 $file)
4. If not running, remove $file and repeat step 2
5. If running, is $file == $$?
6. If not, sleep for a while, then go to step 3.
How does that sound?
--------------
"But what of all those sweet words you spoke in private?"
"Oh that's just what we call pillow talk, baby, that's all."

There is a major problem here if two processes start in the same second (that is have the same timestamp).
-- gam3
A picture is worth a thousand words, but takes 200K.

I've done exactly that, but if you use the file's last modified time, then you don't need to worry about the filenames for the jobs as long as they are uniq.

Several things could be done to improve the code: 1) the pid could be output with the queue numer ($lock) so that the kill 0 recomened by RazorbladeBidet could be used; 2) better exit handling is needed so that improvement 1 is not needed; 3) the queue (countfile) could have a more robust structure.

use strict;

use Fcntl ':flock';
use Time::HiRes qw ( usleep );

sub check_queue {
    my $lock = shift;
    my $top;
    open(FILE, "<countfile") || die;
    flock(FILE, LOCK_SH);
    seek(FILE, 0, 0);
    $top = readline FILE;
    flock(FILE, LOCK_UN);
    close FILE;
    return $lock == $top;
}

sub add_to_queue {
    my $lock = 0;
    if (open FILE, "+<countfile") {
        flock(FILE, LOCK_SH);
        seek(FILE, 0, 0);
        for (readline FILE) {
            $lock = $_;
        }
        $lock++;
    } else {
        open(FILE, ">>countfile") || die;
    }
    flock(FILE, LOCK_EX);
    seek(FILE, 0, 2);
    printf FILE "%d\n", $lock;
    flock(FILE, LOCK_UN);
    close FILE;
    return $lock;
}

sub remove_from_queue {
    my $lock = shift;
    my @procs = ();
    if (open FILE, "+<countfile") {
        flock(FILE, LOCK_SH);
        seek(FILE, 0, 0);
        for (readline FILE) {
            if ($_ == $lock) {
                next;
            }
            push @procs, $_;
        }
        flock(FILE, LOCK_EX);
        seek(FILE, 0, 0);
        truncate(FILE, 0) || die;
        print(FILE @procs);
        flock(FILE, LOCK_UN);
        close FILE;
    } else {
        die;
    }
}

my $done = 0;
my $lock = add_to_queue();
eval {
    open LOCK, "./lockfile" || die;

    while (!$done) {
        flock(LOCK, LOCK_EX);
        last if check_queue($lock);
        flock(LOCK, LOCK_UN);
        usleep 50000;
    }
    print STDERR "Doing work on $lock ", scalar localtime, " \n";
# Do your work here
    sleep 3;
    print STDERR "Done working on $lock ", scalar localtime, " \n";
    remove_from_queue($lock);
    flock(LOCK, LOCK_UN);
};
if ($@) {
    remove_from_queue($lock);
}
[download]

Thank you for pointing out that flaw, gam3 - I had figured the timestamp was more granular than that. I will try something like you have suggested instead.

However, I'm thinking of several other options:
1. Not requiring a program to remove itself from the queue. This will force the other programs to check to see if the process is still running, yes, but a process could always die, so, IMHO, I think the check_queue would always have to have a kill 0.
2. Having a program signal the next process to wake up by sending an alarm and then bumping the sleep length up to a larger number. The wasted time between one process execution and another is less important than the wasted CPU cycles from waking up and checking on processes.
Update: I just thought of this... what if instead of the single queue file, or creating a file named PID.lock and sorting by modification time, I were to use Time::HiRes' gettimeofday and name the file gettimeofday.PID.lock (with the values in there of course)

Then... sorting the files numerically, should work, no?
--------------
"But what of all those sweet words you spoke in private?"
"Oh that's just what we call pillow talk, baby, that's all."

#!/usr/bin/perl -w

use strict;
use Fcntl ':flock'; # import LOCK_* constants

my $lockfile_path = "./lockfile";

open(LOCKFILE, ">$lockfile_path") or die "Can't open $lockfile_path: $
+!";
flock(LOCKFILE,LOCK_EX);
print "got lock @ARGV\n";
sleep(10);
print "finished sleeping @ARGV\n";
flock(LOCKFILE,LOCK_UN);
[download]

./test.pl 1 &
./test.pl 2 &
./test.pl 3 &
./test.pl 4 &
[download]

I didn't see that documented anywhere (and I have no man pages installed for flock, fcntl or lockf on the AIX machine I am working on!) so I tested it out.

By hand it seemed to work... but running it in a script of

#!/bin/ksh

testlock.pl 1 &
testlock.pl 2 &
testlock.pl 3 &
testlock.pl 4 &
testlock.pl 5 &

gave me this:



got lock 4
finished sleeping 4
got lock 3
finished sleeping 3
got lock 1
finished sleeping 1
got lock 5
finished sleeping 5
got lock 2
finished sleeping 2

So I don't think that's supported on all platforms
--------------
"But what of all those sweet words you spoke in private?"
"Oh that's just what we call pillow talk, baby, that's all."

I had only tried it by hand, but it worked fine when I ran it as per your ksh script just now. I'm on Solaris 5.8. Guess it's not something you can rely on though.

1. mknod $$.lock
2. open $$.lock,"rw"
3. (hard)link $$.lock nach reallock

• Can link:
  1. (re)open reallock,">"
  2. (if error begin anew)
  3. close $$.lock
  4. unlink $$.lock
  5. You Can Work Now
• Can't link:
  1. open reallock,"rw"
  2. reopen reallock,"<"
  3. close reallock(rw)
  4. read from reallock(<)
  5. on eof unlink reallock
  6. close reallock(<)
  7. begin anew