in reply to Locking/unlocking program execution using a queue

Obviously you can use a database for this. It might be overkill though, but using transactions and the right isolation level can make for a very nice queue.

I guess an alternative is to have a directory that is used for locking. The directory would contain a mutex lock file that is used to show that a job is already running as well as a file per job queued up (you could use numeric increasing filenames to ensure the jobs are processed in order.) The process would look like this:

  1. Process starts, tries to get lock on mutex file.
    1. If the lock is obtained the directory is scanned for job files. If job files exist then the path the process was originally tasked to perform is written to the directory as a job file and the first job in the queue is performed.
    2. If the lock fails then there is already a task running, so simply write a job file and exit.
  2. Directory is processed.
  3. Job file is deleted.
  4. Prior to termination and the relinquishing of the lock the process rescans for pending job files. If any are present it processes them in order until there are no jobs pending.

Anyway, thats probably what i would do if I wansnt going to to use a DB.

---
demerphq

  • Comment on Re: Locking/unlocking program execution using a queue

Replies are listed 'Best First'.
Re^2: Locking/unlocking program execution using a queue
by RazorbladeBidet (Friar) on Mar 30, 2005 at 15:59 UTC
    Yes, I'd considered using a database, but I wanted to avoid it if possible. Plus it's too easy :) I would imagine the I/O overhead for the files (and shell overhead for execution) would be more than the DB overhead (plus I'm already using a DB, so the connection overhead is already accounted for).

    There are certain things in memory of that particular process (command line parameters, time of execution, etc.) that require me to keep the process around (sleeping) - I could write all necessary things out, but... that sounds too complicated and less maintainable... so I'm thinking:
    1. Create a queue file w/ PID ($$.queue)
    2. Get oldest queue file (ls -rt | head -1)
    3. Check to see if that process is running (kill 0 $file)
    4. If not running, remove $file and repeat step 2
    5. If running, is $file == $$?
    6. If not, sleep for a while, then go to step 3.
    How does that sound?
    --------------
    "But what of all those sweet words you spoke in private?"
    "Oh that's just what we call pillow talk, baby, that's all."
      There is a major problem here if two processes start in the same second (that is have the same timestamp).
      -- gam3
      A picture is worth a thousand words, but takes 200K.
Re^2: Locking/unlocking program execution using a queue
by cazz (Pilgrim) on Mar 30, 2005 at 15:41 UTC
    I've done exactly that, but if you use the file's last modified time, then you don't need to worry about the filenames for the jobs as long as they are uniq.