The manager approach is a viable solution, but I believe it has the additional hassles of:
• writing a separate manager module
• starting and stopping the separate manager module when the program runs/stops
• detecting failures: as the manager is one more component in your system, what happens if it goes wrong? what happens if it segfaults? what happens if it does not work properly?

Sure, it adds complexity with all that that implies. Sometimes, if the application is already large enough, there's something that is already a manager (who started the separate worker processes, for example?). In that case, the marginal costs of moving the responsibility of queueing coordination to the manager are not as high as writing it from zero. As always, YMMV.

Open and lock a file (should be different from the file to be processed, so you can open and close that file multiple times if necessary without ending the lock)
Open file to be processed and do whatever
Unlock the lock file

I wrote myself small subs for locking and unlocking, and I just cut and paste them in wherever I need them. Saves time coding.

Rather than cutting and pasting code, I strongly recommend moving it into a module which is then used by all your scripts. This means that if you find a bug, you can fix it for all scripts in one place rather than have to edit them all (and I bet you'd miss one here or there). It also means that if for any reason you need to rewrite it because you've moved your code to a different platform or different filesystem where locking semantics are different, you again have to only edit it in one place.

1. Process A opens lock.txt
2. Process B opens lock.txt
3. Process A flocks()
4. Process B tries to flock().... and fails

If you want to be more safe, you should ask yourself: what happens if now_access_the_common_resource() crashes or loops? will the file stay locked forever? how do you detect that the other processes are starving?

My suggestion is: if your program already uses a RDBMS, why not using its own locking features? each and every decent RDBMS already has consistent locking features implemented to make its own basic work.

You could - for example - have a one-row table with a status field that each process tries to update like UPDATE mytable SET status=1 WHERE status=0 LIMIT 1. After each update you check the number of affected rows; it it's 1 you got access to the resource (and atomically locked it), if no row was affected you could not obtain the access, so you sleep() random seconds and try again. To release the resource you simply UPDATE mytable SET status = 0 WHERE table_id = 123.

The nice part of this approach is that it works not only for one resource but for limited sets like resource pools too: just add ten lines to the table, and the first ten processes will be allowed to pass, the eleventh will have to wait and so on.

Your program will likely work in 99% of the cases (maybe 99.9999%), but as the opening and locking are two separate steps, it might happen that:

1. Process A opens lock.txt
2. Process B opens lock.txt
3. Process A flocks()
4. Process B tries to flock().... and fails

So you should really encapsulate your code above in a polling loop that detects locking failures and retries every n seconds on failure.

Eh... what?!?!? There are several things wrong with what you say here.
1. Opening a file will always work whether it is locked or not. Actually, the fact whether it's may be locked is irrelevant.
2. Nobody specified use of LOCK_NB — instead he's using LOCK_EX (2), so flock will simply wait if the file is locked, until it can get a lock. No looping necessary. flock will only return after a failure if something is seriously wrong. Your program shouldn't loop then, but die.
There simply is no race condition.

If you want to be more safe, you should ask yourself: what happens if now_access_the_common_resource() crashes or loops? will the file stay locked forever? how do you detect that the other processes are starving?

Hold that thought. Well, one surely nice thing by OS-native file locking, is that a flock is automatically released if a process ends without releasing the lock itself, or closing the file. That includes crashes and being killed by other processes. There simply is no problem.

A problem could indeed be looping forever without releasing the lock... so don't do that.

My suggestion is: if your program already uses a RDBMS, why not using its own locking features? each and every decent RDBMS already has consistent locking features implemented to make its own basic work.

You could - for example - have a one-row table with a status field that each process tries to update like UPDATE mytable SET status=1 WHERE status=0 LIMIT 1. After each update you check the number of affected rows; it it's 1 you got access to the resource (and atomically locked it), if no row was affected you could not obtain the access, so you sleep() random seconds and try again. To release the resource you simply UPDATE mytable SET status = 0 WHERE table_id = 123.

That approach is simply too attrocious for words. Your solution clearly suffers of the problem you just warned against: what if the program that holds the lock, dies? The status field of that database row will simply never get cleared.

And you're actually not using "the locking features" of the database, you're using the fact that the update of the database is atomic. I'm not even 100% convinced that it is. You are using the return value of the count of affected rows for a purpose that it was never intended for.

There even are ways to try create a file if it doesn't exist, atomically. Look up O_EXCL in perlopentut. That would seem to be a far simpler, and more reliable way to do it. And even that seems to me like you're working too hard, because you still have to guard against premature death of your process, and take measures that the file will indeed get deleted in due time.

Just stick with locking.

Well, one surely nice thing by OS-native file locking, is that a flock is automatically released if a process ends without releasing the lock itself, or closing the file. That includes crashes and being killed by other processes. There simply is no problem.

I quite like the idea of all subsequent access being blocked if there was a failure. It brings the failure to the admin's attention so he can investigate why it failed in the first place.

I frankly don't know much about unix locking, and you are right, but have written concurrent production code in major apps for major players. What I was saying is that you don't need table locking on the database (that has its own uses), you just want to have a mutex access to a pool of resources, and the database does that excellently. And if you add a datetime field that you happen to update with the update, you can know who locked what and easily inspect it externally. And this is definitely a plus when running concurrent stuff, where bugs are rather hard to detect.

You could - for example - have a one-row table with a status field that each process tries to update like UPDATE mytable SET status=1 WHERE status=0 LIMIT 1. After each update you check the number of affected rows; it it's 1 you got access to the resource (and atomically locked it), if no row was affected you could not obtain the access, so you sleep() random seconds and try again. To release the resource you simply UPDATE mytable SET status = 0 WHERE table_id = 123

bart's criticisms of this approach are well taken, however, there are ways to take advantage of database locking to solve your problem. In Oracle, for example, you could have the same table described, but do a SELECT FOR UPDATE mytable or even just LOCK mytable. Then you would do your work on your resource. When you are finished, do a ROLLBACK or an UNLOCK mytable. Most other RDBMS's will have similar, but not necessarily identical language to effect the same behavior. This has the advantage that if your code fails unexpectedly, the database will remove the lock for you. You have to be careful if the resource (or one of the resources) you access is the database itself; in that case doing the rollback will not have the results you want ;)

All that said, the file locking is still the simpler and more straightforward solution in most (99.9999%) cases.

--DrWhy

"If God had meant for us to think for ourselves he would have given us brains. Oh, wait..."

If you want a lock in oracle, you're going about it the entirely wrong (though viable of course) way. Use dbms_lock.request example

   v_lock_sucess:=dbms_lock.request(lockhandle=>v_temp_lockhandle,
                                     lockmode=>dbms_lock.x_mode,
                                     TIMEOUT=>5,
                                     release_on_commit=>TRUE);
[download]

If you want to be more safe, you should ask yourself: what happens if now_access_the_common_resource() crashes or loops? will the file stay locked forever? how do you detect that the other processes are starving?

You seem to have a poor understanding of how flock (and database locking) works. In addition to flock blocking until the lock succeeds (as bart explained), crashing is not a problem. When the application crashes, the filehandle will get reclaimed by the OS. The lock gets released along with the filehandle.

You're right that an application can hold the lock indefinitely. For example, the application could enter an infinite loop, or do a blocking call. Switching to a databse (especially in the manner you describe) will not magically solve these problems. Welcome to the very complex world of race conditions and fault tolerant computing.

use IPC::SysV qw(IPC_PRIVATE S_IRWXU IPC_CREAT);
use IPC::Semaphore;

$sem = new IPC::Semaphore(IPC_PRIVATE, 10, S_IRWXU | IPC_CREAT);
# and the crucial part:

$sem->op($semaphoreNumber,0,0);
# where first zero means wait-for-zero operation,
# ie, your code will stop and wait (CPU-friendly) 
# until your semaphore becomes 0.
#etc...
[download]

Why hasn't anyone suggested this yet?

This is a SolvedProblem(TM) in computer sciences, ...or maybe I'm missing something?

OTOH, I find spool-model generally more usefull, ie - you've got dedicated process for processing incoming jobs, and your workers just drop their jobs into a spool, be it directory, sql table or shared memory segment (I would recommend IPC::ShareLite).

This is the way mail servers, fax srvrs and many others work...

Why is everyone soo keen on using files when there are superior methods around? When it goes to synchronising jobs between multiple machines files are not that great either, you get multiple problems with most remote filesystems... on the other hand there are remote IPC solutions available and they are created and tested specifically to work with such scenarios.

I don't know if this is the case, but I usually have such feelings when interfacing with php/mysql people... they have no CS background or are unable to use it...ask the simplest questions about the basic things... and then proceed to solve them with strangest hoops... and THEN they warp the reality with their weight and make young programmers think that this is the right solution.

To address your question, file locking is preferred because flock() semantics have been more portable than SysV IPC, historically, among unix-like systems. Also the system semaphore count, in some OS flavors, was set with a kernel configuration parameter, and required admin management. Filehandles were relatively cheap, in terms of allocatable resources, and can be got without bothering the SA. However, when dealing with reasonably up to date and homogeneous systems, your answer is of course prefereable.

open(FH, ">>lock.txt") or die "can't open lock.txt: $!\n"; 
flock(FH, 2) or die "Can't flock lock.txt: $!\n"; 
now_access_the_common_resource(); 
flock(FH, LOCK_UN);
close(FH);
[download]

although, it may actually be better to not explicitly unlock in this case. For example, if someone else is waiting for the lock on FH, they might conceivably get the lock after the LOCK_UN step, but before the close(FH) AND therefore before the first process has flushed it's write operations to disk. alternatives are using autoflushing ($| = 1) or using the FileHandle->autoflush() method.

There was a really good series of articles by Sean Burke in The Perl Journal a few years back about file locking, and as soon as I read it, I put some of his code into a module for creating a "semaphore file" -- an empty file whose sole purpose is to establish and hold a lock while the important "one-at-a-time" business goes on elsewhere. I've been using it for years without a hitch, on multiple OS's.

I even posted it here, in response to another SoPW thread. I just checked the url for the Burke article it was taken from, and that's still working also.