in reply to Order of flock and open

What I have done in similar cases is lock a different file. E.g. before opening "ImportantData.txt", my programs first open and lock "ImportantData.lock" and hold it until they are really done with the file "ImportantData.txt".

Updated:Checking back, I see a lot of people tell you "no problem", "don't worry", etc. Wrong! You are right to be concerned. They do not understand the issues.

Example: have two processes which are each going to add 1 to the value in the file.

Process 1 opens file IN, "<in.txt"
Process 1 locks file IN
Process 1 opens file OUT, ">out.txt";
Process 1 locks file OUT
Process 2 opens file IN, "<in.txt";
Process 2 tries to lock IN and waits
  (The vulnerable window is greatly reduced if you use LOCK_NB, but no
+t
  eliminated.  But then you have to close and reopen IN in a busy loop
+).
Process 1 reads the value "100" from IN
Process 1 writes the value "101" to OUT

In any order (or atomically):
  Process 1 closes IN (thereby unlocking it)
  Process 1 closes OUT (thereby unlocking it)
  Process 1 rename "out.txt" to "in.txt"

Process 2 acquires lock on file IN
Process 2 opens file OUT, ">out.txt";
Process 2 locks file OUT
Process 2 reads the value "100" from IN
Process 2 writes the value "101" to OUT

In any order (or atomically):
  Process 2 closes IN (thereby unlocking it)
  Process 2 closes OUT (thereby unlocking it)
  Process 2 rename "out.txt" to "in.txt"

[download]
Result: file "out.txt" has the value "101" instead of the correct "102"

If you use use the default blocking lock, you have a huge vulnerability. You not only have to worry about another process opening the file between your open and lock, you have to worry about any process opening it from the time you lock it until the rename is completed!

Replies are listed 'Best First'.
Re: Re: Order of flock and open
by bluto (Curate) on Apr 30, 2003 at 19:41 UTC
    You are absolutely right -- you must watch out in this case. Most of the other replies don't bring up the fact that you must lock the entire process (read/write/rename). If you do this with a separate file, it is rather straightforard.

    UPDATE: merlyn is right that the "code" below is fishy and should not be used. Take it as an example of what can go wrong if you try to get too tricky with locking. There are almost always conditions you will overlook (as I have.)

    That said, I believe you can do this by locking both files, but you must pay attention to the order you perform the operations. You should be able to do the following in order... 

    open and lock the input file
open lock the output file
read the input file 
write the output file
rename the input file
rename the output file
close both files (which unlocks them)
delete old renamed input file

    [download]

    I'm not sure this works on all operating systems, but should work for Unix/Linux. You must use a non-blocking lock for the input file and you must close/reopen it in a loop if it is already locked. This is because someone can completely replace the input file before you get the lock for it and then your open file descriptor is pointing to the old stale (possibly deleted) input file. You should probably also use unique names (i.e. that don't collide with other processes) for the output file and the old input file.

    This all seems rather complicated, which is why locking a separate file is much simpler. But, YMMV...

    bluto

[reply]
[d/l]
[reply]
        I agree that it is fishy and I would never use it in practice (the reason for the lack of real code and the disclaimer in the last line of the post) since over the years I've learned that once you start messing around with operation ordering too much, and start requiring non-blocking locks, chances are good the locking strategy is incorrect.

        That said, I'm not entirely convinced that it can't be fixed, perhaps by using only a single rename, only that it isn't worth it. Your solution is more elegant in any case.

        bluto

[reply]
Re: Re: Order of flock and open
by Aragorn (Curate) on Apr 30, 2003 at 19:29 UTC
    This technique of locking a separate file before opening the real data file is called locking with semaphore files. Thelonius++

    Arjen

[reply]

In Section Seekers of Perl Wisdom