Atomic update counter in a file

pbijnens has asked for the wisdom of the Perl Monks concerning the following question:

I use this small one liner to update a counter in a file:

perl -pi -le '++$_' .counter
NR=$(cat .counter)

Many instances can update the counter and it must be really an atomic update.

First question: is the -i option doing some exclusive locking on the file? I don't think so, because from the description I read that perl actually opens the existing file, unlinks, and writes a new file. So there is a tiny window that the file ".counter" does not exist.

Getting the number from the file in the bash also happens after perl is done, so there is another race condiditon, which I initially tried to solve as: (ugh)

NR=$(perl -pi -le '++$_; print STDERR' .counter 2>&1)

I also have the problem with the first creation of the file. I use the magic increment on strings to generate numbers of fixed with, so the initial value must be "000". Testing for existance, creating the file and then using it, is another race condition, so I wrap the whole block in the bash script with:

# ... longer bash script here, and then coming to:
if lockfile -1 -r1 .counter.lock
then
    trap 'rm -f .counter.lock' 0 1 2 3 13 15
    test -s .counter || echo 000 >.counter
    perl -pi -le '++$_' .counter
else
    echo 'error: could not lock .counter" >&2
    exit 2
fi
NR=$(cat .counter)
#... continuing bash script

Becoming complicated (and "lockfile" is from the "procmail" package, not really standard). So I was rewriting the whole bash thing into perl. Due to other features that I need in the rest of the program, I use Path::Tiny, and then it comes to this:

use Path::Tiny;
# ... some perl script up to:
my $FH;
my $nr;
if (-e ".counter") {
    $FH = path(".counter")->openrw_raw({locked=>1}) or die("open .counter: $!\n")
    { local $/; $nr = <$FH>; }
    die(".counter contains garbage\n") unless $nr =~/^\d+\n$/;
    chomp $nr;
    truncate($FH,0) # not really needed
    seek($FH,0,0);
} else {
    $FH = path(".counter)->openw_raw({locked=>1}) or die("open .counter: $!\n");
    $nr = "000";
}
$nr++;
print $FH $nr, "\n";
close($FH) or die("close .counter: $!\n");
# ... and it continues using the value of $nr

This one still has a race condition between the test for existance and the creation of the new file.

Path::Tiny does open with lock exclusive, but then the perl "openw" would fail, while the "lockfile" bash construct would at least wait up to 1 second and retries (if needed more retries than I specified now in the bash version).

Any idea how to solve that? Why is this becoming so complicated? I have a feeling that this could be much more elegant.

Comment on Atomic update counter in a file

Replies are listed 'Best First'.
Re: Atomic update counter in a file by RichardK (Parson) on Nov 18, 2015 at 15:41 UTC
You didn't say which operating system you're on, and the implementation details differ. But in general file systems just aren't atomic so these things are difficult, and they never work on network file systems. However, you might find it easier to split this into 2 files, one just for the locking that you never need to remove so there's no race, and the other one for the data. `# pseudo code :) lock(.counter.lock); update(.counter); unlock(.counter.lock); # of course you need to read the counter under the lock too lock(.counter.lock); read(.counter); unlock(.counter.lock);` [download] As long as everyone who accesses the counter uses the lock you should be ok	[reply] [d/l]
Re: Atomic update counter in a file by Anonymous Monk on Nov 18, 2015 at 09:51 UTC
This one still has a race condition between the test for existance and the creation of the new file. Which is why typically you never perform that test -- its redundant and introduces a race condition	[reply]
Re^2: Atomic update counter in a file by Anonymous Monk on Nov 18, 2015 at 10:37 UTC
Then what do you suggest to atomically differentiate between creation of the file and update of the file?	[reply]
Re^3: Atomic update counter in a file by pbijnens (Novice) on Nov 18, 2015 at 11:43 UTC
Answering my own question... This seems to do the trick: use Path::Tiny; #... some perl stuff my $FH = path(".counter")->filehandle({locked=>1, exclusive=>1}, "+>>", ":raw") or die("open .counter: $!\n"); seek($FH,0,0); my $nr; { local $/; $nr = <$FH> }; $nr \|\|= "000\n"; die("garbage in .counter\n") unless $nr =~ /^\d+\n$/; chomp($nr); ++$nr; truncate($FH,0); seek($FH,0,0); print $FH $nr, "\n"; close($FH) or die("close .counter: $!\n"); # and continuing using $nr The mode "+>>" creates the file when it does not exist yet, and allows for read and write. The "locked" and "exclusive" mode make sure we're the only one using it.	[reply]