[14166] error @0.102147> [hit #336]request to GET / crashed: YAML Erro
+r: Invalid element in map
   Code: YAML_LOAD_ERR_BAD_MAP_ELEMENT
   Line: 315
   Document: 1
 at /opt/perl5.14/lib/site_perl/5.14.2/YAML/Loader.pm line 352. in /op
+t/perl5.14/lib/site_perl/5.14.2/Dancer/Handler.pm l. 98
[download]

The application uses Dancer::Session::YAML to create and store persistent sessions. I easily identified the problematic session file (I add access time to a session) and found the problem at the end of the file:

...
      question:
        - "You can select more than one answer.\n"
e answer.\n"
[download]

Ouch! Seems like the file has been overwritten without being truncated first. The application runs on Starman which forks workers to serve the requests. It seemed to me as if two instances had been trying to write the session file - a race condition. After discussing the issue with a colleague, I inspected the source code of the Dancer::Session::YAML module. The last subroutine was the only one printing anything:

sub flush {
    my $self         = shift;
    my $session_file = yaml_file( $self->id );

    open my $fh, '>', $session_file or die "Can't open '$session_file'
+: $!\n";
    flock $fh, LOCK_EX or die "Can't lock file '$session_file': $!\n";
    set_file_mode($fh);
    print {$fh} YAML::Dump($self);
    close $fh or die "Can't close '$session_file': $!\n";

    return $self;
}
[download]

The problem is the open line: it might overwrite (clobber) the file even before the lock is granted. To demonstrate the problem, I wrote the following script:

#!/usr/bin/perl
use warnings;
use strict;

use Fcntl qw(:flock :DEFAULT);

my $pid = fork;
die "Cannot fork.\n" unless defined $pid;

if ($pid) {
    open my $parent, '>', 'tmp'       or die "open parent: $!";
    sleep 1;
    flock   $parent, LOCK_EX          or die "lock parent: $!";
    print  {$parent} "Parent\n";
    close   $parent                   or die "close parent: $!";

} else {
    open my $child, '>', 'tmp'        or die "open child: $!";
    flock   $child, LOCK_EX           or die "lock child: $!";

    # Longer than parent:
    print  {$child} "Child here...\n";
    close   $child                    or die "close child: $!";
}
[download]

The parent opens the file, but while it sleeps, the child writes something to it. The parent then wakes up, gets the lock and writes to the file - but it just writes over whatever the child has written. If the child's output is longer than the parent's, the invalid session problem turns up.

I have just read the chapter on file-locking in Programming Perl (the 4^th edition). To obtain the exclusive lock, the book recommends using sysopen, but after some googling and experimenting I found a simpler solution (only works if the file already exists, though, otherwise sysopen is inevitable — or maybe +>> can help?):

open my $fh, '+<', $file or die "Can't open '$file': $!\n";
flock $fh, LOCK_EX or die "Can't lock file '$file': $!\n";
truncate $fh, 0;
print {$fh} $content;
close $fh or die "Can't close '$file': $!\n";
[download]

The +< mode does not overwrite the file. truncate deletes the previous content of the file (or shortens the file), so the previous content does not matter.

Proud of myself, I opened the Dancer's bug tracker... only to find the problem has already been fixed on github in a completely different way: the session is written through the Dancer::FileUtil module, using its atomic_write:

sub atomic_write {
    my ($path, $file, $data) = @_;
    my ($fh, $filename) = tempfile("tmpXXXXXXXXX", DIR => $path);
    set_file_mode($fh);
    print $fh $data;
    close $fh or die "Can't close '$file': $!\n";
    rename($filename, $file) or die "Can't move '$filename' to '$file'
+";
}
[download]

There is more than one way to... you know what. Anyway, I learned a lot.
Update: The note on file existence with +<.