ShaZe:

It looks like your program is monitoring a file that a different program generates. I suspect that the other program doesn't flush its buffers after writing, so the data in the file is simply incomplete at the time you read it. You can demonstrate it with these two programs. The first program writes a line of text to a file every second:

$ cat file_gen.pl
#!/usr/bin/env perl
use strict;
use warnings;

open my $FH, '>', 'a_file.txt';
$|=1;
for (1 .. 1000) {
    print $FH "$_: here's 80 stars: ", "*"x80, "\n";
    my $t = time;
    print "$t: line: $_\n";
    sleep 1;
}
[download]

The second program reads the file every 5 seconds, and dumps its contents to the console:

#!/usr/bin/env perl
use strict;
use warnings;

while (1) {
    my $t = time;
    print "$t: Here's what we have so far:\n";
    open my $FH, '<', 'a_file.txt';
    my @lines = <$FH>;
    close $FH;
    print join("", @lines),"\n";
    sleep 5;
}
[download]

If you run the file generator in one terminal, and the file monitor in the other monitor, you'll see that many lines are written by the first program, while the monitor program doesn't see anything for a while. After a while, the first program will write enough data to the output buffer that the file finally gets updated, and the next time the file monitor does its thing, you'll see a large number of lines suddenly appear in the file. It's quite likely that the last line it shows you is a part of a line, though, unless you get lucky and the end of the line happens to correspond with the end of a buffer.

For details on buffering, read perldoc perlvar in the section "Variables related to filehandles" and also IO::Handle for information on file buffering as well as other operations you can do with files.

...roboticus

When your only tool is a hammer, all problems look like your thumb.

It seems you are right. The file seems to be fine on the server side however the other program used for monitoring is not seeing the updated file.

Taking this new information into account, I am assuming it would be a good approach to always force flush after writing sensitive data to make sure they are saved in case of a sudden interruption? That's assuming the buffer never fully write to the file if it is terminated earlier.

Thank you very much for the help! I guess I now have to tweak a few things now to make sure this kind of data lost never occur ever again.

1. There is a file, fully written and readable.
2. Along come the readers and life is good.
3. Jill adds something to the end of the file, but Bob only sees half.
   • Bob does not see the trailing EOL, so he is well aware the file is truncated.
4. Bob keeps reading, IO::Select until he is content or gives up timing out and handling the error. Ignoring partial data, ect.
   • In many cases Bob can just ignore the last record, it's likely he will read it later anyway.
5. Bob needs to edit the file, cleaning up after Jill or anything else that's not appending to the file.
6. Bob safely creates a new file with a similar name that's always used when updating the file.
   • sysopen O_CREAT|O_EXCL
7. Bob fills the file till it's complete, reading and copying an newly opened original.
8. Bob closes the new file, flushing it to disk.
9. Bob renames the new file ontop of the original.
10. Bob looks at the original(still open) for any additions till he is satisfied that he can quite.
    • fork and Select with time out is good for this.
11. There is a file, fully written and readable...

Note: Bob will need to set the correct group and group writable permissions as well as have write access to the folder containing the file. The temp file should be in the same folder as the original, because this does not work across mount points.

The above is completely Atomic and works for line buffered files. By Atomic we mean that a reader sees one or the other versions of the file. If it's imperative that the reader always has the freshest data, then you need to use file locks. If using file locks consider using binary data as you'll incur less buffering overhead that is not doing any good.