As I know, Perl's flock should prevent other scripts to write into a file if it is already opened by another script.

This is not true.
File Locking is a cooperative mechanism. If programs want to "share a file", they have to cooperate amongst each other in terms of reading and writing. That basically means that they have to be "polite" with each other.

Oh, I want my chance to speak! You can "get in line" for your chance, but you have to be aware that somebody else is currently "speaking". You have to wait your turn and know that you have to do that.

File "locking" is a way for programs to cooperate if they choose to do so. It does not prevent some "rogue" program from causing a lot of trouble.

I don't think that Version#1 works because I don't see an UNLOCK operation.

Update: I had to re-boot my Windows PC due to incessant messages... continuing...

I would suggest that you review flock in the Perl Docs. Basically what this suggests is that you acquire a "write lock", an "exclusive lock" to the file. Do what you want as quickly as possible, position the file pointer to the end of the file (if you've been doing some seeking around), release the lock. If everybody (all programs) using that file does this, then everybody will be happy even if they share the same file handle.

If whoever is writing to this file is "not cooperating" with file locks then there are going to be problems. A write or "print" operation is not "atomic" in a general sense. Version #1 can get a malformed line.. could be a partial line at the end of the file due to the other writer not "cooperating".

In Version #2, this will save memory (no quickie memory copy) at the expense of perhaps slowing down the "writers" to the file. But be aware that no matter what you do, if the programs are not cooperating, there is the possibility of malformed lines at the end of the file.

I guess I should point out that a "file lock" is a very perfomant operation because the OS keeps this status as a memory resident structure. Acquiring and releasing a file lock on a zero byte file is one way to coordinate some kinds of intra-process communications.

Another Update: See: temp files.
my $tempfile = "./tempfile".(rand()*9999);
is not a good way for this. Ask the OS for a guaranteed unique temp file name. When you are finished with it, I would clean it up (delete it), however the OS will view this file as "trash" and delete it whenever the next "clean up" operation is run on the system - but don't leave trash laying around if you can help it - drinking beer is ok, but don't leave the cans in park.

In reply to Re: Trying to optimize reading/writing of large text files. by Marshall
in thread Trying to optimize reading/writing of large text files. by nikkimouse

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.