Re^2: read file line by line then do something and write to same file

I wouldn't call that the general solution, as one of its most likely failure modes would result in data loss (imagine loss of power before the new file is written, or an inability to write the new file).

A more common pattern is the one that Perl's -i switch accomplishes for one-liners: Rename the input file, open the output file in the name of the original input file, open the renamed input file, iterate over the input, write to the output, close the output, and as a final step, unlink the renamed input.

With that pattern rename is atomic, and unlink happens only after the new output file is successfully closed.

Dave

Comment on Re^2: read file line by line then do something and write to same file

Replies are listed 'Best First'.
Re^3: read file line by line then do something and write to same file by wjw (Priest) on Jun 15, 2014 at 23:59 UTC
Good point. I did not do a very good job of articulating that. Thanks for the correction... I appreciate it. ...the majority is always wrong, and always the last to know about it... Insanity: Doing the same thing over and over again and expecting different results... A solution is nothing more than a clearly stated problem...otherwise, the problem is not a problem, it is a facct wjw	[reply]
Re^3: read file line by line then do something and write to same file by oiskuu (Hermit) on Jun 16, 2014 at 19:36 UTC
A cursory `strace perl -pi` reveals a more sloppy pattern than described, however. I see open/unlink/open/read+write/close/close. The unlinked source file is still readable; it gets released upon close. A relatively safe update sequence is: mkstemp/read+write/fsync/rename. The final rename is atomic, but that itself does not prevent races with multiple updaters (of course).	[reply]
Re^4: read file line by line then do something and write to same file by davido (Cardinal) on Jun 16, 2014 at 19:44 UTC
Yes, I believe you're correct regarding the `strace`. I recall now that the documentation is not 100% truthful as to what is happening behind the scenes. If you use `-i.bak` instead of `-i`, you'll probably see an open/rename/open/read+write/close/close instead of open/unlink... Dave	[reply] [d/l] [select]