in reply to Re^3: Read in hostfile, modify, output
in thread Read in hostfile, modify, output

To replace an important file with a new version, first get the new file onto the system in its entirety. Now a quick two step dance happens. At a minimum, rename the existing file to something else

That's not how you use rename() if you want it to be atomic, you just rename the new file over the old file. There is never a moment when the file doesn't exist. Second, you should read up on journaling filesystems, the chance for corruption is much lower and in many cases the filesystems can indeed guarantee that there will be no corruption even in the case of sudden power loss.

Replies are listed 'Best First'.
Re^5: Read in hostfile, modify, output
by Marshall (Canon) on Dec 21, 2016 at 00:41 UTC
    I am not sure what OS and file system that you are using.
    Under Windows, this cannot happen. The rename will fail if the target file exists.
    C:\Projects_Perl\testing>echo "this is orginial file" > originalfile.t +xt C:\Projects_Perl\testing>echo "this the new file" > newfile.txt C:\Projects_Perl\testing>rename newfile.txt originalfile.txt A duplicate file name exists, or the file cannot be found. C:\Projects_Perl\testing>rename originalfile.txt originalfile.bak C:\Projects_Perl\testing>rename newfile.txt originalfile.txt C:\Projects_Perl\testing>type originalfile.txt "this the new file" C:\Projects_Perl\testing>
    This thread doesn't get into journaling filesystems. The vast majority of folks here are using standard versions of Windows or Unix variants.

    I am aware of the issues you describe, but we are getting into very specialized things with that discussion. I think launching an OS specific rename command with "override" options is also beyond the scope here.

      Under Windows, this cannot happen. The rename will fail if the target file exists.
      C:\Projects_Perl\testing>rename newfile.txt originalfile.txt
      A duplicate file name exists, or the file cannot be found.
      It can happen. Use the Windows MOVE command instead of RENAME:
      MOVE /Y newfile.txt originalfile.txt
      Or use Perl:
      perl -e "rename('newfile.txt', 'originalfile.txt')"

        Thanks for the info!

        Its been more than a decade since I worked on a serious Windows installation program for a huge 24/7 application. I am sure that my recollections are fuzzy from Perl 5.6, Win NT days! I remember spending a lot of time dealing with some of the more unusual cases, like what happens if we have a new fileA, but currently running processes are linked to an old .dll version or have open the old version of fileA or are using the info from the old version even though they have closed their filehandle to the old file version. That kind of system coordination stuff can be a mess and beyond the scope in this thread.

        The general advice in this thread:

        1. Get the new copy "ready to go" on the HD.
        2. Make a backup of the original file and have a plan to revert to that if something fails.
        3. Do the file replacement (newFile replaces oldFile) in the fewest, fastest steps possible.
        Are fundamentally sound, excellent practical advice.

        From the perl 5.24.0 C sources, file win32.c:

        DllExport int win32_rename(const char *oname, const char *newname) { char szOldName[MAX_PATH+1]; BOOL bResult; DWORD dwFlags = MOVEFILE_COPY_ALLOWED; dTHX; if (stricmp(newname, oname)) dwFlags |= MOVEFILE_REPLACE_EXISTING; strcpy(szOldName, PerlDir_mapA(oname)); bResult = MoveFileExA(szOldName,PerlDir_mapA(newname), dwFlags); if (!bResult) { DWORD err = GetLastError(); switch (err) { case ERROR_BAD_NET_NAME: case ERROR_BAD_NETPATH: case ERROR_BAD_PATHNAME: case ERROR_FILE_NOT_FOUND: case ERROR_FILENAME_EXCED_RANGE: case ERROR_INVALID_DRIVE: case ERROR_NO_MORE_FILES: case ERROR_PATH_NOT_FOUND: errno = ENOENT; break; case ERROR_DISK_FULL: errno = ENOSPC; break; case ERROR_NOT_ENOUGH_QUOTA: errno = EDQUOT; break; default: errno = EACCES; break; } return -1; } return 0; }
        we see that the Perl rename function on Windows is implemented using the Win32 MoveFileEx function.

        Here is a standalone C program using the Win32 MoveFileEx function:

        // movetest.cpp // compile with: CL /W3 /MD movetest.cpp // example run: movetest fromfile tofile #include <windows.h> #include <stdio.h> int main(int argc, char* argv[]) { if (argc != 3) { fprintf(stderr, "usage: movetest file1 file2\n"); return 1; } char* from = argv[1]; char* to = argv[2]; fprintf(stderr, "move '%s' to '%s'...", from, to); if (!MoveFileEx(from, to, MOVEFILE_REPLACE_EXISTING)) { fprintf(stderr, "failed: error=%lu\n", (unsigned long)GetLastErr +or()); return 2; } fprintf(stderr, "done.\n"); return 0; }
        After compiling, you can run with:
        movetest newfile.txt originalfile.txt
        which overwrites the original file even if it already exists (if you have permission to do so).

        Well, it's pleasing to see that Perl is doing the rename via a single Win32 MoveFileEx function call in preference to a DeleteFile followed by a MoveFile (which would have no chance of being atomic). However, the jury seems to be out on how truly atomic Windows rename is. After reading this stackoverflow question I'm still confused. Alternatives to MoveFileEx appear to be ReplaceFile and MoveFileTransacted.

        Via brute force search of the perl 5.24.0 source code, I further noticed that MoveFileEx is also available to Perl code via the Win32API::File module. In the latest Perl distribution, Win32API::File::MoveFileEx is called by ExtUtils::Install.

      Atomic rename is specified by POSIX, nothing "specialized" about it. Yes, Windows is more complicated and you have a point in that case, but newer versions of Windows provide some advanced API functions.

      Most modern filesystems, like ext3, ext4, NTFS, and HFS+ support journaling, so "The vast majority of folks here" with their "standard versions of Windows or Unix variants" are already using them. Do you still think "There is no truly atomic operation on the file system"?

        "Atomic rename()" by POSIX specifies a single rename operation.

        Show some working code that demo's replacing a file using Perl that uses a single rename.

      Under Windows, this cannot happen. The rename will fail if the target file exists.

      What? Works for me (it's just not atomic, as far as I can tell).

      echo foo >foo echo bar >bar perl -e "rename(foo,bar)"