Re^5: using lexically scoped variable as a filehandle

Well yes, but think about it. For the close to fail, rather than a preceding write, then you would have to be wanting to write just less than one buffer-full (512b/4kb/whatever), more than the filesystem could accomodate.

You program would have to write enough data so that it filled the last block on disk, and then just a little bit more before deciding to stop writing and close the file. A few bytes less and no error occurs because the file is successfully written. A few bytes more and you (should) never make it to the close because the write/print will have failed. That's a pretty small window of opportunity. And that's the simple case.

Most filesystems do caching outside of the auspices of the C-runtime cache, which means that you'd likely successfully close the file, flushing the last few bytes to OS cache, and the failure wouldn't occur until sometime after, possibly long after your process has terminated. In the meantime, some other process may delete or truncate a file, or the OS terminates a swapped out processes and frees a lump of disk space. A million things.

The mounted filesystem case is somewhat different, but again, the odds that the filesystem would go away at exactly the moment between your having successfully written the last buffer load (to cache), and deciding to close that file when the flushing of that last buffer would fail, are really very slim.

The really interesting question is what do you do about it when you detect this situation? At that point, unless you are lucky enough to be running on a system with multiple disks, even trying to log the failure is likely to encounter the same full disk scenario. You could also think about re-tries in the hope that some other process might have free'd some space up. Or you could trying deleting temp files etc. But in the end, if the scenario happens, there is unlikely to be a good recovery strategy.

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

"Too many [] have been sedated by an oppressive environment of political correctness and risk aversion."

Comment on Re^5: using lexically scoped variable as a filehandle

Replies are listed 'Best First'.
Re^6: using lexically scoped variable as a filehandle by Tanktalus (Canon) on Feb 21, 2007 at 16:56 UTC
I don't know about you, but very few of the systems I work with have asynchronous I/O enabled (and I'm not sure perl can use it even if it is - it's often a separate API). The OS cache should be acutely aware of the filesystem's fullness, even if it doesn't write immediately (which, during a flush, I think it does, but that's not material to my point). It should be able to return any errors immediately, as this cache is being shared among all processes working with that filesystem. Since the C library is flushing during a close, it will get that error immediately as well. I'm not disagreeing with you - I think the above actually accentuates your point, not detracts from it. I'm saying that it's actually rarer than you said that this problem would come up.	[reply]
Re^7: using lexically scoped variable as a filehandle by BrowserUk (Patriarch) on Feb 21, 2007 at 17:43 UTC
Actually, this has nothing to do with Asyncronous I/O, at least as I understand that term. In my world, AsyncIO relates to whether I/O to files block the calling user mode thread until the kernel mode process completes, or not. For a better description see Synchronous & Asynchronous IO (MSDN). What I was referring to is termed File caching (MSDN), and is the default behaviour on Win32 systems. I seem to recall HP/UX and AIx systems did something similar also, but I could be wrong on that. This behaviour is external to processes and can only be defeated by explicit, low level requests to the OS at the time of opening the file. ( See the discussion associated with `FILE_FLAG_NO_BUFFERING` and `FILE_FLAG_WRITE_THROUGH` at CreateFile (MSDN). Even then it imposes special conditions. Basically, most file IO on a win32 system, and certainly any done from within Perl without recourse to Win32::API/Win32API::File, goes through the system cache. This allows the OS to update the disk out of sequence to the application writes, and to read ahead of the application reads, thereby making IO more efficient by requiring less seeking. It is certainly possible for an application to re-write/overwrite an existing file block and for the process to terminate (marginally) before the dirty system cache block is flushed to disk. Whether this is also possible for data being appended to the end of an existing file I am unsure. The difference being that in the former case the cached data already has disk space reserved to back it. In the latter this is probably not the case. I guess the upshot is that for critical applications, is is necessary to check for and handle possible failures in all system calls. For non-critical applications, the need to check for the rare case of close failing is far less than the need to check for open failures and write failures. Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice. "Too many [] have been sedated by an oppressive environment of political correctness and risk aversion."	[reply] [d/l] [select]