Re^2: Buffered IO and un-intended slurping

The missing information:
1) Linux system, \R = \n
2) Files are opened O_RDONLY, no writing
3) All IO in this module is either buffered or all 'sys'; no mixing of the two
4) I have added code to do a sysseek to 0,CUR and compared that which the computed offset. This has not yet fired, to indicate a failed seek. But I have removed all the buffered mode code.
5) The files are continuous. They were created from text databases with shell utilities
6) Possibly relevant is that this module is running in multiple processes, potentially at the same time. But simultanious reads should not be a problem....

It is always better to have seen your target for yourself, rather than depend upon someone else's description.

Comment on Re^2: Buffered IO and un-intended slurping

Replies are listed 'Best First'.
Re^3: Buffered IO and un-intended slurping by Marshall (Canon) on Jan 03, 2010 at 13:22 UTC
6) Possibly relevant is that this module is running in multiple processes, potentially at the same time. But simultaneous reads should not be a problem.... Oh, to further clarify, checking seek() return should be on the all the seeks, the "dummy seek to the same position" shouldn't fail. Of course "shouldn't" doesn't mean that it couldn't! Normally tell() should do same "under the covers" thing and yield the current byte position to check against "legal byte positions". You can put an "or die "xxxx" " on all seek(),tell()'s. I was mainly concerned that sometimes perhaps you seek past EOF or before BOF and all sorts of "bad" things could happen, some of them non-obvious. With a failure 1/1,000 operations, this is seldom enough that there isn't some super obvious thing, but it is often enough to be able to re-create the problem in some "reasonable time frame". I presume some hours or even minutes. One question I have is: Are you able to recreate the problem with only one process running? An overnight run with many thousand's queries without errors with one process might be a clue. I have certainly opened the same file from multiple process and used read sequential many times...a normal sort of thing to do. I am wondering if somehow on your system, somehow the seeks() are causing problems. I suppose that it is possible that your code is fine and the OS and its file system occasionally goofs. I have done what you are trying to do, but my thing that did the seeking, reading, writing was just a single process.	[reply]

Replies are listed 'Best First'.

Re^3: Buffered IO and un-intended slurping
by Marshall (Canon) on Jan 03, 2010 at 13:22 UTC

6) Possibly relevant is that this module is running in multiple processes, potentially at the same time. But simultaneous reads should not be a problem....

Oh, to further clarify, checking seek() return should be on the all the seeks, the "dummy seek to the same position" shouldn't fail. Of course "shouldn't" doesn't mean that it couldn't! Normally tell() should do same "under the covers" thing and yield the current byte position to check against "legal byte positions". You can put an "or die "xxxx" " on all seek(),tell()'s. I was mainly concerned that sometimes perhaps you seek past EOF or before BOF and all sorts of "bad" things could happen, some of them non-obvious.

With a failure 1/1,000 operations, this is seldom enough that there isn't some super obvious thing, but it is often enough to be able to re-create the problem in some "reasonable time frame". I presume some hours or even minutes.

One question I have is: Are you able to recreate the problem with only one process running? An overnight run with many thousand's queries without errors with one process might be a clue.

I have certainly opened the same file from multiple process and used read sequential many times...a normal sort of thing to do. I am wondering if somehow on your system, somehow the seeks() are causing problems. I suppose that it is possible that your code is fine and the OS and its file system occasionally goofs.

I have done what you are trying to do, but my thing that did the seeking, reading, writing was just a single process.

[reply]