Re^3: Strange IO + concurrency issue

in reply to Re^2: Strange IO + concurrency issue
in thread [SOLVED] Strange IO + concurrency issue

Notice "if ($f)"

I suspect this test does not do what you think it does. In particular, consider the case that one process opens the file and writes to it, then another process opens the same file and, perhaps, writes to it, then the if ($f) test executes in the first process. Is the result any different because of what the other process did? I think you will find it is not and, therefore, that the statements in the if block are not as protected as you think they are.

Whether your problem is solved depends on what you are trying to do, which you don't say, but what you are calling a solution seems strange to me. I suggest you reconsider.

Running your program on Windows several times, each trial I get a variable subset of the 25 potential output files with names beginning with 'c' and several of them have less than 40_000_000 characters in them.

1. file size of some c_* files is less than 40000000, however this should not happen, because of line: die if -s $filename != -s $newfilename;

The die will not prevent the production of files with less than 40_000_000 characters because the test is performed after the copy, not before. Even if you moved the test before the copy, because these statements are not protected by the lock file, the test could pass but another process could modify the source file before the copy executes, resulting in a file of different length (and, perhaps, content if in your real program the different processes write different content).

Changing the test on the die to 40_000_000 != -s $newfilename makes little difference to the outcome: there is still a variable subset of the possible 25 copied files and there are still several of them with fewer than 40_000_000 characters. Of course, it makes a difference as, when the test runs, the size of $filename might be other than 40_000_000. This will be somewhat random, depending on how execution of the various processes is interleaved. But, perhaps this is exactly as you intend and I am worrying for nothing.

Comment on Re^3: Strange IO + concurrency issue Select or Download Code

Replies are listed 'Best First'.
Re^4: Strange IO + concurrency issue by vsespb (Chaplain) on Sep 30, 2013 at 08:28 UTC
I suspect this test does not do what you think it does. In particular, consider the case that one process opens the file and writes to it, then another process opens the same file and, perhaps, writes to it, then the if ($f) test executes in the first process. Is the result any different because of what the other process did? I think you will find it is not and, therefore, that the statements in the if block are not as protected as you think they are. Notice also "my $f = undef" and "unless (-e $filename)" `my $f=undef; getlock sub { unless (-e $filename) { open ($f, ">", $filename) or confess; binmode $f; } }; if ($f) {` [download] ( I admit that this code is unclear ) Whether your problem is solved depends on what you are trying to do, which you don't say, but what you are calling a solution seems strange to me. That was proof of concept code, it contained a bug. Bug found, thus solved. what you are trying to do, which you don't say This was proof-of-concept code. I.e. I simplified my 1000 lines program to this code. Thus what I am trying to do is behind the scene, and we should focus only on technical part of problem, not business requirements. If I post my original code, this would require at least couple of hours to set up things and reproduce the problem for anyone who tries to. the test could pass but another process could modify the source file before the copy executes Yes, but btw there are no concurrent writes to same file. Changing the test on the die to 40_000_000 != -s $newfilename makes little difference to the outcome: there is still a variable subset of the possible 25 copied files and there are still several of them with fewer than 40_000_000 characters. Well, it indeed does not fix program to output correct files, but it causes some processes to DIE, thus eliminating a bug. See assertions. Actual fix is to extend lock to whole program, until copy is done (as other posters suggested). Or even just extend it to the point when write to source file is done and file is closed.	[reply] [d/l]

Replies are listed 'Best First'.

Re^4: Strange IO + concurrency issue
by vsespb (Chaplain) on Sep 30, 2013 at 08:28 UTC

I suspect this test does not do what you think it does. In particular, consider the case that one process opens the file and writes to it, then another process opens the same file and, perhaps, writes to it, then the if ($f) test executes in the first process. Is the result any different because of what the other process did? I think you will find it is not and, therefore, that the statements in the if block are not as protected as you think they are.

my $f=undef;
  getlock sub {
   unless (-e $filename) {
    open ($f, ">", $filename) or confess;
    binmode $f;
   }
  };
  if ($f) {
[download]

Whether your problem is solved depends on what you are trying to do, which you don't say, but what you are calling a solution seems strange to me.

what you are trying to do, which you don't say

the test could pass but another process could modify the source file before the copy executes

Changing the test on the die to 40_000_000 != -s $newfilename makes little difference to the outcome: there is still a variable subset of the possible 25 copied files and there are still several of them with fewer than 40_000_000 characters.

assertions

[reply]
[d/l]

In Section Seekers of Perl Wisdom