How I'm sure syswrite has finished?

casiano has asked for the wisdom of the Perl Monks concerning the following question:

Dear Monks,

I have a race-condition problem.

The problem arises because a child process makes a call to

syswrite handler, message
[download]

and finishes. The message is huge and is received by another process running on a remote machine (via SSH). I thought closing the handler would be enough to sync but it does not seem to work. The fragment of code is as follows (I'm using Parallel::ForkManager but the problem arises also with plain fork)

# Parallelise the writing into the pipes
my $pm = Parallel::ForkManager->new($lp);

foreach (@str_handles) {
  my $pid = $pm->start and next;

  syswrite $_->{handle},
           join(" ",
                  $_->{chunksize},
                  $_->{A_cols},
                  @A_lines[$_->{start}.. $_->{end}],
                  @B_lines,
                  "\cN"
               );
  $pm->finish;
}
[download]

The problem disappears if I insert a sleep(1) after the syswrite call because the sleep gives time for the write to complete. Is there a way to know when the call to syswrite has fully finished?

Thanks for your help

Casiano

Comment on How I'm sure syswrite has finished? Select or Download Code

Replies are listed 'Best First'.
Re: How I'm sure syswrite has finished? by Perlbotics (Archbishop) on Mar 14, 2009 at 12:13 UTC
Hi. An explicite close might help... `syswrite ... ; close $_->{handle} or die "cannot close file - $!"; # added $pm->finish;` [download]	[reply] [d/l]
Re^2: How I'm sure syswrite has finished? by casiano (Pilgrim) on Mar 14, 2009 at 12:52 UTC
No, I already tried and gives the same error.	[reply]
Re^3: How I'm sure syswrite has finished? by oko1 (Deacon) on Mar 14, 2009 at 13:37 UTC
What is the "same" error? It's kind of hard to troubleshoot un(der)specified problems. -- "Language shapes the way we think, and determines what we can think about." -- B. L. Whorf	[reply]
Re^4: How I'm sure syswrite has finished? by casiano (Pilgrim) on Mar 14, 2009 at 21:50 UTC
Re: How I'm sure syswrite has finished? by Joost (Canon) on Mar 14, 2009 at 13:55 UTC
Syswrite is done when it returns. Unless you pull the power or the drive out of the machine you don't need to know when the data is physically on the magnetic platter. If you do have problems please be more specific. "What should it profit a man, if he should win a flame war, yet lose his cool?"	[reply]
Re^2: How I'm sure syswrite has finished? by casiano (Pilgrim) on Mar 14, 2009 at 21:52 UTC
The problem is that at the other side there is a process	[reply]
Re: How I'm sure syswrite has finished? by perl5ever (Pilgrim) on Mar 14, 2009 at 18:47 UTC
I agree with ChemBox that the parent should wait for the children to finish. Have a look at the `wait_all_children` method. Your problem is likely happening after the `for` loop. Note that the parent process will get done with the `for` loop much earlier than the child processes. One way the messages sent by the children may get truncated is if the parent is doing something after the `for` loop to cause the `ssh` processes to terminate.	[reply] [d/l] [select]
Re: How I'm sure syswrite has finished? by ikegami (Patriarch) on Mar 14, 2009 at 19:54 UTC
[ This was meant as a reply to Re: How I'm sure syswrite has finished? ] if the parent is doing something after the `for` loop to cause the `ssh` processes to terminate. I think it's simpler than that. When the parent exits, doesn't that kill the children? Calling `wait_all_children` is still the solution as that would make sure the children are done.	[reply] [d/l] [select]
Re^2: How I'm sure syswrite has finished? by casiano (Pilgrim) on Mar 14, 2009 at 22:18 UTC
Many thanks ikegami, I tried `wait_all_children` (see answer below) but without success. Somehow the `$pm->finish` (I also tried a plain `exit`) with the same result aborts the whole writing?. I guess that somehow the call to `syswrite` produces a decoupled process in the remote machine than is in charge of fulfilling the writing (since it is a huge message). When the `exit` of the forked local child occurs the remote process is aborted?	[reply]
Re^3: How I'm sure syswrite has finished? by Anonymous Monk on Mar 15, 2009 at 20:54 UTC
Re^2: How I'm sure syswrite has finished? by casiano (Pilgrim) on Mar 14, 2009 at 22:06 UTC
I tried this: `153 foreach (@str_handles) { 154 my $pid = $pm->start and next; 155 156 $b = syswrite $_->{handle}, 157 join(" ", 158 $_->{chunksize}, 159 $_->{A_cols}, 160 @A_lines[$_->{start}.. $_->{end}], 161 @B_lines, 162 "\cN" 163 ); 164 # Wait until the writing has finished 165 #sleep(1); 166 $pm->finish; 167 } 168 $pm->wait_all_children;` [download] But the behavior is the same. Only the sleep seems to work. My guess is that `$pm->finish;` [download] produces a "fake termination": the local process ends while the remote is still writing?	[reply] [d/l] [select]
Re^3: How I'm sure syswrite has finished? by perl5ever (Pilgrim) on Mar 16, 2009 at 05:16 UTC
I think you need to show us more of your program - especially the part wher eyou create the ssh connections. Also, the output of `strace` or `truss` would help.	[reply] [d/l] [select]
Re: How I'm sure syswrite has finished? by ChemBoy (Priest) on Mar 14, 2009 at 17:26 UTC
If it were on a local machine I'd suggest waiting for the other process, but that's not really an option in this case (unless you can wait for the local process? I don't know how P::FM works in any detail, nor how you're set up, so this might be totally useless). Looking at the problem as a whole, rather than the specific solution you've already tried, you might want to change your remote process to send an acknowledgement of some kind back at you when it's done—it requires a little more coding and a little more network traffic, but it seems likely to me that it would work, which appears to be an advantage over the current approach. If God had meant us to fly, he would never have given us the railroads. --Michael Flanders	[reply]
Re^2: How I'm sure syswrite has finished? by casiano (Pilgrim) on Mar 14, 2009 at 21:40 UTC
Many thanks. Apologies for the late answer but I have been away of work a few hours. I tried already your suggestion of trying to produce an acknowledge from the other side (the other side is a process running in a remote machine via SSH. It was started using IPC::Open2) didn't work well. May be I did s.t. wrong. As several people mentioned is difficult to give advice without the whole context, but I can post the two programs (local and remote) if you like. Many thanks Casiano	[reply]