thanks for the pointer on the potential race condition. I haven't seen that as a problem yet, but i can see where it could be.

I do have a waitpid, just lost it when copying. (i also have a lot of other things i've tried and commented out in that area).

when open2 does fail, i actually a message stuck into $out (i assume this is because my $cmd has stderr redirected to stdout).

this is what i am matching with:

$err = 1 if (/^thread failed/);
[download]

and in that case i exit from the loop and send myself the done message.

I had suspected that this (the open getting stuck) was the problem which is why i also put in the extra kill just to make sure, but at this point, all of my tests are actually finishing successfully, (i.e. all output coming out, and no spurious tests left on any of the severs).