need to process the standard output and standard error.

Do you need them as separate streams? How are you processing them? Could you show the code where you are processing the output?

The idea is to have parallel processes running to avoid long running processes block processing other items. I am mainly interested in the error output of the external commands. The exit codes of the external command are not suitable to tell what went wrong.

If you want both stdout and stderr (and don't mind them mixed together:

#! perl -slw
use strict;
use threads;

sub thread {
    my $output = '';
    open CMD, '-|', q[ perl -E"say( qq[$$:saying $_] ), warn( qq[$$:wa
+rning $_\n] ), sleep(1) for 1 .. 4" 2>&1 ] or die $!;
    $output .= $_ while <CMD>;
    close CMD;
    return $output;
}

printf "Got\n'%s'\n", $_->join for map threads->create( \&thread ), 1 
+.. 4;

__END__
C:\test>t-junk
Got
'33168:warning 1
33168:warning 2
33168:warning 3
33168:warning 4
33168:saying 1
33168:saying 2
33168:saying 3
33168:saying 4
'
Got
'30208:warning 1
30208:warning 2
30208:warning 3
30208:warning 4
30208:saying 1
30208:saying 2
30208:saying 3
30208:saying 4
'
Got
'15844:warning 1
15844:warning 2
15844:warning 3
15844:warning 4
15844:saying 1
15844:saying 2
15844:saying 3
15844:saying 4
'
Got
'15032:warning 1
15032:warning 2
15032:warning 3
15032:warning 4
15032:saying 1
15032:saying 2
15032:saying 3
15032:saying 4
'
[download]

If you only want stderr:

#! perl -slw
use strict;
use threads;

sub thread {
    my $output = '';
    open CMD, '-|', q[ perl -E"say( qq[$$:saying $_] ), warn( qq[$$:wa
+rning $_\n] ), sleep(1) for 1 .. 4" 2>&1 >nul ] or die $!;
    $output .= $_ while <CMD>;
    close CMD;
    return $output;
}

printf "Got\n'%s'\n", $_->join for map threads->create( \&thread ), 1 
+.. 4;

__END__
C:\test>t-junk
Got
'34456:warning 1
34456:warning 2
34456:warning 3
34456:warning 4
'
Got
'32700:warning 1
32700:warning 2
32700:warning 3
32700:warning 4
'
Got
'19172:warning 1
19172:warning 2
19172:warning 3
19172:warning 4
'
Got
'22176:warning 1
22176:warning 2
22176:warning 3
22176:warning 4
'
[download]

---

With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority". I knew I was on the right track :)

In the absence of evidence, opinion is indistinguishable from prejudice. Not understood.

Now, I use IPC::Run3 but this also fails, although less often as Capture::Tiny ..

Can you be more specific about how IPC::Run3 fails? It's difficult to solve the problem when we don't even know what the problem is. OK -- almost impossible.

I don't think run3 can be used in a threaded environment. Looking at the module code, it makes the redirects, then does system() i.e. fork/exec. A thread-safe version would open the streams; fork(); redirect i/o; exec().

The second problem is waiting upon the processes. You'll need a mechanism to catch child notifications and forward them to the interested threads (via Thread::Queue probably).

In the end, it might be smarter to implement a separate job-controller thread to launch and reap the external jobs.

Update. On a second thought, if your workers only ever wait on a single process at a time, you can get away with just the waitpid($pid, 0); and no job control is necessary...

It may seem pretty inefficient forcing the data through the file-system, but in practice, usually it is not unless the slave processes write huge quantities of data.

Simple... and neat using File::Temp:

use File::Temp;

sub cmdrun {
    my $cmd = shift;
    my $out = File::Temp->new();
    my $err = File::Temp->new();
    local $/;

    `$cmd >$out 2>$err`;
    $?>>8, scalar <$out>, scalar <$err>
}

use Data::Dumper;
print Dumper cmdrun( q(perl -le "warn q(NOISE); print q(OUTPUT); exit 
+42;") );
[download]