rovf has asked for the wisdom of the Perl Monks concerning the following question:

I have an external program which reads from STDIN and returns the result by STDOUT. I would like to feed my data to this program and retrieve the result (and the exit code too, but this is a different matter) without creating a temporary file for the data which I feed to the filter (and using backticks to get the result). BTW, I'm on Windows. How can I do this?

Here is my first attempt. For testing, I use perl -wpe 1 as external program, which acts as "poor man's cat": 

use strict;
use warnings;
use IPC::Open2;
$|=1;
$SIG{PIPE} = sub { print STDERR "SIGPIPE received\n"; exit; };
my $pid = open2(my $reader, my $writer, 'perl -wpe 1');
print $writer "$_\n" for (qw(first second third));
print "Data written\n";
close $writer;
print "Reading from pipe:\n";
while(my $data=<$reader>) {
    print "Received: ".<$reader>."\n";
}
close $reader;
print "No more data\n";
waitpid($pid, 0);
print "Finished\n";

[download]
The program outputs 

Data written
Reading from pipe:

[download]

and then hangs. Obviously, reading from the pipe does not work. Indeed, I found the following warning in IPC::Open2]:

This whole affair is quite dangerous, as you may block forever. It assumes it's going to talk to something like bc, both writing to it and reading from it. This is presumably safe because you "know" that commands like bc will read a line at a time and output a line at a time. Programs like sort that read their entire input stream first, however, are quite apt to cause deadlock.

Maybe this is what happens in my case (though I don't understand *why* this restriction exists). Maybe someone could give me a hint, how to implement this properly?
-- 
Ronald Fischer <ynnor@mm.st>

Replies are listed 'Best First'.
Re: Can't get it working: Bidirectional Pipe on Windows
by BrowserUk (Patriarch) on Mar 09, 2012 at 11:55 UTC

    Consider using Win32::Socketpair instead.

    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.

    The start of some sanity?
[reply]
      IIRC, since perl 5.10, the socketpair builtin is also available on Windows (it also emulates it running a TCP connection through localhost), so, is there any reason to prefer using Win32::Socketpair over it?

      I say, because I have not been maintaining Win32::Socketpair for a long time, but maybe it is still useful.

      BTW, does anybody volunteer as its new maintainer?

[reply]
[d/l]

        is there any reason to prefer using Win32::Socketpair over it?

        It depends whether you're asking about the interface or the implementation.

        I don't know what differences there might be in the implementations, but winsocketpair is obviously easier to use than socketpair. 

        my ($fd1, $fd2) = winsocketpair();

        [download]

        vs 

        use Socket qw( AF_UNIX SOCK_STREAM PF_UNSPEC );
socketpair(my $fd1, my $fd2, AF_UNIX, SOCK_STREAM, PF_UNSPEC)

        [download]

        (Just to make things confusing, it has has to be a UNIX socket, not an INET socket.)

[reply]
[d/l]
[select]
        since perl 5.10, the socketpair builtin is also available on Windows

        I was unaware of that.

        is there any reason to prefer using Win32::Socketpair over it?

        I mostly remember Win32::SocketPair for winopen2(). When I needed bi-di comms with a child process, I found that simply, effective piece of code worked reliably when IPC::Open2, (and the raft of huge & complicated 'portable' modules like IPC::Run with its gargantuan interface, multiple packages and "pump processes"), just hung me out to dry.

        I say, because I have not been maintaining Win32::Socketpair for a long time, but maybe it is still useful.

        I haven't used it for a while, I've only had the need for it once. But I've recommended it a few times without getting negative feedback, so if it ain't broke don't fix it :)

        With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.

        The start of some sanity?
[reply]
        since perl 5.10, the socketpair builtin is also available on Windows
        I should have said that my code is required to run on Activestate Perl 5.8.8.... But thanks for pointing this out.

        -- 
        Ronald Fischer <ynnor@mm.st>
[reply]
[d/l]
      If I understand this right, Win32::SocketPair would require that the other process also uses sockets. Is this correct? However, the remote process gets the data from stdin and writes the response to stdout....

      -- 
      Ronald Fischer <ynnor@mm.st>
[reply]
[d/l]

        No. Look at the definition of winopen2. It dups stdin and stdout to the socket pair prior to starting the child process, so it inherits the sockets as its stdin and stdout; it then restore the originals in the parent once the child is running: 

        sub winopen2 {
    my ($pid, $oldin, $oldout);

    my ($server, $client) = winsocketpair
    or return undef;

    open $oldin, '<&', \*STDIN or return ();
    open $oldout, '>&', \*STDOUT or return ();

    if (open (STDIN, '<&', $server) and
    open (STDOUT, '>&', $server)) {
    $pid = eval { system 1, @_ or die "system command failed: $!"};
    # print STDERR "error: $@\n" if $@;
    }
    close STDOUT;
    open STDOUT, '>&', $oldout
    or carp "unable to reestablish STDOUT";

    close STDIN;
    open STDIN, '<&', $oldin
    or carp "unable to reestablish STDIN";

    #printf STDERR "pid %d, fileno %d, stdout %d, stdin %d\n",
    #    $pid, fileno($client), fileno STDOUT, fileno STDIN;

    return ($pid and $pid > 0) ? ($pid, $client) : ();
}

        [download]
        With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.

        The start of some sanity?
[reply]
[d/l]
Re: Can't get it working: Bidirectional Pipe on Windows
by Marshall (Canon) on Mar 09, 2012 at 13:24 UTC
    As another data point, I did some tests with my own simple "echo" program. I was able to confirm that the first problem, "hangs with blank output" was caused by output buffering. I was unable to find any way to get an EOF recognized in "echo2.pl" for reasons that I don't understand. closing $writer didn't do it. So this still hangs, just further along in the process. I was hoping to find some way to get "echo2.pl" to exit, but that didn't happen.

    The pipe will have a finite size and if you try to cram too much stuff into it via a blocking write, then it can "fillup" and you can deadlock while trying to cram even more into the pipe. But that's not the problem in this simple example.

    I think BrowserUk's suggestion is a good one. Although it is not completely clear to me why this fails. And of course, it is seldom that we have control over buffering settings the other process. 

    use strict;
use warnings;
use IPC::Open2;

$|=1;
$SIG{PIPE} = sub { print STDERR "SIGPIPE received\n"; exit; };
my $pid = open2(my $reader, my $writer, "perl echo2.pl");
print "pid=$pid\n";

print $writer "$_\n" for (qw(first second third ));
print "Data written\n";
print $writer chr 4; #try manually send EOF - didn't work!
close $writer;       #reader still "hangs"

print "Reading from pipe:\n";
while( my $in=<$reader>) {
    print  "Received: ".$in."\n";
}
close $reader;
print "No more data\n";
waitpid($pid, 0);
print "Finished\n";

=echo2.pl
#!/usr/bin/perl -w
use strict;

$|++;
print "first test line\n";
while (<STDIN>){print;}
=cut

__END__
C:\TEMP>perl ipc_open2.pl
pid=5688
first
second
third
Data written
Reading from pipe:
Received: first test line

Received: first

Received: second

Received: third
####now hung up in reader ###
#### for some reason it doesn't see that input pipe
#### closed

    [download]
    Update: Things are a lot easier if this is like a "one line in", "one line out" interface. Or if the response has a "end of transmission" flag, something like blank line means "end of transmission" so that we can quit trying to read more lines that will not be forthcoming. 
    use strict;
use warnings;
use IPC::Open2;

$|=1;   # this is for main program
        # $writer is alread unbuffered by open2
my $pid = open2(my $reader, my $writer, "perl echo2.pl");
print "pid=$pid\n";

for ( qw(first second third) )
{
   print $writer "$_\n";   # one line in, one line out
   my $result = <$reader>;
   print $result;
}

=heading #### echo2.pl #####
#!/usr/bin/perl -w
use strict;

$|=1;
while (my $in=<STDIN>){print $in;}
print STDERR "out of loop\n";  #never gets here!
=cut  #### echo2.pl #####

__END__
C:\TEMP>perl ipc_open2.pl
pid=3244
first
second
third

    [download]
[reply]
[d/l]
[select]
      Thanks for your comment. I also think that it has to do with buffering, and in this case I have zero knowledge about the properties of the other process, except what I already said.

      I also thought first that somehow EOF won't be recognized, so I explicitly sent a "Windows Textfile EOF" (Control-Z), but - as I already had expected - this had no effect. I think I will follow the solution suggested by BrowserUK. To be honest, I had hoped that I would find a solution which will then work on Unix too (although this is not required), but it seems that Windows and Unix are too different in this respect...

      -- 
      Ronald Fischer <ynnor@mm.st>
[reply]
[d/l]
        I would re-consider this idea of not making a temporary file for the input. This simplifies the situation a lot. And prevents deadlocks that can occur in a single process while simultaneously feeding stuff in and reading stuff out.

        The File::Temp module is multi-platform and generates a unique file name that you can use. The best is to delete this file yourself when you are finished with it, although this file will be created in a directory that is periodically cleaned-up - don't assume that file will be there for any significant length of time! - if you are a single user on Windows, you have to run the "file cleanup" utility yourself!

        I would at least try making intermediate interface file before having to attempt complicated code to deal with this. You may be over-estimating the logistical requirements of a temporary interface file and under estimating the complexity of other solutions. The simple idea will be more easily portable.

[reply]
Re: Can't get it working: Bidirectional Pipe on Windows
by locked_user sundialsvc4 (Abbot) on Mar 09, 2012 at 16:01 UTC

    Having once wasted a bit of time as you are now doing with bidirectional transfers over a single pipe, I said “to well with this!” and switched to using a pair of unidirectional pipes.   Problem solved.   It is, simply, a “more satisfactory all-around” stratagem in my humble.   You’re not getting value from using a single pipe to warrant the emptied hair follicles.

[reply]
      I think that if you look at the code instead of just the node title, you will see that the OP does has 2 uni-directional pipes - one for transmission and one for reception.

      The challenge here is how to make this work in an easy multi-platform way. Can you help us with a solution?

[reply]

Back to Seekers of Perl Wisdom