KaZeR has asked for the wisdom of the Perl Monks concerning the following question:

Dear Monks,


I have a perl script which copy files via scp (and open3) from one server to another. After copying, some tasks have to be run remotely (by the sending server): it could be untarring an archive, or moving the file to another place.


Overall, it works nicely. But when i transfer a file greater than a given size (somewhere around 30M it seems) then the file transfer works correctly, but the following tasks (launched by another call to open3) doesn't catch an EOF on STDOUT.

Related code :

  my $pidscp = open3(*HIS_IN, *HIS_OUT, *HIS_ERR, $configTransferBin, 
+@args);
  my @outlines = <HIS_OUT>;

[download]
With smaller files, the script will go further.
With bigger files, it just hangs here ( I've tested by let it run for more than 1 hour). The output isn't verbose at all : at worst, stdout should get only 3 or 4 lines), and the size of the datas sent to STDOUT by the remote command doesn't depend upon the size of the actual file.

Any of you ever faced an issue like this one?

Thanks in advance.

Replies are listed 'Best First'.
Re: IPC::Open3 STDOUT does not EOF?
by ikegami (Patriarch) on Jul 11, 2010 at 18:45 UTC

    Your child probably filled up the STDERR pipe and blocked trying to add more. You have a race condition.

    • The child is blocked printing to STDERR until the parent empties some of the child's STDERR.
    • The parent is blocked reading from child's STDOUT until the child sends to STDOUT.

    If you don't need to capture the child's STDERR, this problem can be avoided by sending the child's STDERR to the parent's STDERR or to a file (such as /dev/null). 

    open3(*HIS_IN, *HIS_OUT, '>&STDERR', ...)

    [download]
    open(local *NULL, '>', '/dev/null') or die;
open3(*HIS_IN, *HIS_OUT, '>&NULL', ...)

    [download]

    If you need to capture the child's STDERR, you'd do better to use a higher-level module such as IPC::Run3 or IPC::Run. Otherwise, you'll need to use select or threads or non-blocking IO.

[reply]
[d/l]
[select]
Re: IPC::Open3 STDOUT does not EOF?
by ww (Archbishop) on Jul 11, 2010 at 17:55 UTC
    OS? Could you be running up against a 32M file-size limit?

    and what do you mean re smaller files "will go further?" Go to completion; Go byond 32M? And what's going to STDOUT?

    You really need to provide more info, including error msgs, etc.

[reply]
Re: IPC::Open3 STDOUT does not EOF?
by Khen1950fx (Canon) on Jul 11, 2010 at 22:41 UTC
    I tried to replicate the problem but couldn't get it to hang. Here's the code that I tried. 
    #!/usr/bin/perl

use strict;
use warnings;
use IPC::Open3;

my $conf;
my @args;

local(*HIS_IN, *HIS_OUT, *HIS_ERR);

my $childpid = open3(*HIS_IN, *HIS_OUT, *HIS_ERR, $conf, @args);
close HIS_IN;
my(@outlines) = <HIS_OUT>;
waitpid($childpid, 0);
if ($?) {
    print "Status is: $?\n";
}

    [download]
[reply]
[d/l]
      You didn't specify which command you used. Try 
      #!/usr/bin/perl

use strict;
use warnings;
use IPC::Open3;

my @cmd = ( perl => (-e => 'print STDERR "x"x1024 for 1..100') );

local(*HIS_IN, *HIS_OUT, *HIS_ERR);
my $childpid = open3(*HIS_IN, *HIS_OUT, *HIS_ERR, @cmd);
close HIS_IN;
my(@outlines) = <HIS_OUT>;
waitpid($childpid, 0);
if ($?) {
    printf "Status is: %04X\n", $?;
}

      [download]

      On linux, readline never returns.

[reply]
[d/l]
[select]
        You're right. I tried it, and readline never returned. Thanks.
[reply]
[d/l]

Back to Seekers of Perl Wisdom