Expect and PTY dropping data

salva has asked for the wisdom of the Perl Monks concerning the following question:

Hi,

I am trying to solve this bug in Net::SFTP::Foreign.

The problem happens when Expect is used to spawn the slave SSH process so that interprocess communication goes through a PTY. After some investigation I have found that the operative system drops data when syswrite is called with more than 512 bytes or when consecutive enough syswrites summing up more than 512 bytes are performed.

Is there any way to work around this "feature"? as for instance, some way to configure the PTY to not drop data or to increment its buffer size?

Or, is there any way in Expect to get rid of the PTY once the interactive phase is over, using a regular pipe for the rest of the IPC?

It seems that HP-UX and AIX are affected by this problem, Linux is not.

update: a testing script:

$| = 1;

use Net::SFTP::Foreign;
# $Net::SFTP::Foreign::debug=1|2|4|32|64;

while (1) {
  my $s = Net::SFTP::Foreign->new('localhost',
                                  password => '...');

  my $r = $s->put($ARGV[0], "/tmp/delete.me");
  print "ok: $r\n";
}
[download]

to test, install Net::SFTP::Foreign and Expect, set the correct password inside the script and run it passing as argument the path to a file of 100KB aprox.

Comment on Expect and PTY dropping data Download Code

Replies are listed 'Best First'.
Re: Expect and PTY dropping data by pc88mxer (Vicar) on May 21, 2008 at 14:22 UTC
After some investigation I have found that the operative system drops data when syswrite is called with more than 512 bytes or when consecutive enough syswrites summing up more than 512 bytes are performed. `syswrite` returns the number of bytes written, and the caller is responsible for re-syswriting the bytes that didn't get written. Is it possible that the return value is not being checked? Or is the OS simply reporting that the `syswrite` call succeeded even though data is being dropped?	[reply] [d/l] [select]
Re^2: Expect and PTY dropping data by salva (Canon) on May 21, 2008 at 14:56 UTC
At the Perl level, syswrite reports that all the data has been written. At the OS level, a process monitoring tool such as truss shows the write call also reporting that all the data has been written. And BTW, the dropped data is not the chunk tail but the head! update: this is the interaction between both processes at the OS level: 27168: write(4, "\0\0\00501\0\0\003", 9) = 9 27170: read(7, "\0\0\00501\0\0\003", 16384) = 9 27170: write(8, "\0\0\00502\0\0\003", 9) = 9 27168: read(4, "\0\0\00502\0\0\003", 16384) = 9 27168: write(4, "\0\0\0" 03\0\0\0\0\0\0\0\r/ t m ".., 38) = 38 27170: read(7, "\0\0\0" 03\0\0\0\0\0\0\0\r/ t m ".., 16384) = 38 27170: write(8, "\0\0\0\rf \0\0\0\0\0\0\004\0\0\0".., 17) = 17 27168: read(4, "\0\0\0\rf \0\0\0\0\0\0\004\0\0\0".., 16384) = 17 27168: write(4, "\0\0\015\n\0\0\001\0\0\004\0\0\0".., 25) = 25 27170: read(7, "\0\0\015\n\0\0\001\0\0\004\0\0\0".., 16384) = 25 27170: write(8, "\0\0\018e \0\0\001\0\0\0\0\0\0\0".., 28) = 28 27168: read(4, "\0\0\018e \0\0\001\0\0\0\0\0\0\0".., 16384) = 28 27168: write(4, "\0\0@ 1906\0\0\002\0\0\004\0\0\0".., 16413) = 16413 27170: read(7, "a l / l i b \nL I B C = \nL ".., 16384) = 1024 [download] Note how reads and writes for both processes interleave until the last one where only 1024 bytes are read by the child and that those bytes do not correspond to the head of the last write operation performed on the parent!	[reply] [d/l]
Re^3: Expect and PTY dropping data by moritz (Cardinal) on May 21, 2008 at 15:02 UTC
This sounds like an operating system bug. If it is indeed, you shouldn't even think about fixing it on the perl level. Write a bug report against the OS instead.	[reply]
Re: Expect and PTY dropping data - solved by salva (Canon) on May 23, 2008 at 07:32 UTC
Or, is there any way in Expect to get rid of the PTY once the interactive phase is over, using a regular pipe for the rest of the IPC? Yes, there is, I have found it! This code spawns a new process with its STDIN and STDOUT redirected to regular pipes/sockets and also attaches a PTY to the child to simulate manual interaction that is handled by Expect. `my $pty = IO::Pty->new; my $expect = Expect->init($pty); my $child = open2($in, $out, '-'); if (defined $child and !$child) { $pty->make_slave_controlling_terminal; exec @cmd; exit -1; } unless (defined $child) { die "unable to spawn new process"; } $expect->expect($timeout, "..."); ... # and once the authentication phase is over, $in and $out are used for + the rest of the IPC.` [download] Also, I have to retain the PTY open until the child program finishes (not just after the authentication is over), otherwise, it gets a `SIGPIPE` and exits. Passing a dash (`-`) to `open2` (from IPC::Open2 or `open3` from IPC::Open3), makes it fork the current Perl process instead of running an external program.	[reply] [d/l] [select]