strace -o /tmp/out -f perl -e 'print `ssh localhost -tt -l root tcpdum
+p -c10 -nntttt`'
[download]

On my computer that works pretty fine, and I don't see any tty access. Maybe just updating the ssh client could solve your problem.

update: Ah, I can see what's happening:

ssh -tt ... reads the terminal flags from STDIN (ioctl(0, TCGETS, ...)), sets new ones (ioctl(0, TCSETSW,, ...)) and runs the remote command. Then, upon exit, it restores the original flags.

The issue is that with multiple ssh processes running in parallel, some of them may read the already modified flags and so later, reset STDIN to an incorrect state.

An easy workaround is to wrap the part of the code where the threads are started and then joined with an extra couple of TCGETS/TCSETSW ioctl calls. Another option is to just redirect STDIN from /dev/null.

update 2: The following program works correctly on my computer:

use strict;
use warnings;

use threads;
use threads::shared;

my @threads;
my @servers = (('localhost') x 100);

my $lock:shared;

foreach my $server (@servers) {
    chomp $server;
    push (@threads, threads->create (\&dumpServer, $server));
}
foreach (@threads) { $_->join(); }

sub dumpServer {
  my $server = shift;
  my $net = `ssh -l root -tt $server '/usr/sbin/tcpdump -c10 -nntttt 2
+>&1' </dev/null`;
  lock($lock);
  print "TEST - $server\n$net\n\n";
}
[download]

Thank you, awesome find! This will save me from doing another 8 hours of troubleshooting. :)

Well, your solution worked :) At least as far as fixing the term issue. However, a side effect is that the processes continue running on the servers after the SSH connection is killed (arrrrgh). So sorta back to square one.

I have had some luck with the following line, but it feels like a cheesy way (and not 100%) to fix the issue. Before the print and in various places:

system('stty sane');

So confirmed that it's definitely a term settings issue. Over 95% of the time there are no print issues. However, now trying to pipe the output into any other tool crashes the program (leaving tons of threads running and dormant processes). Other users will certainly pipe the output so that's a problem. Maybe I need to pass Perl code to the servers like you were saying (to kill the processes once SSH is killed). This way I don't need to use that buggy -tt SSH option.

try using /dev/zero instead of /dev/null.