Maybe you can turn on some extra params to the connection to see more of what's going on, like

 
    my $ssh = Net::SSH::Perl->new( "host", 
                                   debug => 1,
                                   options => [
                                                 "ConnectTimeout 30", 
                                                 "LogLevel Debug3",
                                                 "ServerAliveInterval 
+10",
                                                 "ServerAliveCountMax 
+5",
                                               ],
              );
[download]

With a bit of luck, you might get ssh behaving in a way that would not make Perl show the unwanted behavior even with a flaky connection or over a link that faces packet loss.

Hth,