Any time you start multiple concurrent IO sessions, you are asking for trouble. Why are you using the -t option? Are you really running some curses-like application? If I run your program without the -t option, I don't see the 'connection closed', and the tty is fine. Remember, just because you are using the -t, all of the concurrent output is coming to your shell/tty. It's like having 5 programs opening the same file for writing at the same time. Results are unpredictable.
Have you considered forking to a perl program using an SSH module? You have more control.