How many workers are you running in parallel?

OpenSSH server has a configuration directive (MaxStartups) which allows to limit the number of not-yet-authorized connections. The default value is 10.

Regarding strace, as root run:

echo 0 > /proc/sys/kernel/yama/ptrace_scope
[download]

Then you can attach strace processes to your workers adding the following code to them:

system "strace -f -o /tmp/strace.$$.out -p $$ &";
[download]

That will dump all the OS calls to the file strace.$$.out. There, you should be able to see the Perl process writing the password to the master pty side and the ssh process reading it from the slave side.