Use scp and ssh keys. That's exactly what they're for.
In reply to Re^2: Net::SFTP 'invalid packet header' by pileofrogs in thread Net::SFTP 'invalid packet header' by timecatalyst