CAVEAT: I'm a bad, bad man and you shouldn't listen to me.

I have a patch for OpenSSH that lets it use .netrc files (used by FTP). This lets me call scp without providing a password and everything just works! Since I'm already foolish enough to use .netrc with FTP I don't suffer any loss of security. If you want the patch I could post it. Or you could do some similar surgery on Net::SCP.

-bad bad sam