You're partly right, but...

How is it that the script's source is legible? If the permissions are managed properly then someone has to authenticate as the application's user account to read the thing. If one can authenticate as the user, why can't they bind to LDAP as that user, or read the disk-based private key if not using LDAP to store them? You could say you'd require credentials other than being logged in as the user to get the keys from LDAP, but then how are you storing those other credentials for the LDAP connection in any way that's different from how you'd store the SSH passwords OP is trying to hide? The big advantage to certificates is the information complexity compared to passwords, not that they are stored differently. One could shuffle passwords off into LDAP, too.

Clearly the solution is to ask for the password on launch of the script and have it entered. However, anyone who could attach a debugger to it could still look at its memory and find the strings it has stored. This solution is a partial solution, then, but still better than just shuffling stored credentials from one place to another.

At what point does the security actually improve from storing the credentials differently? Rather than playing ball-and-cups with the credentials, wouldn't it be better not to store them on an account one is expecting to be breached?