Does'nt that mean that everyone with high enough priviledges will be able to see my passphrase in plain text (if they find it) by examining the memory of the perl process?

Yes, but if they can do that, they can also trojan your perl installation (e.g. add code so it captures your password and quietly e-mails it somewhere). You could keep checksums of everything to detect this, but how do you know they didn't also tamper with your checksumming software? Defending yourself against someone with root access is a pretty much impossible project.