In general it's much easier to write secure Perl code than secure C code. If your code is carefully written and uses taint mode, and the modules you're using are carefully written pure Perl, I would guess your system is less likely to contain an exploitable buffer overflow than OpenSSH. Still, it would be very easy to get the encryption wrong or make other mistakes that would open up your system to security problems; an advantage of OpenSSH is that many many many people have inspected and anlyzed the code.