Traditionally both cert and key should be PEM formatted. Since version 1.988 it also supports DER and PKCS12. It will automatically detect the format but you must use the same format for both key and certificate. I have no idea what's wrong in your specific case and I have no way to reproduce it but it should work in general as I described and there is also the test t/cert_formats.t in the distribution to check the relevant combinations.