I think part of the problem comes from the module using Module::Build instead of ExtUtils::MakeMaker

And that's an excellent reason that the OP should submit a bug report against Authen::Passphrase (ie on the basis of its ill-advised choice).
Let's face it ... Module::Build is a heap of shit, and was rightly removed from perl core for that very reason.
They would've preferred to flush it straight down the toilet, but that would've been an ecological disaster.

Cheers,
Rob