"It seems like a security risk to me ... Yet I can't find anyone else having this issue because ..."

Because it is not an issue.

"When building perl (I'm actually installing with plenv) ..."

And this is your mistake. Use perlbrew.