I'm apparently the renegade here: I use the system Perl (Debian's) without reservation. I don't even use local::lib because 99% of what I do is system-wide, so anything that depends on having access to customized personal-scope module installs is worthless to me. This has caused me exactly zero problems in the two-plus decades I've been managing things this way. The closest thing to "problems" is needing to reinstall CPAN modules every couple years when a new Debian version goes stable, but I'd have to do that with any Perl version number change, regardless of who initiates the change.
But a big part of the reason why this works for me is that I don't really care what version of Perl I use. I still write Perl 5.10-compliant code and, although I frequently see people talking about the latest new features added to the language like they're the greatest thing since sliced bread, none of them particularly excite me and I don't miss them. If you do care about your Perl version and/or want all the latest and greatest new features, then using perlbrew is the obvious way to go, since the OS provider's preferences may not align with your own. Similarly, if you're mostly working with things that require specific module versions or are only used by a specific user, then local::lib may make more sense than installing things system-wide.