I go one step farther and always leave the system Perl alone entirely, except for bootstrapping App::perlbrew. That and App::cpanminus (installable with perlbrew install-cpanm) make installing perl modules really easy and pleasant.

Previously I used local::lib to manage local module installation from CPAN, but I found that some modules need newer versions of core modules, and often that need conflicts with the system perl.