Your question depends on a lot of factors, mainly, is this a development box? Is this your home account? etc.

Personally, I roll my own perl and cpan what I need so I have a very clean (well, clean enough) development environment that is completely separate from the operating system's idea of what it needs. Sometimes when you are supporting existing code, you need your environment to match as closely to what existed when you released. If you are like me and like to keep your OS updated weekly to catch patches and new features, you soon realize that using the OS to manage your perl installation can be hazardous.

Linux distributions in general apply patches to the things they use internally. This includes perl modules. That means bad things can happen if you rely upon a certain behavior in a certain module, and the distro maintainers pull the rug out from under your feet with a patch.

It really doesn't require that much effort to compile perl from source with a prefix that points to a location separate from the rest of the OS (ie. in your home directory with your exec path set to ~/bin or whatever). This way in a multi-user environment two users can require two different versions of a particular module with the operating system requiring a third and if they all have their own space, everyone is happy.