...except then I would be worried about perl version conflicts and what not

When you adhere to a few basic rules, there are no conflicts to be expected.

Essentially, those rules are:

• Always take care to call the perl binary belonging to the version you want to use. I.e. put #!/usr/local/bin/perl in your shebang line if you want to use your own (non-system) Perl and that's the location where you've installed it to.  Similarly, if you want to get that same version when you type perl on the command line (without path), make sure it's found first along your PATH.  (If you want to use another version, you can always type the absolute path, e.g. /usr/bin/perl to get the system Perl.)

• Don't mess with PERL5LIB on a global level (like in /etc/profile, or some such).  If you really need to fiddle with it, do it locally, i.e. in a specific shell environment, via a wrapper, or by prefixing a command line like so: PERL5LIB=/whatever/dir your-command-here

Every perl binary by default 'knows' where its modules are located (when once correctly set up), so it will find existing modules in, or install new modules to exactly those locations.

Personally, I have at least 2 to 3 Perl versions on all my machines, and I've never had any conflicts due to this. Quite the contrary, certainly less potential conflicts than when messing with the Perl that comes with the system...  The only downside is that you sometimes have to install the same modules several times.

As to using only the pre-built deb packages provided by the distro together with the system Perl, I'd say that's in principle fine, as long as you don't always want the latest and greatest. Virtually all commonly used modules are available, but not always in the most recent version...  OTOH, that's the easiest way, because nothing needs to be built, which is particularly nice with modules that aren't trivial to build, as for example when they depend on various C libs which itself depend on other libs that must be the correct versions, etc.