Yeah, you're probably hosting it up.

The one trick I've always had to remember is to use the appropriate Perl related to the mod_perl instance I want to run. There's always a Perl binary somewhere associated with the mod_perl installation; run the CPAN shell or the test script with that binary and install the modules for that installation.

I don't think removing things will help you that much, unless you are very sure to remove one complete mod_perl installation and its entire related Perl installation.