Try cleaning up perl directories in /usr/local: ubuntu-packaged Perl is installed with /usr prefix, and CPAN is installing everything with /usr/local prefix by default, so you should get your system Perl back if you delete the modules installed in /usr/local/ (Perl searches for its data in /usr/local before /usr).

By the way, if you want to install a newer Perl version, it might be better to use perlbrew or perlall.