I've never tried this myself, but the cpanp command line utility which comes with perl 5.10(and is part of the CPANPLUS package for older versions of perl) provides an uninstall option.