I've done all of these. The primary reasons to build your own perl are to use your own compile options (so you can turn off threads, for example) or to avoid Debian auto-upgrading a module you depend on.

If neither of those concern you, making packages with the script you mention or the one in CPANPLUS seems good because you can uninstall them. The problem you'll encounter is that some modules have build scripts that are hard to deal with, but there's no way around this.