Ooh, this reminded me that many packaging systems have some mechanism by which you can check that what's on the disk match up with what was installed. For instance on RPM based Linuxen you can do something like rpm --verify perl and that'll show you if anything's changed (owner, mode, checksum, etc) from the manifest that was stored when the package was installed. Even if it doesn't support doing this directly the packaging system should have a way to get a manifest of what was installed that you could use in determining what to look at.

Granted that won't be of as much use if you use CPAN to install things after-market (as opposed to an RPM packaged version), but it's another option.