This is a topic that had a lot of discussion at various Perl conferences last year. In particular, you might be interested in CPANTS which is Michael Schwern's idea to build an automated CPAN Testing Service.

"The first rule of Perl club is you do not talk about Perl club."
-- Chip Salzenberg