I agree that one has to deal with CPAN more or less, but it is quite normal to have concerns over quality.

A mudule like DBI, as you said, is one of the most downloaded module. What does that mean? that means that it gets a better chance to have bugs found, and fixed because of all the attention it attracted. This creates a positive cycle, it is getting better, so more poeple use it; more people use it, so it gets even better...

I am not sure about Class::Factory, but in general, it is always a good idea to ask for others' experience with a CPAN module before you use it, or the other way around, try it, and share your good or bad experience with others.