I like to use the CPAN shell's readme command to check out unfamiliar modules. I tend to make my first judgement of quality from it.

I like to see:

1. A description of the purpose and scope of the module.
2. Optionally, a comparison to other similar modules.
3. Installation instructions, abbreviated if there is an INSTALL file.
4. A list of prerequisites, both Perl modules and system libraries (important!)
5. Credits: who's writing and patching the module.
6. Contact information, including homepage.

A text copy of the perldoc output is sometimes verbose, but if the pod quality is good it tells me about all I need to know about the module. I'd prefer a shorter specially-written README but I don't hold a doc copy against the module.