"I just want to download it, not install it." said the OP.

CPAN::Shell->get('My::Module') (untested) should download and extract into your .cpan directory - wherever that may be. AFAIK CPAN doesn't expose anything with any more control.