Most perl module installations just involve moving text files. Their "source" is what perl runs. Building from the tarball is slightly more reliable since the installation locations are localized by perl and CPAN.pm.

Some modules use XS or other foreign code, frequently for loading and call system libraries. I prefer to build those from source. That way I know that the C compiler or whatever is compatable with my perl and system libraries.