What are you all doing for this?

I'd use the same procedure I'd use for any other software. I'd certainly not run a fetch/configure/make/test/install cycle on production machines. To name a few reasons why not:

• It's unlikely I want production machines even be able to connect to CPAN mirrors.
• Production boxes should run the services they are supposed to run - and as little else as possible. No compiling. No test suites.
• It doesn't take care of synchronizing.

I'd use puppet, cfengine or some other tool to synchronize my server farm; and only run the fetch/configure/make/test/{install|make-package} cycle on a single box.