If you want a sort of "best of both worlds" approach, you could build a system that can compile your modules, and then just share them around by either making tar.gz packages or using NFS. Then you can quickly recompile for a new platform, but don't need to do it on every machine.

The way Krang handles this is by building binary distributions, which are just tar.gz packages with everything compiled.