I have worked on a similar project. An nightly cron gathers a lot of data that doesn't change frequently, from databases and corporate sites outside our server pool. The cron script rewrites a large module used by many other modules. It frees the database load to deal with more volatile data and less commonly accessed data.

It's fast and complex, but modifiable. Not a best preactice perhaps, but it does its job well.