As others have already mentioned, this is a realm that requires a lot of planning and thought. And there are plenty of off-the-shelf packages available out there.

With that, I'd throw my two cents in for mon: http://www.kernel.org/software/mon/

It's written almost entirely in perl, it's extremely extensible, and it's fantastic for application-level monitoring, which I feel like a lot of the network-monitoring applications aren't well-suited to.

We use it to monitor and support an extremely complex server infrastructure that has a lot of dependencies and moving parts - mon has done a fantastic job for us.