hmm ... back in the days when I started working with monitoring tools like Tivoli, HP OpenView and BMC Patrol² this was integral part of Alarming/Escalation and Correlation in

• Event Management (ITIL).

Sending e-mails was an optional part of alarming.

> The current system allows for many, many variations: ...

Hmm ... this sounds like an interesting problem for refactoring:

Complicated answer: Build a new system.

Here some tips: (not necessarily all and in this order and with same priority)

• use code inspection techniques to analyze the code (flow)
• ask your main clients for samples of their configurations
• identify the main features
• write a test suite gradually covering all aspects
• document it on the way ¹
• create a safe simulation environment (container?)
• gradually break the system down in modules
• build a new version X.0 with these modules
• run it concurrently in your simulation environment to see if the two versions work the same
• role out X.Y when you feel it sufficiently covers your needs.

I see this as an agile process doing fast prototyping and gradually evolving.

It concentrates first on the core features to have quick satisfactory results.

The decision if 80% or 99.9% coverage is finally sufficient for a roll-out is up to your management.

Feedback from your clients/beta-testers can be used to improve the testing and simulation and evolve your new system.

But I bet you will find and fix plenty of bugs and misconceptions on the way, such that your perception of what the system should exactly do will also change. :)

Easy answer:

• hire me ;-))

UPDATE

Of course if full compatibility to the legacy system³ is not important, incorporate a ready to use foreign libraries or products ².

more updates

¹) also try to visualize the events and how they interfere for documented cases

²) I remember hearing about an open source tool called "Big Brother" which was (partly?) written in Perl (?) and forked into various different new projects "Big Sister" etc. There must be code and modules somewhere for Event-Management, but I wasn't able to locate them with DDG. Compare also Nagios

³) Though a migration path to reuse legacy configuration and processes might prove to be not easier than building a new system. Most of the tips given above about incrementally testing and adapting your system apply here too.