hmm ... back in the days when I started working with monitoring tools like Tivoli, HP OpenView and BMC Patrol² this was integral part of Alarming/Escalation and Correlation in

• Event Management (ITIL).

Sending e-mails was an optional part of alarming.

> The current system allows for many, many variations: ...

Hmm ... this sounds like an interesting problem for refactoring:

Complicated answer: Build a new system.

Here some tips: (not necessarily all and in this order and with same priority)

• use code inspection techniques to analyze the code (flow)
• ask your main clients for samples of their configurations
• identify the main features
• write a test suite gradually covering all aspects
• document it on the way ¹
• create a safe simulation environment (container?)
• gradually break the system down in modules
• build a new version X.0 with these modules
• run it concurrently in your simulation environment to see if the two versions work the same
• role out X.Y when you feel it sufficiently covers your needs.

I see this as an agile process doing fast prototyping and gradually evolving.

It concentrates first on the core features to have quick satisfactory results.

The decision if 80% or 99.9% coverage is finally sufficient for a roll-out is up to your management.

Feedback from your clients/beta-testers can be used to improve the testing and simulation and evolve your new system.

But I bet you will find and fix plenty of bugs and misconceptions on the way, such that your perception of what the system should exactly do will also change. :)

Easy answer:

• hire me ;-))

UPDATE

Of course if full compatibility to the legacy system³ is not important, incorporate a ready to use foreign libraries or products ².

more updates

¹) also try to visualize the events and how they interfere for documented cases

²) I remember hearing about an open source tool called "Big Brother" which was (partly?) written in Perl (?) and forked into various different new projects "Big Sister" etc. There must be code and modules somewhere for Event-Management, but I wasn't able to locate them with DDG. Compare also Nagios

³) Though a migration path to reuse legacy configuration and processes might prove to be not easier than building a new system. Most of the tips given above about incrementally testing and adapting your system apply here too.

It sounds like your legacy system has a whole lot of special use cases that have been identified over the years and received custom features to handle them. While the code might be a technical-debt nightmare, the business logic is probably fairly valuable, and replacing it with a completely new system that would need tailored for these use cases might actually be a worse option than paying off the technical debt.

Here's how you can pay off the technical debt:

Your first main goal should be to move all the logic as-is (as much as possible) to a module with a sane API. In a broad sense, the per-client configuration would be the attributes of the object, there would need to be a data store that keeps a history of the notifications sent to that client, and then the main method of the module would be one that delivers new events. If this code is as bad as it sounds, it probably sends the emails directly. You'll need to factor that out so that it "emits" email data in some way that can be passed to a separate mailer object. One final "hidden" parameter is the current value of 'time', which you need to factor out as well.

In a procedural sense, you would:

1. Create a new package MyApp::Notify (or any more specific name)
2. Copy all the event code into a "handle_event" method of this new module
3. Going line-by-line through "handle_event",
   • If the code references a configuration variable, create an attribute on the object for that, and document what it is and who needs to set it.
   • If the code sends a message, change that to returning a value of some sort describing what needs sent.
   • If the code looks at the database or something to see the last message sent for this event, convert that to a more generic "history lookup" method on this object
   • If the code makes references to the current system time, either change that to use a 'time' parameter passed to the handle_event method, or time attribute on the event object
4. Avoid any other refactoring in the event logic. Focus on this one change first before getting carried away with other code cleanup.
5. Find every place that emits events, and have it send that event to this object, created on demand for the current customer. The best way to do that depends on what framework you're using.
6. Write the adapter that takes the return value of handle_event and generates the notification (email, SMS, etc)
7. Do one last painful manual test of all of this, and deploy it. Fight through the bugs you introduced for the next month or so.
8. Finally, begin writing tests for your event module. Create lots of examples of (config, example history, event, expected output) and verify that handle_event generates the correct return value for each event under each set of circumstances. Remember the part above where you converted history lookups to a method of the MyApp::Notify object? That's so now you can mock that method to return the history you want to be tested.
9. Write more tests
10. Write even more tests
11. Begin refactoring the code to be less spaghetti. Use your test suite to validate that you didn't break anything.

Your situation is probably more complicated than that, of course, but that's a general pattern for refactoring a monolith of nasty code which has worked well for me.

Another pro-tip that you might find useful is to use a spool table in your database for outgoing messages. So, instead of the code emitting a message immediately, the code instead writes a message (or parameters to generate one from a template) into a database table with various useful columns like when the message should be delivered, from, to, what caused it to be queued, and so on. Lots of benefits here:

• You can commit the message as part of a database transaction, so either the data is committed and a notification is queued to be sent, or the transaction fails and no notification gets sent.
• You can queue messages to be sent at a later date, so you can release a large batch of messages all at once. If the process that generates the messages fails halfway through, you have a time window to delete the queue for that process and start over. Saves sending embarrassing apologies to customers.
• You can generate the data for a email template independent of the template. This gives you a chance to correct a template before they get mailed without having to re-generate the messages.
• If some rows of the spool table fail to get delivered, you can have subsequent processes analyze and deal with that instead of trying to trap mail-sending errors in the middle of some other application logic. (Definitely add a monitor to alert you about overdue spool table rows.)
• You can write tests that verify some application logic generates a row in the spool table, instead of needing to test against a fake SMTP server.

Hope that helps.

Working with dates and times, especially calculating differences, is always extremely complicated and requires lots spaghetti code.

You have to deal with different time zones (that get re-defined all the time for political shenanigans), daylight "savings" times (different if you cross the equator, different in different countries and redefined for political shenanigans), different calendars altogether (julian, gregorian, japanese, china, north korea, ...) including different times of the year when the year number (or name) changes, leap days, leap seconds, etc...

Tom Scott has a video on YouTube called "The Problem with Time & Timezones" that goes into more details, but, basically, you have to pull in dozens of data files (in different formats, no less) that describe the current and past situations, do a whole lot of complicated parsing and calculations and then you get a result...

...that may not be correct if you schedule an event in the future, because by then a country might have decided to entirely skip a day and/or the IERS(*) might have released a new Bulletin C that adds a 61th second to a specified minute on the last day of March or June or September or December.

Edit: And, oh, if you calculate times from late 1793 to 1805 in France (for with the regions only somewhat match the current borders), please note that they used the French Republican calendar, with 12 months with 30 days each, plus 5 (or 6 in leap years) "extra" days, weeks were 10 days long. And each day had 10 hours with 100 minutes each, each with 100 seconds. And those seconds don't match up in length with the seconds you are probably used to.

Edit 2: Also keep in mind, when daylight savings time starts, the clocks go backwards (usually an hour, could be more, could be less), so usually the same hour repeats in the local time zone. You need to know which version of that hour is the correct one and send out the reminder notification accordingly...

Edit 3: As a side note, did you know that French (and other countries) railways used (past tense, maybe, hopefully) their own time zone(s)? Why French Railway Stations Delayed Their Clocks (& What Happened When They Did).