Re: Module to limit email floods?

I assume it is your script, and it is cron'd? I used to monitor roughly like this:

if (problem) {
   fixIt;
   logIt;
   if (-f $indicator) {
      pageMe;
      unlink $indicator;
   } else {
      touch $indicator;
   }
} else {
   unlink $indicator;
}
[download]

Thus, I'd get paged only if a problem persisted. With reliable "fixIt" and detection methods, and an appropriate cron schedule, that served me well.

If your condition isn't programmatically fixable, I'd change that around to perhaps this:

if (problem) {
   logIt;
   if (! -f $indicator) {
      pageMe;
      touch $indicator;
   }
} else {
   unlink $indicator;
}
[download]

Thus, you'd only get paged initially. Or you might also page yourself when the error is cleared; or check the timestamp of the indicator and page every hour; or whatever.

(I remember a similar bill when first testing such scripts, despite "unlimited" texts on my pager...)

Update: The best way to limit alerts is to limit alert conditions...
pileofrogs, what was the condition that lasted several days? How was it resolved? Could it not have been resolved automatically (with perl)?

Comment on Re: Module to limit email floods? Select or Download Code

Replies are listed 'Best First'.
Re^2: Module to limit email floods? by thargas (Deacon) on Oct 15, 2010 at 12:28 UTC
My problem with all these different methods of rate-limiting is that there is no One True Way, even within a single system. You may want to only get a single message for one kind of error condition, but for some others you may want periodic messages, and for some you might even want to have the messages get escalated to some other person or use some other method of notification. What we really need is a completely separate, simple, configurable notification system, with an API, web interface and command-line tools. If done properly, any other system that needed notification-sending could use it. Unfortunately, to be really effective, it needs to know the current status of "things", which is getting into the realm of monitoring. I don't see the various monitoring systems changing to use this hypothetical notification system, so the only way I can see to make this work would be to include adapters for various monitoring systems which would continuously import state. Then the notifications could be triggered off it. That would be a lot to do, hoping that it would be used.	[reply]
Re^3: Module to limit email floods? by Anonymous Monk on Oct 15, 2010 at 13:16 UTC
I think you described log4perl (or whatever) + syslog + nagios , maybe	[reply]
Re^4: Module to limit email floods? by thargas (Deacon) on Oct 19, 2010 at 17:09 UTC
Hmm. Maybe you could implement this with log4perl as one of the pieces using a custom appender(s) and systematic use in the logging programs/modules, but I doubt that it'd be very usable or efficient. No. I was thinking of something more like a mq or soap with structured messages feeding a database for the input+state. And something that doesn't exist for configuration+notification.	[reply]