Re^2: Email Thresholding

Thanks for your thoughts. I do have some Perl code that I can include once I can get to it.

Agreed that this was intended more of a problem space solution but with a Perl implementation since my code is in Perl.

As to the specifics, my code runs every 3 minutes and checks the last 5 minutes worth of logs from a DB. Every matching event is then logged to a different table in the DB. Once all events are gathered, the match table is then run through line by line to generate the emails.

I know, rather lazy as I did not think this would blow up and spam e or my engineers. But, now that it has, here we are.

With the input gathered so far, I think I had a couple of thoughts:

run through the match table at the beginning of the script and store that last matches in a hash for easy lookup later
aggregate the match table query for the look to get a count of each match in the period, rather than all of them
For each match in the loop, check the has to make sure it has not been an hour since last detection. If more than an hour, send the email

Any additional thoughts?

Comment on Re^2: Email Thresholding

Replies are listed 'Best First'.
Re^3: Email Thresholding by mr_mischief (Monsignor) on Apr 02, 2015 at 17:19 UTC
I would query the database with the time constraint of the last 60 minutes. If you're not timestamping your entries with a native DB timestamp, start doing that. I would consider how many varieties of alert I could have, and if that's three or four, I'd limit each type to one per hour rather than one overall. For auditability you're going to want a record of the emails being sent anyway. Have a table where you record the email being sent. Select any sent for your class of alert (or for all if you go that route) from the last hour, by timestamp. If there are none, aggregate all the events from the last hour which you selected above, send an email, and insert your row into the `email_sent` table. The more we discuss this, the more it sounds like Nagios, Mon, Argus, Big Brother, Tripwire, or some other monitoring/IDS solution. You might be able to make a plugin to one of those or at least look to them for how to solve these issues.	[reply] [d/l]

Replies are listed 'Best First'.

Re^3: Email Thresholding
by mr_mischief (Monsignor) on Apr 02, 2015 at 17:19 UTC

I would query the database with the time constraint of the last 60 minutes. If you're not timestamping your entries with a native DB timestamp, start doing that.

I would consider how many varieties of alert I could have, and if that's three or four, I'd limit each type to one per hour rather than one overall.

For auditability you're going to want a record of the emails being sent anyway. Have a table where you record the email being sent. Select any sent for your class of alert (or for all if you go that route) from the last hour, by timestamp. If there are none, aggregate all the events from the last hour which you selected above, send an email, and insert your row into the email_sent table.

The more we discuss this, the more it sounds like Nagios, Mon, Argus, Big Brother, Tripwire, or some other monitoring/IDS solution. You might be able to make a plugin to one of those or at least look to them for how to solve these issues.

[reply]
[d/l]