Alerting On Critical State Changes

jabowery has asked for the wisdom of the Perl Monks concerning the following question:

I'm thinking of writing a system monitor that goes through and checks a bunch of critical system states so that when anything critical changes, an email is fired off.

An obvious way of doing this would be to have a timed loop that calls configurable vector of status check routines/command lines, returning the associated state vector, and then seeing which, if any, of the states changed since the last time through and firing off an email containing those changes.

What is the best practice for this kind of thing?

Comment on Alerting On Critical State Changes

Replies are listed 'Best First'.
Re: Alerting On Critical State Changes by Your Mother (Archbishop) on Mar 25, 2015 at 03:16 UTC
Maybe … use Proc::Daemon + maybe Proc::ProcessTable … start: check … act … sleep … repeat. No wrong® way to do this. Tons of tools. Whatever seems sensible, robust, easy for you.	[reply]
Re: Alerting On Critical State Changes by karlgoethebier (Abbot) on Mar 25, 2015 at 09:10 UTC
"I'm thinking of writing a system monitor..." This a tricky job. Please take look at these tools first: Nagios Icinga NagVis Regards, Karl «The Crux of the Biscuit is the Apostrophe»	[reply]
Re: Alerting On Critical State Changes by afoken (Chancellor) on Mar 25, 2015 at 06:49 UTC
Do you want to re-invent nagios? Alexander -- Today I will gladly share my knowledge and experience, for there are no sweeter words than "I told you so". ;-)	[reply]
Re: Alerting On Critical State Changes by LanX (Saint) on Mar 25, 2015 at 12:07 UTC
Your question could mean anything from how do I build a raft till how to rule the oceans with my fleet. ;) Nagios and other system monitoring tools where already mentioned. CPAN is also full of modules. But if you need a self made approach with deeper insights I'd certainly recommend having a look at Perl for System Administration from O'Reilly You'll find plenty of well explained best practices there. Cheers Rolf _{(addicted to the Perl Programming Language and ☆☆☆☆ :)} PS: Je suis Charlie!	[reply]
Re: Alerting On Critical State Changes by GotToBTru (Prior) on Mar 25, 2015 at 03:12 UTC
Google? In general, your options are determined by OS. Since you're asking on a Perl forum, CPAN search? Dum Spiro Spero	[reply]
Re: Alerting On Critical State Changes by Anonymous Monk on Mar 25, 2015 at 03:22 UTC
cron and diff :)	[reply]
Re: Alerting On Critical State Changes by sandy105 (Scribe) on Mar 26, 2015 at 11:48 UTC
In our project we have healthcheck scripts on every server which monitor cpu/ram usage and queue for processes. All of these have multiple thresholds and alerts are sent via sendmail for support teams when maximum threshold are near. which is similar to what you have described	[reply]