HTH, --traveler

++

Why reinvent the wheel? There are plenty monitoring projects out there. I was about to suggest MRTG and Nagios. If they cannot do the job in this particular case, they are good enough to be adopted and customized.

Leonid Mamtchenkov aka TVSET

exactly...MRTG will do all the data collection for you and you can show off nice uptime graphs. all you need is the mibs installed for what you want to monitor (via SNMP) over the TCP/IP network. microsoft supplies mibs for winnt/2k/xp server. once you have mrtg installed on the monitoring pc you install wincron on it to run mrtg every 5 minutes or whatever, to check the current values in the mibs on the servers. OR you could do something silly like write client/server perl programs yourself and install on each server and the client and hope they work nicely together over time. if you don't understand SNMP & mibs, then don't hesitate to learn this simple protocol and dive into using MRTG, which is also fairly easy to set up. you can learn SNMP and all about mibs in about 1 hour from a good TCP/IP book (a true TCP/IP networking book NOT a microsoft networking book which shows lots of screen dumps and crap)

First you will most probably want this to run as a service(s). Win32::Daemon or Win32::Daemon::Simple will help you with this.

Win32::Service will tell you if a service is running, Win32::FileOp::GetDiskFreeSpace() will tell you how much free space is on a disk drive, so will Win32::AdminMisc, Win32::IProc (installable with PPM from http://jenda.krynicky.cz/perl) will give you information about processes. I don't know what do you mean by "system health" though.

As far as the SMSs are concerned, I believe this depends on your cell phone operator. I send them via SMTP. (Though it seems I'll have to change the operator since the (censored) Eurotel started including adds in the SMSes so I can only use 60 chars:-(

Jenda
Always code as if the guy who ends up maintaining your code will be a violent psychopath who knows where you live.
   -- Rick Osborne

Edit by castaway: Closed small tag in signature

Nuke
nuke@nuke3d.com
www.nuke3d.com

Depend upon your level of experties, it will take some time to get used to with perl. Here at the best we help you to learn perl and integrate existing work (modules).

We would like to see what you have done so far and can take some hints from that. How long is the project is estimated for , any documentation with clearly defined task etc.. would be helpful. This all to guide you in the better direction.

If it's too urgent, the exteternal parties handling the services would be better. If it can take time, you can learn Net:: modules etc.. I would suggest Perl for system administration to start with when you become familiar with perl to handle your task at hand.

artist

There is no ready-made perl script that will fit your problem; you will have to come up with a good design for your project yourself. Some of the questions you might face are

• Exactly which parameters need to be monitored?
• Which parameters will be monitored directly on the server, and which will be monitored remotely?</Ii>
• How will the different processes that monitor the individual servers communicate with your master processes? (e.g. sockets)
• How will the monitoring be administrated? (e.g. web interface)

Then, once you have broken down your project into small tasks, it will be easier to solve the individual problems. You will find that CPAN has many modules available that will help you solve your subproblems, and that your code will mostly be glue code to combine the services these modules provide.

Apart from the modules recommended, also take a look at the WIN32 suite of modules -- they give you a feel of what is at your disposal to monitor a Windows system. You will find that you can even browse the registry or call OLE/COM objects -- probably more than you need.

And for those problems you feel you can't handle, you can ask more precise questions here, at PerlMonks (the more precise, the better we will be able to help you), e.g.:

• How to find out free disk space on Windows?
• How to communicate via sockets between two processes in a network (hint: see the perlfunc manpage as well as the Perl FAQ)?
• How to find out whether a server is running a web server? (hint: try to connect to it and getting a page, e.g. using LWP)
• How to find out whether a server is exporting the correct shares? (hint: there's a module on CPAN to list shares on a SMB machine)

etc. Hope this gives you an idea of how to start.

If you don't know much about Perl at all, I suggest you grab a copy of e.g. Learning Perl first and invest at least three days writing small scripts to hone your skill. You could try your hand at solving some of your small subproblems at this stage (e.g. getting a web page from a web server). It will save you a lot of time later, when you design and implement your master monitoring software.

With mon you wouldn't have to worry about the scheduler/alert part, and focus on writing the actual "plug-ins" that will query the servers and the alert scripts that will page you or send you messages on your cell phone, using the advice given by the other monks...

hope this helps,

update: I haven't used mon to monitor windows servers, but I don't think there will be any problems, since it is pure perl and you write the monitoring/alert modules.

Regards,
Gareth

Not a Perl solution and it sounds likes you have a short time table so I would suggest ServersAlive

It is PC based and it is very good and monitors many things. You can use it free for 10 things monitored. The software costs $99 US. I have it watching a couple hundread computers for availability, disk, services, and processes. It has a paging/email function and you can have it attempt restarts, etc.....

Doing it in Perl is fun but if you have a timetable "do it or else!" It's a question of how much you will cost versus buying something. If something is available, good, and it's cheap, why write something?

I know I am probably speaking blasphmy here! ;-)

If they are from smaller vendors, there might not be anything built in for monitoring. A third-party app might be the best solution if you have to get the project done ASAP. Otherwuise you'll need to look for modules that can capture SNMP and such, at least to start with.

Hope that might provide some direction.

"Ex libris un peut de tout"

Ha! I've done this before...but for HP-UX. Basically the app at the time used a Perl daemon on a crontab to periodically issue a bunch of Unix commands, munge output, and send data back UDP in a lame '###' delimited protocol to a monitoring server. The monitoring server was also Perl, and each packet was then fed into a database as it was recieved and interpreted. A JSP+Applet system exposed the database tables to the web. Initially the monitoring service was also Java, but Java was slow and unwieldy for what we needed to do. An easy project really, and it can be done well any number of ways. I'd definitely want some lightweight system doing this rather than using some evil enterprise management project. Much easier and more fun in Unix though. --Attila (who is not logged in...)

True, it probably also depends on what other requirements you have for the monitoring process and how big your server farm is. Our homegrown Unix solution is pretty nice, use it quite frequently for alert notifications for the system I'm responsible for.

"Ex libris un peut de tout"

First a little background. You must decide which method of monitoring you want to implement. There are three basic types of monitoring, Remote, By Agent, and Invasive. Remote is the easiest because you periodically ping the servers to see if they are alive and attempt to see if other services are running also. The downside to the Remote method is that it does not give you much detail like disk space and other critical services that are not viewable to the outside world. The second method is By Agent. An agent is a small software program that has one or more subroutines that check vital services and functions, collects that information, and then attempts to send this data back to the Controlling Application. The Controlling Application is the software program that collects all the incoming Agent data, stores it, sends the Agent software control commands, and allows reporting on the data received. The last method is the most problematic and that is the Invasive method. The way this method is implemented is that you must insert Agent like subroutines in all of your applications that are running on each server and have that data sent back to the Controlling Application.

I recommend the By Agent method. But because of your short time table, keep the subroutines of the Agent short and sweet.

Here is the pseudo code for the Controlling Application

• Listen for incoming data packets
  • Get data packets
  • Store data packets
  • Check rules
  • Perform actions based upon rules
• Ping all servers in round robin style
• Ping a server
• Send command to all Agents
• Send command to one Agent
• Copy stored data packets to other server for reporting
• Install Agent on a server
• Update Agent on a server
• Verify version of Agent on a server

• Listen for command from Controller
  • Execute command from Controller
• Perform monitoring functions in round robin style
  • Send data packet to Controller based upon rules

Here are some notes about the Agent program. A subroutine that I would have the Agent perform is what is called a low tech test to see if the server is alive. Each time that the Agent performs any other monitoring function, I would have the subroutine write today’s date and time to a text file called pulse.txt. A simple test to see if the Agent is frozen up or the Operating System not allowing writing to a file can be verified by reading the pulse.txt and seeing if the last date and time stamp is close to the current date and time. When a server is rebooted, the Agent will detect that the pulse.txt is not current and will attempt to send a message to the Controller that it detected this problem. Also in worst case, the Controller can attempt to read pulse.txt remotely and see if there is a problem. Some of the basic Agent subroutines that you will create are disk space, services running, and CPU load. I would also include event log monitoring, which there are libraries in Perl already written that you can use to monitor your event logs for specific events. You can create subroutines that monitor Web servers, Application servers, and Network servers. The types of monitoring are endless, just be sure that you have a good business case to justify it. At first I would stick to the following subroutines, Pulse, Disk space, CPU load, critical Event Log events, Memory usage, and critical Services running. Later I would add Performance Monitoring (PERFMON) Log events, and check for runaway or hung processes.

The biggest technical issue to be decided is how the data packets will be sent from the Agent to the Controller. Some applications use SMTP, email, SNMP, create WAP pages and others COM/DCOM messages. Your solution will be based upon your network environment, its stability, how much control you have over it, and what communication methods are available to you. This issue will make or break this project so research it well and come up with a solution that you can live with. Include in your decision making how you will detect missing data packets, data packets not sent, and other worst case scenarios.

I would create a rules based alert system that you can easily transmit to your Agents from the Controller. For example a rule about disk space would be that an alert would be sent if the disk space on any drive dropped below 5%. I would try to keep the rules the same for all servers; otherwise you will be maintaining a different rule set for each server. Come to think of it, maybe you could come up with a rule set based upon the type of server functions it performs. So you would have a rule set for database servers and a different one for email servers.

I would also have a version number in a file, like agentver.txt, that each Agent uses. Then the Agent can report back to the Controller what version of the Agent it is and what version of the rule set that it is using.

Here are some notes about the Controller program. The Controller will have two types of rule sets, Action rules and Ping rules. The Ping rules determine how often the servers are pinged, retry time after one failed ping, multiple failed pings etc. The Action rule set spells out what to do upon each event that the Agent reports back on. So if an agent reports a disk space problem of 5%, then the rule determines what to do next, including creating a log of alerts, sending emails, text paging, and other actions. The Action rule for disk space problems of less than 2% can be different than for 5%.

Reporting directly from the Controller can slow the perform of the Controller to unacceptable levels. I would recommend replicating the master data packet log on the Controller to a Reporting server. The Reporting server can then perform analysis and reports and graphs of the data packets ad hoc. You will have to work out the issues of log rotation, retention, and data storage issues.

Richard

There are three types of people in this world, those that can count and those that cannot. Anon

Helo sir I am very much impressed and moved by your sincerety in writing this reply. Thank you very much for your ideas. Kindly do not mistake me for taking time to reply for your help. Thank you Karthik Param kpm@roberts-companies.com

First off, what's the backup plan if your cell phone is off or you're out of range? Where do they get sent next?

Second, is this your personal cell phone or a company phone? Does your cell provider charge for text messages? It would really suck to have a month plagued with server outages and then get a $100 bill for 1000 text messages. If your company requires that you be on call 24/7, and answering text messages at 4 am is part of the job, they should be paying for the phone. Make sure they know about your plan.

Third, make sure that the phone number is stored in an easily-accessable variable, preferably in a config file, rather than hard-coded. If you move to another company or position or go on vacation, you don't want to keep getting messages.

Fourth, since this code is going to live on 12 different servers, revision control is going to be very important. If you aren't already familiar with CVS, now would be a good time to learn.

-Logan
"What do I want? I'm an American. I want more."