pc88mxer has asked for the wisdom of the Perl Monks concerning the following question:

I need to write a simple server, like an HTTP server, but that would also be monitoring log files like syslog files or apache log files. The server will collect statistics about the messages written to the log files and serve those statistics via HTTP. It therefore needs to be multi-threaded or use select() so that it can detect when there are new HTTP connections to service or new data written to the log files it is monitoring. The server won't be getting a lot of traffic, but it should be able to handle more than one connection at a time.

Is there a framework that would make this easy to implement? How would you go about doing this? Has this already been done before? Note the server component could be SOAP instead of HTTP - anything that is easy to access. Thanks!

  • Comment on simple server implementation suggestions?

Replies are listed 'Best First'.
Re: simple server implementation suggestions?
by jbert (Priest) on Nov 14, 2007 at 15:54 UTC
    I'd use something like Danga::Socket and write an event-based system. There's already an http server or two (perlbal, mogstored) written using Danga::Socket it which you could modify or take chunks from.

    I think you can add non-socket fds into the event loop (see the OtherFds method) so you can look for events on your watched files too.

    I guess POE is always a popular suggestion too, although I've not used it.

[reply]
      This looks very interesting - especially Perlbal.
[reply]
Re: simple server implementation suggestions?
by perrin (Chancellor) on Nov 14, 2007 at 16:01 UTC
    Did you ask about this on the mod_perl list? I think the simplest solution is to write a basic daemon that polls the log files (this is easy enough with help from CPAN) and spits out HTML files every few seconds. Then you just server the HTML through any web server you have handy. It's much lower tech and quicker to build than a non-blocking I/O scenario.

    If you're determined to use non-blocking I/O, you're probably best off with POE, since it has a lot of parts already built.

[reply]
      I did ask the pre-cursor to this question on the mod_perl list. I've determined that shared-memory is not the best way to collect the statistics I need from all of the child apache processes (and multiple servers), and that a specialized server is a better way to go.

      I'll consider the HTML-page solution and also look into POE.

[reply]
Re: simple server implementation suggestions?
by locked_user sundialsvc4 (Abbot) on Nov 14, 2007 at 18:48 UTC

    You have threads and processes: use them. One process does not have to do all this.

    Therefore, your overall chore breaks down into several independent tasks, which for that matter could be launched from the Unix/Linux command-line by a simple shell-script using the '&' background-jobs inherent to its shell.

    Also carefully consider exactly when the program will need to collect its data and exactly when it will need to offer its results. For instance, if you have one process(es) that sleeps, then wakes up, checks a log-file for new stats, writes them to a file and goes back to sleep ... an ordinary HTTP-server process could build a web-page output that summarizes the current results.

    Keep this very simple. "There is more than one way to do it." Look for several. Put down your coding-pen until the very last moment, if indeed you have to pick it up at all. Existing tools, like tail, might do the job nicely.

Re: simple server implementation suggestions?
by BrowserUk (Patriarch) on Nov 14, 2007 at 17:27 UTC
[reply]
      I don't plan to use it to monitor more than a dozen log files.

      As for statistics, it could be whatever. Most, I imagine, would be of the form: 

      $counter++ if $logmessage =~ m/some regexp/

      [download]
[reply]
[d/l]

Back to Seekers of Perl Wisdom