gnu@perl has asked for the wisdom of the Perl Monks concerning the following question:

Being that I am still in design time, I thought this might be a good place to start this.

I am starting a program for managing and monitoring log files on various flavors of Unix/Linux. The objectives are as follows:

* To be able to view log files or the last X number of lines from them.
* Rotate theses files on demand instead of waiting for logrotate (some machines don't have it).
* Retrieve different files from different servers.

These are the basic tasks I must perform, other 'spiffy' things will probably be added as time goes on.

What I am looking for is other ideas on how to go about getting the data from the remote servers. I am limited by not being able to have any sort of server process running on the remote machines. I will have both telnet and ftp access to those machines.

The first idea I had was to use open3 to telnet to the remote server and either cat or tail the logfile. This proved to be problematic. I then switched to using Net::Telnet. This worked fine in my original trials, but I am concerned about hangs. I have not yet tried ftp simply because I do not want to bring the file over to the local system, I just want to view it's contents, but it is still an option.

Essentially what I wanted to do was get the data from the remote server and store it in a hash with the servername:file as the key to the data. I am aware that this might also be a problem if someone tries to bring a 1GB file, but I am still working on the safety mechanisms for that. First I need to focus on how I am even going to get the data before I make it 'safe' to do.

Any ideas on how I can handle this unique circumstance?

Thanks in advance, Chad.

Replies are listed 'Best First'.
Re: Viewing log files on remote servers.
by cLive ;-) (Prior) on Oct 08, 2002 at 20:15 UTC
    To be able to view log files or the last X number of lines from them. my $last_X_lines = `tail -nX'; # where X is the number required

    Save to a new file if you want to FTP down just that part of the file.

    Rotate theses files on demand instead of waiting for logrotate (some machines don't have it).

    Net::Telnet - and do it manually.

    Retrieve different files from different servers.

    Net::FTP

    before I make it 'safe' to do

    Then I guess you'll be using SSH rather than Telnet :)

    Of course, I'm lazytm, so I'd write a perl script to do everything from the server, FTP it up and run it through Telnet when needed, rather than the other way round (I don't know and can't be bothered to learn shell scripting :). assuming there aren't firewall issues, the scripts can then post/FTP the required data to a script/server somewhere where you can host web based scripts/FTP server.

    .02

    cLive ;-)

    --
    seek(JOB,$$LA,0);

[reply]
[d/l]
Re: Viewing log files on remote servers.
by grinder (Bishop) on Oct 08, 2002 at 21:06 UTC
    I'd like to look at this from a different angle.

    I am a big fan of syslog servers. Therefore I assume you are talking about files that are generated by syslog (which excludes Apache for instance. Other applications, such as Samba, can be compiled to use syslog instead of its internal logging mechanism).

    You set up a single host with a lot of disk space (I generate about 500Mb of logs per day), set up syslogd on it to accept network connections and configure the rest of your servers to log to it. The first win is that you have moved logs off all your other servers. Less worry about running out of space on /var, and should crackers crack a server they can't easily cover up their tracks, because the log files aren't around for them to diddle. Which reminds me, remote logging is the only use I can see for wanting to use the --MARK-- thing in log files: it serves a heartbeat to let you know your other systems are still ticking over.

    But I digress. What you have now is a single set of logfiles on one machine. One that will hopefully rotate and expire log files just the way you like. Your task is now to write one set of scripts to deal with them. Try logwatch and see if that doesn't meet your reporting needs. Otherwise File::Backwards will do the trick of tailing a file quite nicely.

    In any event, if you centralise your logs in one place you make life much easier for yourself in terms of both collection and analysis. I strongly recommend you consider this approach. All that telnet and ftp stuff sounds like cruft waiting to be happen.

    I don't think your circumstances are unique, I think it's been done before, many times...

    <update>I would advocate that your in-house programs be modified to use syslog. It's a snap to do in C, and a number of modules exist for Perl (Unix::Syslog, Sys::Syslog and Net::Syslog). Seriously though, your telnet solution is just not gonna fly. Writing an app to run on 70 servers (even if they're all Linux or Solaris or whatever) and making it work correctly is no fun. You'll be forever coding around the warts and cruft that has built up over the years. You'll expend all your energy just to stay in one place. It will be a bear to deploy, teaching people how to use will suck up all your time, and in the end it will be ignored. Build the infrastructure and people will have an incentive to make their programs work with it. Remember, let laziness be your guiding light.</update>


    print@_{sort keys %_},$/if%_=split//,'= & *a?b:e\f/h^h!j+n,o@o;r$s-t%t#u'
[reply]
[d/l]
[select]
      Actually it is kind of unique. The idea is that different users can have access to different servers. The log files are not all generated by syslog, most of them are generated by in house programs.

      Also, I have no control over what servers may be added or removed. If a server is added I cannot guarantee that it would be configured for centralized logging.

      With the method I have planned, any user with telnet access to a server could watch any log files that their login allows them to see. I could distribute the app among users and not have to worry (relatively) about who sees what data on what servers. We have over 70 servers here not to mention all of the workstations (probably over 400) distributed across our WAN.

      I do agree with you that centralization has it place, unfortunately this does not seem to be one of them. If I were administering a web farm or email farm that might make a lot more sense, but most of the servers are stand alone with varying access and applications.

[reply]

Back to Seekers of Perl Wisdom