jeff061 has asked for the wisdom of the Perl Monks concerning the following question:

First off, i know very very little perl, which is why i'm here. First time posting, but i've used this as a reference page for awhile.

So i have a script that needs to parse out a comma delimited syslog file based on error messages i have in a seperate file. Now i have one last thing to do to get this thing over with. After a downtime i need to search for a corresponding uptime message and append the uptime date and time to the end of what the script parses out. Something to keep in mind, the uptime may come after the script goes through the syslog. right now each time the script runs(every 15 minutes) it parses out the data, sends the output to a server, and clears the syslog. So i am going to have to have a way to keep a entry in memory, or a seperate file, until the uptime message gets logged.

Examples and current script:
The syslog is formatted identical to the current output, it has no uptime information conatined on the same line.

Current output:
server name, date, time, error message, error type(waning, error, notice)

What i need:
server name, date, time, error message, error type(waning, error, notice), uptime date, uptime time

Current script:

$logdir= "d:/logs/"; $logfile= "sysmonitor.log"; $errlst= "errorlist.txt"; &FltErr; sub FltErr { open (FH, "< $errlst"); my @errors = <FH>; chdir($logdir); open (msg, $logfile ); open (FILEHANDLE,">NTDTM.txt"); while (<msg> ) { chop($_); foreach my $error (@errors) { chomp ($error); if ($_ =~ $error) { print FILEHANDLE "$_ \n"; print "$_ \n"; } } } close (FILEHANDLE); } &Mail; sub Mail { `blat NTDTM.txt -t user\@email.com -f user\@email.com -s "Windows DTM +logs"`;

Any help is GREATLY appreciated.

Replies are listed 'Best First'.
Re: comma delimited, syslog parsing
by ptkdb (Monk) on Oct 13, 2003 at 14:29 UTC
    So long as you can absolutely guarantee that commas will NOT appear any of the fields, you can quickly parse the line from the log file into a list with 'split'. Define some constants at the top of your file to represent their positions:
    sub SERVER_NAME() { 0 } sub DATE() { 1 } # ... sub ERROR_TYPE() { 4 } # etc. while( <msg> ) { chomp ; @elements = split /,/ ; if( $elements[ERROR_TYPE] eq 'ERROR' ) { # handle error next ; } }
    As for matching up uptimes and downtimes, that's a little trickier. You need to maintain some state information between each invocation of the script or the script needs to be constantly watching this file. The 'Storable' module will let you store a hash structure in a file and then retrieve it. This psuedo code may give you some guidelines where to start. It does not account for such frustrations as the log file disapearing and being recreated between runs.
    use strict ; use Storable ; use FileHandle ; my($hashRef, $log) ; # program startup # read previous data: if( !-e 'myFile' ) { # myFile not found, assume we're starting from the beginning $hashRef->{lastPosition} = 0 ; # start at the beg # add other values as needed } else { $hashRef = retrieve('myFile') ; } $log = new FileHandle('logfile') ; $log->seek($hashRef->{lastPosition}, 0) ; # go back to previous point +searched ############# ## Process ## ############# ## ## save where the end of the file currently is, so ## we don't have to scan the whole file again next time ## $hashRef->{lastPosition} = $log->tell() ; # get file offset ## ## Store new state data ## store $hashRef, 'myFile' ;
Re: comma delimited, syslog parsing
by dragonchild (Archbishop) on Oct 13, 2003 at 14:06 UTC
    Ok. I'll bite - what's the problem? That looks like it will do what you're looking for ...

    ------
    We are the carpenters and bricklayers of the Information Age.

    The idea is a little like C++ templates, except not quite so brain-meltingly complicated. -- TheDamian, Exegesis 6

    Please remember that I'm crufty and crochety. All opinions are purely mine and all code is untested, unless otherwise specified.

      It does everything but append an uptime to the end. The uptime is based on a seperate corresponding event(i have not linked uptime and downtime events to each other anywhere).

      So basically, it can filter out events based on what i have in the txt file. But it has no way to save the events it filtered, wait for an uptime message that matches the downtime, and append the date and time to the output. The difficult part being that every time the script runs, it clears the syslog. So i need it to be able to keep back events with no uptime until an uptime message is logged, then have it get sent off.

      This part is over my head, don't even know where to begin. Hope i am being at least semiclear.

        Stop coding. You haven't figured out your design, yet. You need to sit down and write out on paper the exact set of steps you want this program to do. Once you have the flow laid out, your coding needs will become a lot clearer.

        Remember - a computer is a being with a 0 IQ that will do exactly what you want over and over. Until you know what you want to do, how can you tell the computer?

        ------
        We are the carpenters and bricklayers of the Information Age.

        The idea is a little like C++ templates, except not quite so brain-meltingly complicated. -- TheDamian, Exegesis 6

        Please remember that I'm crufty and crochety. All opinions are purely mine and all code is untested, unless otherwise specified.

Re: comma delimited, syslog parsing
by ant9000 (Monk) on Oct 13, 2003 at 14:21 UTC
    To really answer you, I need a definition of what you intend for corresponding uptime message. Anyway, I think you want to know "how to preserve (across different script runs) the records that might need to be reprocessed after an uptime message gets logged"... well, I would use a hash tied to a file (see tie or Tie::Hash).
    Whenever you find an incomplete record you create a new key in the hash with the relevant data for value; when you get the uptime you write out the saved data with the uptime timestamp and you undefine the key from the hash.
Re: comma delimited, syslog parsing
by jeff061 (Initiate) on Oct 13, 2003 at 14:56 UTC
    Thanks for the answers. Going to answer a couple comments, still going through the suggestions.

    I know what i need the script to do. Parse out lines with specefic error messages(listed in an external text file), and look for corresponding uptime messages and append the date and time of that to the the original parsed line, and mail it.

    Example of what i mean by corresponding line:
    A error may get logged that says, for example, "SQL server is down", right now this would get parsed out fine, and in the right format. Later on an error message may come in that says "SQL server is up". This may not come in until after the downtime was logged and the syslog was cleared. I need to be able to have the script know to look for that message, and append the date and time of that message to the "sql server is down" event that was already parsed out. There will be dozens of downtime messages, and each with a different uptime event.

    Now, everything works but the whole uptime aspect. Nothing has been done there, and my feeble coding skills are preventing me from getting much further.

    I appreciate the help.

      Allow me to rephrase:
      1. You have a map between downtime message and uptime message.
      2. You want to parse out the downtime message and put it somewhere.
      3. Later, when the corresponding uptime message arrives, you want to append that to the downtime message (parsed earlier).

      Ok. That is a lot clearer, but still not clear enough. Here's a few problems:

      1. Let's say you have two SQL Server instances, and both go down at different times. Both will probably come back up at different times. How are you going to match the uptime message to the right downtime message? (One possible solution is in the next point.)
      2. Your datastore (the place you're storing errors in) ... that's getting more and more complex. But, all you really need is:
        (Error type) - (Who had the error) - (When did it happen) - (When was +it resolved)

        Many of my esteemed brothers will take me to task for this, but it sounds like you really want a database for this. Now, I'm not suggesting you use Oracle or whatever, but even something like DBD::SQLite would be very handy. This way, you can easily find where in your datastore your corresponding downtime message is. Also, it will help you organize what you're actually keeping in your datastore.

      3. Of course, figuring out how to match the two SQL Server downtime messages may be easy. Figuring out how to do that for Net Gateway messages may also be easy. Figuring out a way of doing it for both at the same time may not be so easy.

      I'm not trying to discourage you. These are questions that are necessary (imho) to answer before you can really get something usable up and running. (specially if, if I assume right, you will want (or be asked for) reporting on these downtimes. Any non-trivial reporting usually is benefited by the use of databases.

      ------
      We are the carpenters and bricklayers of the Information Age.

      The idea is a little like C++ templates, except not quite so brain-meltingly complicated. -- TheDamian, Exegesis 6

      Please remember that I'm crufty and crochety. All opinions are purely mine and all code is untested, unless otherwise specified.

        Not discouraging me at all. I have never really scripted or coded at all in my life, my boss is into scripting. Therefor he assigned this to me to try and get me more comfortable with scripting. I appreciate any questions asked that will allow me to help you guys help me.

        Anyways here is a more broad picture of what is happening. For the last few years the unix systems have had a shell script that parses the unix syslog files for specfic errors and events. These would then get sent to a unix server via email. This unix server would then generate reports and emails based on this data. I have been tasked to integrate windows machines into this solution(yes i am the NT Admin, let me apologize in advance). So i have to generate and email data to that unix machine in a way that it can read(which is specific to the formats i mentioned earlier). Once the logs are generated, formatted and sent to that unix server my hands are washed clean of the process, and i have no reason to maintain a DB of these results. It is all being stored and maintained on the Unix server.

        What you rephrased is accurate, however i have not yet mapped downtime events with uptime events. Thats purely logical at this point.

        The sql error was just an example, in reality it will be much more detailed, and will include the name of the instance. If there are any problems between getting uptime and downtime messages crossed this is something i can take care of through software(i'm using servers alive, Kiwi syslog and an event to syslog service).

        Basically every 15 minutes it will be scanning a syslog with roughly 20 lines of events. From this all downtimes need to be matched with an uptime, and if there is no uptime then that downtime event needs to be kept from being mailed until an uptime event is logged.