satanklawz has asked for the wisdom of the Perl Monks concerning the following question:

Okay, I have the equivalent of writers block at the moment. I just need some suggestions, no working code.
What I am doing is this- I read in from a config file who’s structure is the following
/var/log/messages /var/log/secure /var/log/httpd/access_log etc
I take each in, and make and MD5 hash on each. The next part is where I am having the problem.I want to compare the new MD5 hash of the log file to the previous MD5 hash, and if they’re different to run

print “$file is different.\n”;

Am I going about this the right way, or ass backwards? I thought about putting everything into a 3 dimensional array, and just having a counter run through everything… but I don’t know if that is a better or worse way.

Any suggestions/hints/pointers/slaps on my forehead?
The working code that I have so far is:
use Digest::MD5; open CONFIGFILE, "<config.ini" or die "I cant open the config file."; open RESULTS, ">md5TEMP" or die "I cant write the MD5's temp file."; while (<CONFIGFILE>) { chomp; $file = $_; open(FILE, $file) or die "Can't open $file"; binmode(FILE); $md5 = Digest::MD5->new; while (<FILE>) { $md5->add($_); } close(FILE); print RESULTS $md5->b64digest, "\n"; } close RESULTS; close CONFIGFILE;
Thanks, russ.

Fixed for code tags 2002-01-31 -- dvergin

Replies are listed 'Best First'.
Re: Log's and MD5 Hashes
by metadoktor (Hermit) on Jan 31, 2002 at 03:27 UTC
    So I imagine that you ideally want to run this once to save a set of MD5 checksums and then later you would want to run a similar program to compare the 'saved set' versus the 'new set'?

    Well you could create a function that creates a file containing filename,checksum records per line and another function that reads that first file, record by record, and computes the new checksum and does a compare and then outputs your print statement saying that they are different, if they are different.

    metadoktor

    "The doktor is in."

Re: Log's and MD5 Hashes
by Ryszard (Priest) on Jan 31, 2002 at 03:27 UTC
    so, effectively you want to see if one file has a different hash to another file?

    you go thru' the list of files, create the hash and store it against the filename.

    the next pass, you go get the filename and the hash, then using the retrieved filename generate another hash into a different variable, then compare the old and new hashes...

    If your script is a deamon, use a config file to store the files you want to hash, and an $oldhash and $newhash variable to compare any difference between the two values. Make sure you suck in the config file over every iteration so you can pick up any addition/deletions rather than restarting the daemon each time.

    that sounds way too simple an answer, perhaps i dont understand your question properly

      This is what I want it to do.... Report to me what files have changed due to different MD5 hashes, and report to me which files have changed. I probably don't understand your reply enough, mind some sudo (sp) code? I'm terribly sorry if I am being dense, probably the weather. Russ
        Okay- I just realized a better phrasing....
        What I am having an issue with is getting the hashes to match up with a file. You can see that in the code I have sofar it opens up the config and runs a hash against all of the lines in the file, then it stores the hashes into a file. The problem lies here for me:

        if ($NEW-HASH-RUN <> $OLD-HASH-RUN) {
        print "The system file: $file is different.";
        }
Log's and MD5 Hashes -- FINALLY DONE
by satanklawz (Beadle) on Jan 31, 2002 at 06:56 UTC
    In the event that someone winds up wanting to do the same thing i just did, well, here is a cheap way to determine if files have been tampered with. Not exactly tripwire... ;) the format of the config file is a line by line filepath to the file itself. And now, for the code:
    #!/usr/bin/perl -w use Digest::MD5; open CONFIGFILE, "<config.ini" or die "I cant open the config file."; open RESULTS, ">md5TEMP" or die "I cant write the MD5's temp file."; my $a = 0; my $b = 0; my $change=0; open OLDMD5, "<md5" or die "I cant read the MD5's file."; #Do the MD5 Dance while (<CONFIGFILE>) { chomp; $file = $_; open(FILE, $file) or die "Can't open $file"; binmode(FILE); $md5 = Digest::MD5->new; while (<FILE>) { $md5->add($_); } close(FILE); print RESULTS $md5->b64digest, " ", $file, "\n"; #print $md5->b64digest, "\n"; $a=$a+1; } close RESULTS; close CONFIGFILE; open RESULTS, "<md5TEMP" or die "I cant read the MD5's temp file."; my @file1 = <RESULTS>; my @file2 = <OLDMD5>; close OLDMD5; #See what's different while ($a != $b) { if ($file1[$b] ne $file2[$b]) { print $file1[$b]; $change=1; } $b=$b+1; } #If they differ, then update the oldMD5 file to the newest one. $b=0; if ($change eq 1) { open OLDMD5, ">md5" or die "I cant write the MD5's file."; while ($a != $b) { print OLDMD5 $file1[$b]; $b=$b+1; } } close RESULTS; close OLDMD5;

    I know it can be cleaner, but hey, I'm new at this ;)

      Glad you posted this before I replied to the other, you've fixed this up well. There are still some warts, though.

      1. Don't use $a and $b for scratch variables; they are sacred to sort.
      2. Lock your files or use a locked semaphore file. You have races if more than one instance of this runs (e.g. if it is fired by suspicious tcp connections)
      3. When you die, put $! in die's argument list (without any "\n"). That will give you diagnostics you might not get otherwise.
      4. grep can clean up your last while loop, and maybe help the logic, too.
      5. use strict; use warnings;
      6. Take a look at the &Digest::MD5::addfile(\*HANDLE) method
      7. It will take O(N2) to compare all those digests the way you do it. How about making a hash with the digest as key, and checking for existance in the hash?
      You can turn this into a thing of beauty. Good luck, and have fun.

      After Compline,
      Zaxo

      Firstly, you really should be posting comments and follow-up code under the same thread rather than starting new threads for each new discussion on the thread.

      With the code here ...

      while (<CONFIGFILE>) { chomp; $file = $_; open(FILE, $file) or die "Can't open $file"; binmode(FILE); $md5 = Digest::MD5->new; while (<FILE>) { $md5->add($_); } close(FILE); print RESULTS $md5->b64digest, " ", $file, "\n"; #print $md5->b64digest, "\n"; $a=$a+1; }

      ... you could write this a lot more neatly making use of the addfile and reset methods of Digest::MD5 rather than iterating through each line of the file and constructing a new Digest::MD5 object with each loop. eg.

      my $md5 = Digest::MD5->new; while (<CONFIGFILE>) { chomp; $md5->reset; { local *FILE; open (FILE, $_) || warn $!; binmode FILE; $md5->addfile(*FILE); close FILE; } # stuff with $md5 digest methods }

      Now, its beyond this segment of code that the purpose of some of the code flow became a little hazy to me - The second while loop could me written differently with a direct comparison between scalar contexts of the arrays without the use of the superfluous counter variables - Also too, the use of $a and $b as variable names is not a good choice given the magic role which these play within Perl. eg. sort.

      Anyhow, while I would write the code differently, if it does what you need, good luck and well done on your efforts thus far.

       

      perl -e 's&&rob@cowsnet.com.au&&&split/[@.]/&&s&.com.&_&&&print'

Re: Log's and MD5 Hashes
by Anonymous Monk on Jan 31, 2002 at 07:06 UTC
    I got it done. Thanks for all the help. Russ
Log's and MD5 Hashes: UPDATE
by satanklawz (Beadle) on Jan 31, 2002 at 04:38 UTC
    Okay, after taking in some ideas and punching up some code.... I came out with this:

    use Digest::MD5;
    use Algorithm::Diff;
    open CONFIGFILE, "<config.ini" or die "I cant open the config file.";
    open RESULTS, ">md5TEMP" or die "I cant write the MD5's temp file.";
    open OLDMD5, "<md5" or die "I cant read the MD5's file.";
    while (<CONFIGFILE>) {
    chomp;
    $file = $_;
    open(FILE, $file) or die "Can't open $file";
    binmode(FILE);
    $md5 = Digest::MD5->new;
    while (<FILE>) {
    $md5->add($_);
    }
    close(FILE);
    print RESULTS $md5->b64digest, " ", $file, "\n";
    print $md5->b64digest, "\n";
    }
    close RESULTS;
    open RESULTS, "<md5TEMP" or die "I cant read the MD5's temp file.";
    while (<OLDMD5>) {
    $changes = diff (<OLDMD5>, <RESULTS>);
    print $changes;
    }
    close OLDMD5;
    if ($changes != "") {
    open OLDMD5, ">md5" or die "I cant write the MD5's file.";
    while (<RESULTS>) {
    print OLDMD5 $_;
    }
    }


    It fails to work with this error Undefined subroutine &main::diff called at dascript.pl line 27, <RESULTS> line 2.
    What am I doing wrong to compare the two files?
    russ
      Perhaps you need to import the diff function explicitly with:
      use Algorithm::Diff qw(diff);

      -Blake

      how about something like this?
      #!/usr/bin/perl -w use strict; use Digest::MD5; my @fileary; $fileary[0] = { name => '<file>', + oldhash => '', newhash => '' }; $fileary[1] = { name => '<file>', oldhash => '', newhash => '' }; while (1) { foreach (@fileary) { # move the newhash (from last iteration) to the oldhash of # this iteration %{$_}->{oldhash}=%{$_}->{newhash}; #check to see if there is a different has to #last iteration if (%{$_}->{newhash} ne %{$_}->{oldhash}) { print "File ".%{$_}->{name}." has changed\n"; } else { print "oldhash: ".%{$_}->{oldhash}.". newhash: ". %{$_}->{n +ewhash}."\n"; } #go get a filehandle open (FILE, %{$_}->{name}) or die "Can't open '$_': $!"; #go get the hash of the file %{$_}->{newhash} =Digest::MD5->new->addfile(*FILE)->hexdigest; close FILE; } }

      Its not very scalable, however you could dynamically create your hash of arrays from a config file.

      If youre going to run it as a daemon i would suggest a wait/sleep time, and sending an alert to something like NetX/pager/SMS/email etc....