perlguyjoe has asked for the wisdom of the Perl Monks concerning the following question:

Hello Monks! I have a situation where I need to calculate the time between two events in a log file. The log format is like this. Each login has a pair of events, invoke and success. The time of the event is in the second column. I need to correleate the events and report the time it took for them to complete. For extra credit, I will then look to generate 15 minute, 1 Hour and 3 Hour Averages for these events. If you know any way to accomplish this I would be forever grateful.
2013-08-09 13:00:23,784 DEBUG [SSHD] - invoke login agent [username=jo +e] 2013-08-09 13:00:23,791 DEBUG [SSHD] - login agent success [username=j +oe] 2013-08-09 13:10:29,025 DEBUG [SSHD] - invoke login agent [username=bo +b] 2013-08-09 13:10:29,033 DEBUG [SSHD] - login agent success [username=b +ob] 2013-08-09 13:20:19,841 DEBUG [SSHD] - invoke login agent [username=ja +ne] 2013-08-09 13:20:19,848 DEBUG [SSHD] - login agent success [username=j +ane]

Replies are listed 'Best First'.
Re: Calculating time between two events in a log.
by Loops (Curate) on Aug 09, 2013 at 17:42 UTC
    Hello,

    Do you really have html in your log or did you copy and past this data from a web page? Need to know if this is the actual data Perl will be processing. Also, when posting this data, please surround it with <code> tags so that it is more readable... like this:

    2013-08-09 13:00:23,784 DEBUG <a href="?node=SSHD">SSHD</a> - invoke l +ogin agent <a href="?node=username%3Djoe">username=joe</a> 2013-08-09 13:00:23,791 DEBUG <a href="?node=SSHD">SSHD</a> - login ag +ent success <a href="?node=username%3Djoe">username=joe</a> 2013-08-09 13:10:29,025 DEBUG <a href="?node=SSHD">SSHD</a> - invoke l +ogin agent <a href="?node=username%3Dbob">username=bob</a> 2013-08-09 13:10:29,033 DEBUG <a href="?node=SSHD">SSHD</a> - login ag +ent success <a href="?node=username%3Dbob">username=bob</a> 2013-08-09 13:20:19,841 DEBUG <a href="?node=SSHD">SSHD</a> - invoke l +ogin agent <a href="?node=username%3Djane">username=jane</a> 2013-08-09 13:20:19,848 DEBUG <a href="?node=SSHD">SSHD</a> - login ag +ent success <a href="?node=username%3Djane">username=jane</a>

    As for the correlation you're talking about, none of the login pairs take any more than a few milliseconds, is this really what you're trying to compute?

Re: Calculating time between two events in a log.
by kcott (Archbishop) on Aug 10, 2013 at 12:34 UTC

    G'day perlguyjoe,

    Welcome to the monastery.

    Assuming the ,NNN after each HH:MM:SS represents NNN milliseconds, you can use Time::Piece to calculate the time difference like this:

    $ perl -Mstrict -Mwarnings -le ' use Time::Piece; my $tp_format = "%Y-%m-%d %H:%M:%S"; sub get_ms { my ($ts, $ms) = split /,/ => shift; Time::Piece->strptime($ts, $tp_format)->epoch * 1000 + $ms; } my $invoke = "2013-08-09 13:00:23,784"; my $success = "2013-08-09 13:00:23,791"; print "Time diff: ", get_ms($success) - get_ms($invoke), "ms"; ' Time diff: 7ms

    Please do the following prior to posting again:

    -- Ken

      Thanks Ken, and sorry to everyone about my bad post format I have cleaned it up. Your example would work great but I am limited in what modules I can use and Timepiece is not one of them. I should have clarified that. Additionally, the time could be measured in milliseconds or seconds. I am trying to report on the delay which sometime creeps upwards of 30 seconds or more. Any other ideas are welcome, and thanks in advance!
        "Your example would work great but I am limited in what modules I can use and Timepiece is not one of them. I should have clarified that."

        You still can clarify that with an update to your OP explaining the constraints you're working under. I have no idea whether:

        • this is schoolwork and you need to demonstrate that you understand date/time manipulation
        • you didn't know that Time::Piece (builtin) has been included with Perl since version 5.10.0
        • you have a version of Perl prior to 5.10.0 and have some restriction on using CPAN
        • you were unaware that you could install Time::Piece from CPAN
        • there is some other limitation involved

        For future reference, please consider including this type of information when you first post. While we're happy to help, we don't really like hearing: "Your solution is useless to me because of something I'm not telling you about.".

        -- Ken

Re: Calculating time between two events in a log.
by davido (Cardinal) on Aug 09, 2013 at 17:43 UTC

    Have you gotten started yet, or do you need a complete solution?

    Help us help you by posting a minimal segment of code you're currently working with, so we know where you're stuck and how to proceed.


    Dave

Re: Calculating time between two events in a log.
by Anonymous Monk on Aug 13, 2013 at 19:02 UTC
    Here is a solution using Time::Local, (Time::Local was first released with perl 5).
    #!/usr/bin/perl use strict; use warnings; use Time::Local qw/ timegm /; # first released with perl 5 my %invoke; while (<DATA>) { my (undef, $ms, @dt) = reverse split /\D/, $_, 8; $dt[4] -= 1; # months are '0' based my $epoch = timegm @dt; if (/invoke/) { @invoke{ qw/ epoch ms / } = ($epoch, $ms); } elsif (/success/) { my $seconds = $epoch - $invoke{epoch}; my $millisecs = $ms - $invoke{ms}; printf "diff - secs: %2d ms: %3d\n", $seconds, $millisecs; } } __DATA__ 2013-08-09 13:00:23,784 DEBUG [SSHD] - invoke login agent [username=jo +e] 2013-08-09 13:00:23,791 DEBUG [SSHD] - login agent success [username=j +oe] 2013-08-09 13:10:29,025 DEBUG [SSHD] - invoke login agent [username=bo +b] 2013-08-09 13:10:29,033 DEBUG [SSHD] - login agent success [username=b +ob] 2013-08-09 13:20:19,841 DEBUG [SSHD] - invoke login agent [username=ja +ne] 2013-08-09 13:20:19,848 DEBUG [SSHD] - login agent success [username=j +ane]
      Correction to the script
      elsif (/success/) { if ($ms < $invoke{ms}) { $ms += 1000; $epoch -= 1; } my $seconds = $epoch - $invoke{epoch}; my $millisecs = $ms - $invoke{ms}; printf "diff - secs: %2d ms: %3d\n", $seconds, $millisecs; }
        I like this approach. Would you mind sharing the wisdom behind this?