In that case, your script would print out the results for each time interval and then go on to evaluate the next lines of the logfiles, like this:

while(<>) {
        $timestring = $_ =~ $some_regexp;
        $min = &get_nr_of_mins ($timestring);
        if(/start:/) {
                /split/ and ++$split;
                /filter/ and ++$filter;
        }
        elsif(/finish:/) {
                /split/ and --$split;
                /filter/ and --$filter;
        }
        if($min % $granularity == 0) {
                print "$time: $split, $filter\n";
        }
}
[download]

(note, that this gives you the number of processes at the time of the last line eaten from the logfile, not an average value for the last interval).

OTOH, if you want to do more sophisticated analysis of the logfile, this approach might be too simple.

Maybe you can consider rrdtool. This gives you a constant-size database with old data (you can configure what is "old" to you) getting stored with less granularity. You can feed it easily from a perl script. I do it with my logfiles to count events, and it produces very nice graphs. You feed it with a line like system ("/usr/bin/rrdtool update /path/to/my.rrd ".$logstr);, where $logstr is your data vector.

Then you can f.e. use a cron job to read the lines of the f.e. last finished hour (last finished time interval) and count your interesting events. In this way you form the vector to feed to rrdtool (or to a dsv-file).

That still doesn't seem to contradict gaal's approach, and only differs from mantadin's in choosing when and how to report. If I'm missing something, then tell me what is wrong with

  my $REPORT_INTERVAL = 300; # seconds
  my %active = ( 'split' => 0, 'filter' => 0);
  my $next_report = date_to_timestamp("...start of day...");
  my $last_report = date_to_timestamp("...end of day...");
  while(<>) {
    # Parse out the fields
    my ($date, $action, $jobtype, $logfile) = /.../;

    # Update current active job counts
    if ($action eq 'start') {
      ++$active{$jobtype};
    elsif ($action eq 'finish') {
      --$active{$jobtype};
    } else {
      die "Huh? $_";
    }

    # Output counts for all report lines between
    # the last printed report and the time of this
    # log line. Most of the time, this will be empty
    # because we won't have reached the next report
    # time yet.
    my $stamp = date_to_timestamp($date);
    while ($stamp > $next_report) {
      report_counts($next_report, \%active);
      $next_report += $REPORT_INTERVAL;
    }
  }

  # Finish off the report for the report periods
  # at the end of the reporting range.
  while ($next_report < $last_report) {
    report_counts($next_report, \%active);
    $next_report += $REPORT_INTERVAL;
  }
[download]

Based on your proposed solution, it seems like you think that you have to correlate a finish event with the start event for that job -- but if all you want is the counts, then as gaal said, the correlation is unnecessary.

If for some reason you do need to correlate them, then you can always keep all active jobs' state in the %active hash:

  ...
  if ($action eq 'start') {
    $active{$jobtype}{$logfile} = 1;
  } elsif ($action eq 'finish') {
    delete $active{$jobtype}{$logfile};
  }
  ...
  my $split_count = keys %{ $active{'split'} };
  ...
[download]