I think that I would separate the concerns by prefiltering the files and feeding each thread's lines to different processes via pipes:

#! perl
use strict;

#### Warning! Untested code -- just an example.
for my $log ( <*.log> ) {
    open my $fhLog, '<', $log or die $!;

    my %kids;

    while( <$fhLog> ) {
        my( $tid ) = m[\(([0-9A-F]+)\)];
        if( not exists $kids{ $tid } ) {
            my $pid = open my $fh, '|-', 'perl non-interleaved.pl' or 
+die $!;
            $kids{ $tid } = [ $pid, $fh ];
        }
        print { $kids{ $tid }[ 1 ] } $_;
    }

    for my $kid ( keys %kids ) {
        close $kid->[ 1 ];           ## close the pipe
        waitpid $kid->[ 0 ], 0;      ##wait for the kid to finish
        rename $log, "archive/$log"; ## move the file
    }
}
[download]

This way, you don't have to accumulate lots of lines in memory in order to get complete transactions. The same simplified non-interleaved.pl deals with each threads data, and deals with just one transaction at a time.

Individual transactions should probably not be stored in an array, since there is no easy way to find the stored transaction data when you next see a log line about it, and need the info.
If you have unique identifiers for each transaction (which you must), and they are not simple small sequential integers (which I'm sure they aren't), then a hash is the way to go.

For example, you could scan through the log, and everytime you see an event for transaction $Tname, you reference $transactions{$Tname}, to find your notes on what you have seen about it so far.

A hash of hashes of (scalars with a few lists mixed in) would be what I expect. Something like:

$transactions = {
   transactionOne => { startTime => 123,
                       userName => 'Bob',
                       endTime => undef,
                       foosMunged => [1, 5, 99],
                     }
   transactionTwo => { ... }
                };
[download]

Fill in the transaction data as you see it, and once you see the end of a transaction, you can do whatever you need to do with the data you've collected, then delete the hash entry or leave it for later (Given huge files, deleting the old info is probably best). By the time one transaction ends, a few others will be partially populated, but stored safely out of the way behind their own ID keys.

I would try to process the log files in a single pass. You don't give a sufficient sample of your log files to be certain, but perhaps something like the following would work.

use strict;
use warnings;

my %transaction;

foreach my $file (<*.log>) {
    open(my $fh, '<', $file) or die "$file: $!";
    while(<$fh>) {
        chomp;
        if(/(.*?) \(([^\)]+)\)(.*)/) {
            my ($timestamp, $id, $log) = ($1, $2, $3);
            push(@{$transaction{$id}}, [ $timestamp, $id, $log, $file,
+ $. ]);
            my $status = status($transaction{$id});
            if($status eq 'complete') {
                submit($transaction{$id});
                delete($transaction{$id});
            } elsif($status eq 'corrupt') {
                report($status, $transaction{$id});
                delete($transaction{$id});
            }
        } else {
            warn "invalid input line: $_";
        }

    }
    close($fh);
}

foreach my $id (keys %transaction) {
    report('incomplete', $transaction{$id});
}
exit(0);



sub status {
    my $transaction = shift;

    return('incomplete') if(@$transaction < 3);
    return('corrupt') if(
        @$transaction > 3 or
        $transaction->[0][2] !~ /^(Authentication Request|Authenticati
+on Response|Accounting Request)/ or
        $transaction->[1][2] !~ /^Acct-Session-Id :/ or
        $transaction->[2][2] !~ /^User-Name :/
    );
    return('complete');
}

sub submit {
    my $transaction = shift;
    print "submit to database\n";
    foreach my $line (@$transaction) {
        my ($timestamp, $id, $log, $file, $line) = @$line;
        print "\t$file line $line: $timestamp ($id)$log\n";
    }

}

sub report {
    my ($status, $transaction) = @_;

    print STDERR "ERROR: $status transaction\n";
    foreach my $line (@$transaction) {
        my ($timestamp, $id, $log, $file, $line) = @$line;
        print STDERR "\t$file line $line: $timestamp ($id)$log\n";
    }
}
[download]

If you can identify the end of a transaction, and each thread only works on one transaction at a time, you won't have a problem.

Use the thread ID as the hash key, and delete that entry once the transaction is complete and you have done whatever you need to do with the data. That way, you have a nice clean place to start building up info about the thread's next transaction.

The key things:

• Whatever scheme you use to identify a transaction when doing multiple passes of the file, use that as your top level hash key.
• Whatever local variables you used to use to store info about the transaction, use those names as your second level hash keys
• Just keep reading log files in order and don't worry; the inner code should not care which file the lines are coming from, just that it has a stream of lines to analyse.

foreach $file (@files)
{
  open my $FH, '<', $file or die "$file open says $!";
  while (<$FH>)
  {
    ponder($_);
    $transactionHash{$threadID}{oldVariableName} = $stuff;
    delete $transactionHash{$threadID} if $transactionEnded;
  }
}
[download]

As someone who's recently had to solve this (well, slightly different: I didn't have to do real-time) problem at work: Reading over and over is going to be slow. Probably slower than what you want to deal with.

To actually get decent speed I finally moved the assembly of transactions into the database: I insert each line as I read it, letting the database record fill in as it gets information. (Note that you need some way to know which record each line is going to be in, and identify it uniquely over the course of at least a log file. Depending on the format of the logs, it may be worth keeping track of just the 'active' transactions.)

Doing this allowed me to process 1-1.5GB of a data a day in a couple of hours. (Despite having to use a temp database, since the DBD install for the main database is borked beyond repair.)

I've done similar things, and I find it best to make a single pass through the file. To do so, here's the approach I take:

First, I use a hash to contain all current transactions. (I'm assuming that each thread is handling only one task at a time, so within a thread you're not getting multiple transactions intermingled.) In this case, I'd use the thread ID as the key.

Next, read each line. You're going to find that the line is one of:

1. The start of a new transaction. Just emit the current transaction for the thread (if any) and start collecting the data for the new transaction.
2. The end of a transaction (some transactions will have recognizeable ends). Emit the transaction and delete the data from the hash.
3. Additional data for the transaction. Add the appropriate information to the hash.
4. An explicitly ignored line (comments, blank lines, information you're not collecting, etc.)
5. An unrecognized line, in which case you would print a warning or similar action if you care.
6. An unexpected line, i.e., you recognize it but it's unexpected at this time. (Such as a transaction end before you get a transaction start for the thread.)

...roboticus

Update: And if I had read the entire thread, I would've noticed that ig had already given an example of how to do this. Ah, well, it happens when you don't get enough sleep. I also added the <readmore> tags, as the post was a bit longish.

Wow, this is so helpful! Many thanks! Is it actually passing an entire transaction hash to the subroutine? Would I do the DB insert in the complete_xxx($hr) subroutines? Should I store all the complete transactions and then insert them all, or should I constantly be having it insert transactions? Again, thank you for your help.

tzen:

Is it actually passing an entire transaction hash to the subroutine?

Kind of. It's passing a reference to an entire transaction hash to the subroutine. When we refer to the $$hr{type} value, it first looks up the address of the hash with the innermost $hr part. Then it looks up the type member of the hash that $hr references.

Would I do the DB insert in the complete_xxx($hr) subroutines?

I did in my project, but you certainly don't have to. You could build a reformatted file that you can load into your database with a bulk-loading tool, like BCP (in MS SQL).

Should I store all the complete transactions and then insert them all, or should I constantly be having it insert transactions?

You can do it either way. I tend to insert them as I go because some of the files I work with are large, and I don't have enough RAM to hold it all in memory. But if all your data will fit, you can do it that way if it's easier.

...roboticus