in reply to Interlaced log parser
I think that I would separate the concerns by prefiltering the files and feeding each thread's lines to different processes via pipes:
#! perl use strict; #### Warning! Untested code -- just an example. for my $log ( <*.log> ) { open my $fhLog, '<', $log or die $!; my %kids; while( <$fhLog> ) { my( $tid ) = m[\(([0-9A-F]+)\)]; if( not exists $kids{ $tid } ) { my $pid = open my $fh, '|-', 'perl non-interleaved.pl' or +die $!; $kids{ $tid } = [ $pid, $fh ]; } print { $kids{ $tid }[ 1 ] } $_; } for my $kid ( keys %kids ) { close $kid->[ 1 ]; ## close the pipe waitpid $kid->[ 0 ], 0; ##wait for the kid to finish rename $log, "archive/$log"; ## move the file } }
This way, you don't have to accumulate lots of lines in memory in order to get complete transactions. The same simplified non-interleaved.pl deals with each threads data, and deals with just one transaction at a time.
|
|---|