Re: Untangling Log Files

You shouldn't slurp files unless you actually need all of the file in memory at the same time. Your normal action should be to process the file a record at a line.

I'd do something like this:

my %fh; # store handles of new log files

open OLDLOG, $path_to_old_log or die $!;

while (<OLDLOG>) {
  my $proc = extract_process_id_from_old_log_record($_);

  unless (exists $fh{$proc}) {
    open $fh{$proc}, '>', "$proc.log" or die $!;
  }

  print $fh{$proc} $_;
}
[download]

Update: Reread the first line. It made no sense, so I fixed it.

<http://dave.org.uk>

"The first rule of Perl club is you do not talk about Perl club." -- Chip Salzenberg

Comment on Re: Untangling Log Files Download Code

Replies are listed 'Best First'.
Re^2: Untangling Log Files by sauoq (Abbot) on Feb 08, 2007 at 16:03 UTC
`open $fh{$proc}, '>', "$proc.log" or die $!;` Uhm... I don't care for that. It's very likely that there will be more pids in each file than open descriptors permitted by resource limits... and if you fail to open you die, probably half-done and with no way to pick up where you left off. I don't have an immediate fix though someone else suggested closing one handle randomly which would, I guess, work. (So long as you changed your open to '>>' and remembered to delete it from your hash.) Personally, I'd probably take a less elegant—call it more braindead—approach as this seems to be a one-off thing anyway, and just close the filehandle and open a new one whenever the pid changed from the previous record. Update: Well, I just re-read the OP and now I think I may have misinterpreted the bit about "30 processes" the first time around. If there are only 30 pids in the log files, then I like your approach just fine and my criticisms are all moot. `-sauoq "My two cents aren't worth a dime.";`	[reply] [d/l]
Re^3: Untangling Log Files by davorg (Chancellor) on Feb 08, 2007 at 16:08 UTC
Well, the original post said: Around 30 processes So I assumed I'd need about thirty files. And, yes, `die`ing like that is probably a bad idea. This is really only a sketch of a suggested approach. The devil is, of course, in the details :-) -- <http://dave.org.uk> "The first rule of Perl club is you do not talk about Perl club." -- Chip Salzenberg	[reply]