You shouldn't slurp files unless you actually need all of the file in memory at the same time. Your normal action should be to process the file a record at a line.

I'd do something like this:

my %fh; # store handles of new log files

open OLDLOG, $path_to_old_log or die $!;

while (<OLDLOG>) {
  my $proc = extract_process_id_from_old_log_record($_);

  unless (exists $fh{$proc}) {
    open $fh{$proc}, '>', "$proc.log" or die $!;
  }

  print $fh{$proc} $_;
}
[download]

Update: Reread the first line. It made no sense, so I fixed it.

open $fh{$proc}, '>', "$proc.log" or die $!;

Uhm... I don't care for that. It's very likely that there will be more pids in each file than open descriptors permitted by resource limits... and if you fail to open you die, probably half-done and with no way to pick up where you left off.

I don't have an immediate fix though someone else suggested closing one handle randomly which would, I guess, work. (So long as you changed your open to '>>' and remembered to delete it from your hash.)

Personally, I'd probably take a less elegant—call it more braindead—approach as this seems to be a one-off thing anyway, and just close the filehandle and open a new one whenever the pid changed from the previous record.

Update: Well, I just re-read the OP and now I think I may have misinterpreted the bit about "30 processes" the first time around. If there are only 30 pids in the log files, then I like your approach just fine and my criticisms are all moot.

-sauoq
"My two cents aren't worth a dime.";

Well, the original post said:

Around 30 processes

So I assumed I'd need about thirty files.

And, yes, dieing like that is probably a bad idea. This is really only a sketch of a suggested approach. The devil is, of course, in the details :-)

--


<http://dave.org.uk>

"The first rule of Perl club is you do not talk about Perl club." -- Chip Salzenberg

Personally, I would keep a hash of filehandles, but close one at random if there are too many open (hoping the pids are kinda grouped together in the logfile).

++jettero

FYI, the core module FileCache will automate the close/re-open of filehandles.

I like the problem, too, and I certainly like your solution. Randomly closing file handles seems like an nice way of making it scalable.

Thanks,

loris

---

"It took Loris ten minutes to eat a satsuma . . . twenty minutes to get from one end of his branch to the other . . . and an hour to scratch his bottom. But Slow Loris didn't care. He had a secret . . ." (from "Slow Loris" by Alexis Deacon)

To allow for reboot of the system, the daemon will need to keep track of the timestamp of the last message it collated for each machine writing messages to the logfiles (in case their clocks are out of synch.)

There also needs to be a structure of regular expressions that enables not just identification of the originating process but of the timestamp which needs to be converted into a delta time for comparison. In a dynamic environment this might best be achieved using a csv configuration file e.g.:

PROCESS,HOST,LOGFILE,FORMAT,$1,$2
foo,host99,/var/adm/foo.log,\s+\S+\s+(\S+)\s+(\d+\-\d+\-\d+\s\d+:\d+:\
+d+:\s\w{2}),PROC,TIMESTAMP
[download]

It also needs to sleep perhaps five minutes between cycles through all the log files to free system resources for other processes.

Update: a common practice is also to routinely archive and delete logfiles (yet another logfile management daemon!) so that such reprocessing doesn't have to start from the beginning of a very large logfile, and then have to read but ignore millions of entries occurring before the last recorded timestamp. One system I work with regularly archives logfiles when they hit 5 MB instead of by time or line count. It might be convenient for your requirement if the message-collating daemon could also (per cycle) check the size and conditionally do or invoke that archiving itself.

grep -h process_identifier *.log | sort -options > process_identifier.
+newlog
[download]

Your suggestion is pretty much what I am doing already, but I want to automate the process a bit more so that I can generate the all the process-specific log files at once. So I think I shall try something like jettero's solution.

Thanks anyway,

loris

---

"It took Loris ten minutes to eat a satsuma . . . twenty minutes to get from one end of his branch to the other . . . and an hour to scratch his bottom. But Slow Loris didn't care. He had a secret . . ." (from "Slow Loris" by Alexis Deacon)

I see. Well, in "quick and dirty" style, you could wrap that system call in a foreach my $process ('this','that','the other') { ... } loop.

---
It's all fine and dandy until someone has to look at the code.