Since the log files are sorted, there's no reason to have more than two handles open.

while (my $logfile = glob('access-????.log')) {
   open(my $logfile_fh, '<', $logfile)
      or die("Unable to open yearly logfile $logfile: $!\n");

   my ($year) = $logfile =~ /^access-(....)\.log$/;

   my $output_fh;
   my $month = -1;

   while (<$logfile_fh>) {
      # Extract month of log entry.
      my ($new_month) = /$year-(..)/;

      if ($new_month != $month) {
         $month = $new_month;
         my $output_file = sprintf('access-%04d-%02d.log', $year, $mon
+th);
         open($output_fh, '>', $output_file)
            or die("Unable to create monthly logfile $output_file: $!\
+n");
      }

      print $output_fh $_;
   }
}
[download]

Sadly, the files aren't perfectly sorted. At the month changeover, there are dozens of lines out of order. I assume that the server wrote the lines in batches, and didn't bother with FIFO.

However, the bulk of each month is ordered properly. I think trying it your way would be fine -- even if it has to close/open/reopen files 20 times during the change of each month, that's still probably A-OK. Overall, that's just maybe 750 open/closes. Oh. Maybe that is a bit much. Hmm.

Well, it's something to test out, to see how speedy it is. Thank you. :)

For any halfway decent modern OS (or even Windows for that matter . . .) 36 open files should be no problem. The size of the file itself doesn't enter into the equation since all filehandles are underneath is (more or less) a few pointers; the storage requirements for one are miniscule. Granted if you were talking about several more orders of magnitude (thousands of handles) you might run into problems and need to look at something like FileCache, but for this many it shouldn't be a problem.

Update: Oop I didn't even think about it (stayed up too late running SM 4-man last night . . . :), but as is pointed out below each file should already be sorted so you just read until the date changes and then open the new output file. Now if you had multiple files you wanted to merge (say from n separate webservers) into one consolidated log file you would need multiple open files for reading, but again just one output file would serve.

-sam