in reply to Long list is long

the original lists are sorted
This suggests a potential optimization, which would greatly reduce memory requirements and completely eliminate the need to sort your output, but at the cost of requiring a large number of filehandles: Since all input files are sorted and you're always processing the unprocessed line which sorts earliest, the output will automatically be sorted, without ever needing to have more than one line per input file in memory concurrently.

If you aren't able to allocate enough file handles at once to process the entire list of input files in one go, this technique can also be used to do the work in multiple stages - divide the files into N groups, get the combined sums for each group, and then use those combined sums as your new set of sorted input files.

#!/usr/bin/env perl use strict; use warnings; use 5.010; use Array::Heap::PriorityQueue::String; my @file_names = qw( in1 in2 in3 ); my @fh; my $heap = Array::Heap::PriorityQueue::String->new; for (0 .. $#file_names) { my $fn = $file_names[$_]; open my $new_fh, '<', $fn or die "Unable to open $fn: $!\n"; $fh[$_] = $new_fh; add_from_fh_id($_); } my ($current_word) = split /\s+/, $heap->peek; my $total = 0; while (my $item = $heap->get) { my ($word, $count, $fh_id) = split /\s+/, $item; if ($word eq $current_word) { $total += $count; } else { say "$current_word\t$total"; $current_word = $word; $total = $count; } add_from_fh_id($fh_id); } say "$current_word\t$total"; sub add_from_fh_id { my $fh_id = shift; my $read_from = $fh[$fh_id]; my $next_line = <$read_from>; if ($next_line) { chomp $next_line; $heap->add("$next_line $fh_id"); } }
For my testing, I used the input files from kcott's reply, sorted and renamed to 'in1', 'in2', and 'in3'. This gave me the output:
bar 70 blah 69 foo 146 life 42 word 81 yada 3

Replies are listed 'Best First'.
Re^2: Long list is long
by hv (Prior) on Oct 31, 2022 at 17:12 UTC

    It is not fully spelled out, but I think it likely that "the original lists are sorted" was intended by the OP to mean that "the lines are sorted by value" (as the output is required to be), not that the words appear in ASCII order.

    This is additionally hinted at by the initial 4-line example input.

      *facepalm* Good catch. I was so busy being clever that I forgot to read the specs properly. Thanks for pointing it out!