in reply to Long list is long
the original lists are sortedThis suggests a potential optimization, which would greatly reduce memory requirements and completely eliminate the need to sort your output, but at the cost of requiring a large number of filehandles:
If you aren't able to allocate enough file handles at once to process the entire list of input files in one go, this technique can also be used to do the work in multiple stages - divide the files into N groups, get the combined sums for each group, and then use those combined sums as your new set of sorted input files.
For my testing, I used the input files from kcott's reply, sorted and renamed to 'in1', 'in2', and 'in3'. This gave me the output:#!/usr/bin/env perl use strict; use warnings; use 5.010; use Array::Heap::PriorityQueue::String; my @file_names = qw( in1 in2 in3 ); my @fh; my $heap = Array::Heap::PriorityQueue::String->new; for (0 .. $#file_names) { my $fn = $file_names[$_]; open my $new_fh, '<', $fn or die "Unable to open $fn: $!\n"; $fh[$_] = $new_fh; add_from_fh_id($_); } my ($current_word) = split /\s+/, $heap->peek; my $total = 0; while (my $item = $heap->get) { my ($word, $count, $fh_id) = split /\s+/, $item; if ($word eq $current_word) { $total += $count; } else { say "$current_word\t$total"; $current_word = $word; $total = $count; } add_from_fh_id($fh_id); } say "$current_word\t$total"; sub add_from_fh_id { my $fh_id = shift; my $read_from = $fh[$fh_id]; my $next_line = <$read_from>; if ($next_line) { chomp $next_line; $heap->add("$next_line $fh_id"); } }
bar 70 blah 69 foo 146 life 42 word 81 yada 3
|
---|
Replies are listed 'Best First'. | |
---|---|
Re^2: Long list is long
by hv (Prior) on Oct 31, 2022 at 17:12 UTC | |
by dsheroh (Monsignor) on Nov 01, 2022 at 08:33 UTC |