G'day Chuma,
A number of problems with the code you posted:
From your description, I'd say the bottleneck lies with the population of the three arrays: @p, @q and @i. This is all unnecessary work and those arrays are not even needed. See "perlperf - Perl Performance and Optimization Techniques", for benchmarking and profiling techniques, to get a clearer picture of where problems lie.
I created these three dummy test files. In case you're unfamiliar with cat -vet, ^I represents a tab and $ represents a newline.
$ for i in A B C; do echo -e "\n*** $i"; cat $i; echo '----'; cat -ve +t $i; done *** A foo 73 bar 35 word 27 blah 23 ---- foo^I73$ bar^I35$ word^I27$ blah^I23$ *** B bar 35 yada 3 word 27 blah 23 ---- bar^I35$ yada^I3$ word^I27$ blah^I23$ *** C foo 73 word 27 blah 23 life 42 ---- foo^I73$ word^I27$ blah^I23$ life^I42$
Then this test code:
#!/usr/bin/env perl use strict; use warnings; use autodie; my @in_files = qw{A B C}; my $outfile = 'merge_count.out'; my %data; my $out_fmt = "%s\t%d\n"; for my $infile (@in_files) { open my $fh, '<', $infile; while (<$fh>) { my ($word, $count) = split; $data{$word} += $count; } } open my $fh, '>', $outfile; for my $key (sort { $data{$a} <=> $data{$b} } keys %data) { printf $fh $out_fmt, $key, $data{$key}; }
Output (raw and showing special characters):
$ cat merge_count.out yada 3 life 42 blah 69 bar 70 word 81 foo 146 $ cat -vet merge_count.out yada^I3$ life^I42$ blah^I69$ bar^I70$ word^I81$ foo^I146$
Try that with your real files. I suspect it should be faster and not have the bottlenecks. Let us know if you still have problems: show your new code and profiling output (in <readme> or <spoiler> tags).
— Ken
In reply to Re: Long list is long
by kcott
in thread Long list is long
by Chuma
| For: | Use: | ||
| & | & | ||
| < | < | ||
| > | > | ||
| [ | [ | ||
| ] | ] |