in reply to modification of the script to consume less memory with higher speed
Sorting may be the reason why the script is failing, but am not sure at which point the script hangs on your machine. Have you tried not sorting the hash?
#!/usr/bin/env perl use strict; use warnings; no warnings qw( numeric ); my $file_count = @ARGV; my %seen; $/ = ""; if ( @ARGV == 0 ) { print "usage: perl $0 file1.txt file2.txt ...\n"; exit 1; } while ( my $key = <> ) { chomp $key; # obtain "\t" position my $tabPos = rindex $key, "\t"; # extract value my $value = substr $key, $tabPos + 1; # obtain key1 my @lines = split /\n/, $key, 3; my $key1 = $lines[1]; $seen{$key1} //= do { # trim "\t" and value from key substr $key, $tabPos, length($value) + 1, ''; [ $key ]; }; push @{ $seen{$key1} }, $value; } my $tot; # sorting requires 2x memory allocation and possibly # exhaust available memory, hang, or crash # foreach my $key1 ( sort keys %seen ) { # if ( @{ $seen{$key1} } >= $file_count ) { # $tot = 0; # for my $val ( @{ $seen{$key1} } ) { # # $tot += ( split /:/, $val )[0]; # $tot += $val; # Perl ignores the string after number # } # print join "\t", @{ $seen{$key1} }; # print "\tcount:". $tot."\n\n"; # } # } # try this instead for lesser memory consumption while ( my ( $key1, $aref ) = each %seen ) { if ( @{ $aref } >= $file_count ) { $tot = 0; for my $val ( @{ $aref } ) { # $tot += ( split /:/, $val )[0]; $tot += $val; # Perl ignores the string after number } print join "\t", @{ $aref }; print "\tcount:". $tot."\n\n"; } }
Regards.
|
|---|
| Replies are listed 'Best First'. | |
|---|---|
|
Re^2: modification of the script to consume less memory with higher speed
by Anonymous Monk on Aug 02, 2016 at 06:39 UTC |