in reply to modification of the script to consume less memory with higher speed

Greetings, Anonymous Monk.

Sorting may be the reason why the script is failing, but am not sure at which point the script hangs on your machine. Have you tried not sorting the hash?

#!/usr/bin/env perl use strict; use warnings; no warnings qw( numeric ); my $file_count = @ARGV; my %seen; $/ = ""; if ( @ARGV == 0 ) { print "usage: perl $0 file1.txt file2.txt ...\n"; exit 1; } while ( my $key = <> ) { chomp $key; # obtain "\t" position my $tabPos = rindex $key, "\t"; # extract value my $value = substr $key, $tabPos + 1; # obtain key1 my @lines = split /\n/, $key, 3; my $key1 = $lines[1]; $seen{$key1} //= do { # trim "\t" and value from key substr $key, $tabPos, length($value) + 1, ''; [ $key ]; }; push @{ $seen{$key1} }, $value; } my $tot; # sorting requires 2x memory allocation and possibly # exhaust available memory, hang, or crash # foreach my $key1 ( sort keys %seen ) { # if ( @{ $seen{$key1} } >= $file_count ) { # $tot = 0; # for my $val ( @{ $seen{$key1} } ) { # # $tot += ( split /:/, $val )[0]; # $tot += $val; # Perl ignores the string after number # } # print join "\t", @{ $seen{$key1} }; # print "\tcount:". $tot."\n\n"; # } # } # try this instead for lesser memory consumption while ( my ( $key1, $aref ) = each %seen ) { if ( @{ $aref } >= $file_count ) { $tot = 0; for my $val ( @{ $aref } ) { # $tot += ( split /:/, $val )[0]; $tot += $val; # Perl ignores the string after number } print join "\t", @{ $aref }; print "\tcount:". $tot."\n\n"; } }

Regards.

  • Comment on Re: modification of the script to consume less memory with higher speed
  • Download Code

Replies are listed 'Best First'.
Re^2: modification of the script to consume less memory with higher speed
by Anonymous Monk on Aug 02, 2016 at 06:39 UTC

    For this particular use case, I saw lesser memory consumption using the following for output. Your mileage may vary. This is without sorting.

    my $tot; foreach my $key1 ( keys %seen ) { if ( @{ $seen{$key1} } >= $file_count ) { $tot = 0; for my $val ( @{ $seen{$key1} } ) { # $tot += ( split /:/, $val )[0]; $tot += $val; # Perl ignores the string after number } print join "\t", @{ $seen{$key1} }; print "\tcount:". $tot."\n\n"; } }

    Regards.