in reply to memory issues

The problem is that this line:

        if($ipairs{$fk}{$pair}{'pvalue'}){

[download]

Rather that just testing if that value exists, it is autovivifying (creating) that value in the nested hashes and setting it to null.

If you change that line to:

        if( exists $ipairs{$fk}
        &&  exists $ipairs{$fk}{$pair}
        &&  exists $ipairs{$fk}{$pair}{'pvalue'}
        ){

[download]

It should prevent the runaway memory growth. As a nice side-effect, your program should run substantially faster also.

BTW. I assume you mean 32GB not 32MB?

With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.

Replies are listed 'Best First'.
Re^2: memory issues
by asqwerty (Acolyte) on Jan 28, 2013 at 09:24 UTC
    Thanks!!!! It works fine now and, as you said, substantially faster also. This is what I did, 
    foreach my $pair (@lpairs) {
    my $n = 0;
    my $z = 0;
    my $hl;
    my $pvalue = 0;
    my $pvt = 0;
    my $fk;
    foreach $fk (%ipairs) {
        if( exists $ipairs{$fk} &&  exists $ipairs{$fk}{$pair} &&  exi
+sts $ipairs{$fk}{$pair}{'pvalue'}){
        #if($ipairs{$fk}{$pair}{'pvalue'}){
            $pvt = $ipairs{$fk}{$pair}{'pvalue'};
            if($pvt){
                unless($hl){
                    $hl = $ipairs{$fk}{$pair}{'head'};
                }
                $n++;
                $z+= qnorm($ipairs{$fk}{$pair}{'pvalue'});
            }
        }
    }
    if($n>2){
        $z = $z/sqrt($n);
        $pvalue = pnorm($z);
    }    
    if ($pvalue) {
        #printf "$pair -> %.4f\n", $pvalue; 
        printf OF "$hl %.4f $n\n", $pvalue;
    }

}

    [download]
[reply]
[d/l]

      Just a small hint to reduce the verboseness of your code:

          ($ipairs{$fk}{$dline}{'head'}, $ipairs{$fk}{$dline}{'effect'}, $ip
+airs{$fk}{$dline}{'pvalue'}) = $ldata{$dline} =~ /^(.*)\s+(\d\.\d+)\s
++\d\.\d+\s+(\d\.\d+)$/;

      [download]

      can be rewritten using a hash slice like this:

          ( @{ $ipairs{$fk}{$dline} }{qw/head effect pvalue/} ) = $ldata{$dl
+ine} =~ /^(.*)\s+(\d\.\d+)\s+\d\.\d+\s+(\d\.\d+)$/;

      [download]

      Otherwise, your code benefits from a temporary variable or two. Here I repurpose $pvt (not sure if the variable name makes sense after that):

          $pvt = $ipairs{$fk}{$pair};
    if($pvt->{'pvalue'}){
        unless($hl){
            $hl = $pvt->{'head'};
        }
        $n++;
        $z+= qnorm($pvt->{'pvalue'});
    }

      [download]

      (This only works because there already exists a hash reference at $ipairs{$fk}{$pair}. It would not work if you tried to say $pvt = {}, but %$pvt = () would. It's all reference magic and not really easy to explain unless you first understand pointers.)

      (Of course, the hash slice can be rewritten using a temporary variable, too. It's always a good idea to use temporary variables if it makes your code easier to understand. Triply a good idea if it reduces repetition.)

[reply]
[d/l]
[select]
Re^2: memory issues
by asqwerty (Acolyte) on Jan 28, 2013 at 09:25 UTC
    And you are right. It is 32 GB. :-)
[reply]

In Section Seekers of Perl Wisdom