I thought of that, but I recall reading somewhere that if you allocate a big chunk of memory in perl and then release it (by way of undef'ing the variable or whatever) that memory is not returned to the operating system -- it's still owned by the perl interpreter as far as Linux is concerned. So you can write %dataset to disk and undef it, but w/rt children, forking, and copy-on-write, the operating system will treat that memory as if it's still in use. Thus, you gain nothing.

shadrack,

Interesting, I just checked on AIX and the memory was freed for the child, but on Linux(Debian) the memory was still allocated for the child. So it depends on the OS.

You could try this program on your system and see if the memory is freed or not? Note: My test results are at the bottom of the script.

#!/usr/local/bin/perl

    use strict;
    use warnings;

    my $NAME = "";

    my ( $FB_CacheMemoryVSZ, $FB_CacheMemoryRSS ) = &Display_Mem_Usage
+($$,$NAME,0);
    print "Before: Virtual Memory: $FB_CacheMemoryVSZ\tReal Memory: $F
+B_CacheMemoryRSS\n";
    {    my %RCache;
        for my $key ( 1_000 .. 11_000 )
        {    $RCache{$key} = chr( $key % 255 ) x 2_048;
        }
        ( $FB_CacheMemoryVSZ, $FB_CacheMemoryRSS ) = &Display_Mem_Usag
+e($$,$NAME,0);
        print "After:  Virtual Memory: $FB_CacheMemoryVSZ\tReal Memory
+: $FB_CacheMemoryRSS\n";
    }
    ( $FB_CacheMemoryVSZ, $FB_CacheMemoryRSS ) = &Display_Mem_Usage($$
+,$NAME,0);
    print "$$: Virtual Memory: $FB_CacheMemoryVSZ\tReal Memory: $FB_Ca
+cheMemoryRSS\n\n";

    my $fork = fork();
    if ( $fork )
    {
        ( $FB_CacheMemoryVSZ, $FB_CacheMemoryRSS ) = &Display_Mem_Usag
+e($$,$NAME,0);
        print "$$: Virtual Memory: $FB_CacheMemoryVSZ\tReal Memory: $F
+B_CacheMemoryRSS\n\n";
    }
    else
    {    ( $FB_CacheMemoryVSZ, $FB_CacheMemoryRSS ) = &Display_Mem_Usa
+ge($$,$NAME,0);
        print "$$: Virtual Memory: $FB_CacheMemoryVSZ\tReal Memory: $F
+B_CacheMemoryRSS\n\n";
    }

    exit;

sub Display_Mem_Usage
{    # VSZ is size in KBytes of the virtual memory ( VSZ * 1024 )
    # RSS is size in pages of real memory ( 1024 * RSS )
    my $cpid = shift; my $name = shift; my $from = shift;
    my $var = ""; my $fh;

    my $arg = qq| -o "vsz rssize" -p $cpid|;
      open ( $fh, "-|", "/bin/ps $arg" ) or die "Display_Mem_Usage: No
+t open \'$arg\': $!";
    while (<$fh>)
    { $var .= $_; }
    close $fh;
    my $rno = my @ref = split(/\n/,$var); if ( $rno < 2 ) { return ( -
+1, -1 ); }
    my $info = join(" ", split " ", $ref[1]);
    my ($vmem,$rmem) = ( split(/\ /,$info) );
    return ( $vmem , $rmem );
 }


__END__

   AIX:#  perl ckmem.cgi    
   Before: Virtual Memory: 776     Real Memory: 852
   After:  Virtual Memory: 21904   Real Memory: 21980
   41030: Virtual Memory: 21908    Real Memory: 21984
   
   40244: Virtual Memory: 424      Real Memory: 404   ## Child
   
   41030: Virtual Memory: 21912    Real Memory: 21900
   
   AIX:# 
   
   
   Linux:# perl ckmem.cgi
   Before: Virtual Memory: 5288    Real Memory: 1928
   After:  Virtual Memory: 26136   Real Memory: 22804
   26362: Virtual Memory: 26128    Real Memory: 22804
   
   26362: Virtual Memory: 26128    Real Memory: 22808
   
   26369: Virtual Memory: 26128    Real Memory: 21600   ## Child
   
   Linux:#
[download]

Regards...Ed

"Well done is better than well said." - Benjamin Franklin

Well, I'm on Linux, so the results are no surprise:

Before: Virtual Memory: 122236  Real Memory: 2416
After:  Virtual Memory: 144476  Real Memory: 24640
29118: Virtual Memory: 144476   Real Memory: 24640
29118: Virtual Memory: 144476   Real Memory: 24644
29122: Virtual Memory: 144476   Real Memory: 23160
[download]