in reply to Re: out of memory problem after undef
in thread out of memory problem after undef

Because the second structure uses more memory (to build if not when complete) than the first.

Not so!

The AoH with 50 hashes each with 250_000 key/value pairs requires close to 1 GB, when finished and over 1.5GB to construct. Whereas the array of 10e6 integers requires just 220MB, and a peak of 330MB to construct.

The problem is that none of the memory freed up by releasing the AoH is large enough to be reused for the AV for the array, because Perl doesn't defragment or coalese freed memory. So, even though Perl has gobs of free space available to it, it has to go back to the OS for more in order to get big enough chunks for the array.


Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.
"Too many [] have been sedated by an oppressive environment of political correctness and risk aversion."
  • Comment on Re^2: out of memory problem after undef

Replies are listed 'Best First'.
Re^3: out of memory problem after undef
by ikegami (Patriarch) on Dec 03, 2008 at 22:12 UTC

    Oops, I missed the outer 1..50 loop!

    I wonder how hard it would be to provide a function that defragments the heap, and what would be the downsides of calling it when large blocks are needed from the OS.

      I had similar thoughts. This is flying in the dark a bit as I've no real idea if the CRT malloc uses the process heap or another heap or heaps. I tried using the following apis to try and compact the heap after the AoH is undef'd:

      use Win32::API::Prototype; ApiLink( 'Kernel32', q[ HANDLE GetProcessHeap(void) ] ) or die $^E; ApiLink( 'kernel32', q[ DWORD HeapCompact( HANDLE hHeap, DWORD dwFlags + ) ] ) or die $^E; my @idsMap; my @AoH; my $cnt = 1; $|++; for my $i ( 1 .. 40 ) { my $start = "aaa"; my %hash; keys %hash = 250_000; printf "\r$i\t"; for my $j ( 1 .. 250000 ) { $hash{ "$cnt $i $j $start" } = $cnt; $cnt++; $start++; } $AoH[$i] = \%hash; } print "Total size: ", total_size( \@AoH ); undef @AoH; print "did undef of array\n"; <>; print HeapCompact( GetProcessHeap( 0 ), 0 ) or die $^E; ...

      It does something, and takes quite a while doing it, but the result is that the largest contiguous chunk after compaction (of the process heap) is a little over 1MB which isn't enough to make any difference.

      Looking at the process memory map, there appear to be multiple heaps, but I haven't worked out a way of determining which is the process heap.

      One possibility is to enumerate all the heaps and run HeapCompact() on them all. Besides being slow, I'm not sure that it would achieve very much?


      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.