in reply to Re: How do I pre-allocate an array of hashes?
in thread How do I pre-allocate an array of hashes?

I think you missed my point. I was throwing out the idea that maybe that model would translate to Perl. There are three ways I've seen memory allocation affected in general:

  1. Use a different malloc algorithm. You can do this with Perl by building a different one into the perl binary.
  2. Configure an existing malloc interface if it supports it. Some C libraries had this when I coded C but it wasn't protable by any means. What they did is let you "profile" your base allocations (lots of small blocks, a few large blocks, etc.) by passing in values to a configuration API that primed the malloc algorithm. There is nothing equivalent to this in Perl that I'm aware of.
  3. Try controlling malloc at the application level. I mentioned that one way I used to see some people try this was to figure out the overall allocation size used by the application, then allcoate this as one huge chunk and free it. There really is no direct equivalent in Perl, but I was thinking along these lines for this particular example:
    use strict; my $i = 120; my $n = 250000; my $x = {}; my @a; my $j; $#a = $n; # Up front, big malloc/free ... Does it make a difference? #keys %$x = ($i * $n); #$x = {}; while ($n-- > 0) { $x = {}; keys %$x = $i; for ($j = 0; $j < $i; $j++) { $x->{$j} = $j; } push(@a, $x); }
    You'd run this with and without the big malloc/free and see if there's any difference. The big leap here is that these are not straight (flat) memory allocations in Perl -- you're also (I'd think) building up data structures (hashes, arrays). I did not dive into the code so I may be way off base here. But I did try this test just for kicks on Solaris 2.7. The big malloc/free came out a lot worse, so that theory can pretty much be buried ... at least on that platform. Here's a baseline difference. The runs take forever so I didn't get a good sample set. Kind of pointless I think.

    Times with the big malloc/free:

    Total Elapsed Time = 947.1299 Seconds User+System Time = 383.5699 Seconds
    Times without it:
    Total Elapsed Time = 497.0599 Seconds User+System Time = 376.4799 Seconds