in reply to mem usage

Perl data structures (see PerlGuts Illustrated) need more memory than the mere user data they hold.

A quick check shows that reading 1_000_000 empty lines into an array (the way you do it) leads to a process size of 192MB on my machine (with Perl 5.10.1).  So if you have 11_000_000 non-empty ones...

Update: using ikegami's suggestion, the memory usage for the same 1_000_000 empty lines reduces to 78MB without pre-extending, and (interestingly) 86MB with pre-extending the array ($#lines = 1_000_000).   Update2: and 93MB with just $#lines = 100_000 (??)

Update3: with @lines = (), pre-extending the array no longer increases the memory requirements. But it doesn't help to reduce it either (presumably, it just improves speed).

Replies are listed 'Best First'.
Re^2: mem usage
by ikegami (Patriarch) on May 26, 2010 at 18:12 UTC
    There was a bug causing the array to have twice the desired number of elements. I don't know if you were affected by it.
[reply]
Re^2: mem usage
by halfcountplus (Hermit) on May 26, 2010 at 18:05 UTC
    Perl data structures (see PerlGuts Illustrated) need more memory than the mere user data they hold.

    Will read. I presumed they used a bit more space, say a few pointers per element for an array -- but that is really really quite a lot.

    I've also read somewhere that perl allocates in "power of 2" chunks. So when you add a 2nd element, you get allocated space for 4, then at 5 you get 8, at 9 you get 16 and so on. Is this true?
[reply]

      Close. When it grows, the array size doubles in size plus 4. An array with 1000 scalars in it could take up as much as 

          Array overhead
  + ( 999*2+4 ) * size of a pointer
  + 1000 * size of the scalars

      [download]
[reply]
[d/l]

In Section Seekers of Perl Wisdom