I was observing some performance issues on a 'simple' file parsing program on Linux. It seemed to be using a very high amount of memory and then getting killed. As I investigated further, two observations:

- Reading a 640 MB file into an array causes perl/OS to allocate 4.5GB in memory. Seems very high!

- Reading the same file into the same array, but via reference, makes it jump to over 6GB.

The first is too high as it is but the second is even more confounding. Why woudl just using a reference cause a memory use jump by 33%! I used Memory::Usage to dump the mem use (validated against 'top'). Ideally my 640MB file would only use 640MB ... I must be doing something obviously stupid here but can't figure it out.

#!/usr/bin/perl -w
use Memory::Usage;
use vars qw ($muse);
$muse = Memory::Usage->new();

my $name = "bigfile"; # file size: 640MB
my $READIN = qw (<);
my $EXT;
my @data = ();
my $d = \@data;
die "Error opening file: $name" if ( !open $EXT, $READIN, $name );
$muse->record('Begin');

@data = <$EXT>; # option 1
#@$d = <$EXT>;  # option 2

close($EXT);
$muse->record('After file read');
$muse->dump();

[download]
Test result - option 1 read into an array 
$ perl test_read.pl
  time    vsz (  diff)    rss (  diff) shared (  diff)   code (  diff)
+   data (  diff) 
     0  78300 ( 78300)   2120 (  2120)   1412 (  1412)     12 (    12)
+   1164 (  1164) Begin 
    12  4538324 ( 4460024)  4373332 ( 4371212)   1428 (    16)     12 
+(     0)  4461188 ( 4460024) After file read

[download]
Test result - option 2 read into a reference to an array 
$ perl test_read.pl
  time    vsz (  diff)    rss (  diff) shared (  diff)   code (  diff)
+   data (  diff) 
     0  78300 ( 78300)   2116 (  2116)   1412 (  1412)     12 (    12)
+   1164 (  1164) Begin 
    17  6461328 ( 6383028)  6296400 ( 6294284)   1428 (    16)     12 
+(     0)  6384192 ( 6383028) After file read

[download]

In reply to memory use array vs ref to array by dkhosla1

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.