in reply to mem usage

Maybe fragmentation from extending the stack is using up a lot of memory. Try to avoid putting the entire file on the stack, and pre-extending its destination: 

my @lines;
$#lines = 12_000_000;
@lines = ();
push @lines, $_ while <IN>;

[download]

Note that shuffle already exists in List::Util.

Update: Fixed bug.

Replies are listed 'Best First'.
Re^2: mem usage
by halfcountplus (Hermit) on May 26, 2010 at 18:02 UTC
    Thanks. It just barely squeezed thru -- interestingly the kernel killed firefox during the run.
[reply]
      Woops, what I posted is very buggy. Should be 
      my @lines;
$#lines = 12_000_000;
@lines = ();                 <------ was missing
push @lines, $_ while <IN>;

      [download]
      $ perl -e'print "abcdef\n" for 1..11_000_000' |
   perl -E'
      $#a=11_000_000;
      @a=();
      push @a, $_ while <>;
      say int(`ps --no-heading -o vsz $$`/1000)
'
480

      [download]

      480MB for 77MB file with 11 million lines.

[reply]
[d/l]
[select]
        Hmmm, yeah that's much better. What's the significance of this: 
              @a=();

        [download]
        After presizing -- I would have thought that would just clear the array?
[reply]
[d/l]

In Section Seekers of Perl Wisdom