in reply to Re: Howto Avoid Memory Problem in List::MoreUtils
in thread Howto Avoid Memory Problem in List::MoreUtils

this is optimized by perl as a sort in place operation

In which version of Perl did that become true?

Unless I'm doing it wrong, it doesn't seem to be true in 5.8.8?


Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal?
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.
  • Comment on Re^2: Howto Avoid Memory Problem in List::MoreUtils

Replies are listed 'Best First'.
Re^3: Howto Avoid Memory Problem in List::MoreUtils
by dave_the_m (Monsignor) on May 04, 2006 at 10:39 UTC
    n which version of Perl did that become true?
    Since 5.8.4.
    $ perl583 -le '@a=(1,2); $r1 = \$a[0]; @a=sort@a; $r2 = \$a[0];print " +yes"if $r1 == $r2' $ perl584 -le '@a=(1,2); $r1 = \$a[0]; @a=sort@a; $r2 = \$a[0];print " +yes"if $r1 == $r2' yes $

    Dave.

Re^3: Howto Avoid Memory Problem in List::MoreUtils
by salva (Canon) on May 04, 2006 at 11:07 UTC
    well, you are right, I forgot that mergesort needs additional memory to perform the merge, but yet @a = sort @a uses half the memory required by other sorting expressions, for instance:
    use Memchmark 'cmpthese'; # use sort '_quicksort'; cmpthese( none => sub { my @a = map { int rand 100 } 1..1000000; }, sort0 => sub { my @a = map { int rand 100 } 1..1000000; @a = + sort { $a <=> $b } @a }, sort1 => sub { my @a = map { int rand 100 } 1..1000000; @a = + sort { $a <=> $b } @a, 1 } ); __END__ test: none, memory used: 3997696 bytes test: sort0, memory used: 7999488 bytes test: sort1, memory used: 12390400 bytes
    And if the quicksort algorithm is selected, then the sentence on my previous post becomes really true:
    use sort '_quicksort'; ... __END__ test: none, memory used: 3997696 bytes test: sort0, memory used: 3997696 bytes test: sort1, memory used: 12390400 bytes

      Hmm. I guess we have different understandings of the term In-place sort.

      I was trying to sort (using the default algorithm), 20e6 elements which takes 400 MB, but it was consuming 1.4 GB gross (some memory has been return to the system by the time the second memcheck runs below).

      C:\test>junk Mem after building array: 404,556 kb Mem after sorting array: 1,214,032 kb

      Having read your post, I tried the _quickersort option. Things improve marginally, but:

      C:\test>junk Mem after building array: 404,812 kb Mem after sorting array: 973,584 kb

      But that still requires a gross memory usage of 1.3 GB. 2N additional memory is somewhat greater than logN.

      A useful feature, and nice to know it's there, but not quite what I was expecting hoping for when you said in-place.


      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal?
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.
        I guess we have different understandings of the term In-place sort.

        Yes, inside perl internals, "inplace" just means that instead of passing the elements on the stack, a reference to the target array is passed and the sorting is done directly over it.

        ... but if the quicksort algorithm is selected, @a = sort @a is also an in-place sort as described on the article pointed by you (if we ignore degenerated cases).

        If you need 1.4GB to sort 400MB, it means that you are probably converting numbers to strings inside the sort. For instance:

        use Memchmark 'cmpthese'; use sort '_quicksort'; cmpthese( none => sub { my @a = map { int rand 100 } 1..1000000; }, sort_num => sub { my @a = map { int rand 100 } 1..1000000; @a = sort { $a<=> $b } @a }, sort_str => sub { my @a = map { int rand 100 } 1..1000000; @a = sort { $a cmp $b } @a }, sort_str_workaround => sub { my @a = map { int rand 100 } 1..1000000; my ($c, $d); @a = sort { ($c, $d) = ($a,$b); $c cmp $d } @a } ); __END__ test: none, memory used: 3997696 bytes test: sort_num, memory used: 3997696 bytes test: sort_str, memory used: 68202496 bytes test: sort_str_workaround, memory used: 3997696 bytes
        ... and as you can see, there is a simple workaround (though it would make the sorting much slower).