in reply to Re^2: Howto Avoid Memory Problem in List::MoreUtils
in thread Howto Avoid Memory Problem in List::MoreUtils

well, you are right, I forgot that mergesort needs additional memory to perform the merge, but yet @a = sort @a uses half the memory required by other sorting expressions, for instance: 
use Memchmark 'cmpthese';

# use sort '_quicksort';

cmpthese( none => sub { my @a = map { int rand 100 } 1..1000000; },
          sort0 => sub { my @a = map { int rand 100 } 1..1000000; @a =
+ sort { $a <=> $b } @a },
          sort1 => sub { my @a = map { int rand 100 } 1..1000000; @a =
+ sort { $a <=> $b } @a, 1 } );

__END__

test: none, memory used: 3997696 bytes
test: sort0, memory used: 7999488 bytes
test: sort1, memory used: 12390400 bytes

[download]
And if the quicksort algorithm is selected, then the sentence on my previous post becomes really true: 
use sort '_quicksort';
...

__END__

test: none, memory used: 3997696 bytes
test: sort0, memory used: 3997696 bytes
test: sort1, memory used: 12390400 bytes

[download]

Replies are listed 'Best First'.
Re^4: Howto Avoid Memory Problem in List::MoreUtils
by BrowserUk (Patriarch) on May 04, 2006 at 11:48 UTC

    Hmm. I guess we have different understandings of the term In-place sort.

    I was trying to sort (using the default algorithm), 20e6 elements which takes 400 MB, but it was consuming 1.4 GB gross (some memory has been return to the system by the time the second memcheck runs below). 

    C:\test>junk
Mem after building array: 404,556 kb
Mem after sorting array: 1,214,032 kb

    [download]

    Having read your post, I tried the _quickersort option. Things improve marginally, but: 

    C:\test>junk
Mem after building array: 404,812 kb
Mem after sorting array: 973,584 kb

    [download]

    But that still requires a gross memory usage of 1.3 GB. 2N additional memory is somewhat greater than logN.

    A useful feature, and nice to know it's there, but not quite what I was expecting hoping for when you said in-place.

    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal?
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
[d/l]
[select]
      I guess we have different understandings of the term In-place sort.

      Yes, inside perl internals, "inplace" just means that instead of passing the elements on the stack, a reference to the target array is passed and the sorting is done directly over it.

      ... but if the quicksort algorithm is selected, @a = sort @a is also an in-place sort as described on the article pointed by you (if we ignore degenerated cases).

      If you need 1.4GB to sort 400MB, it means that you are probably converting numbers to strings inside the sort. For instance: 

      use Memchmark 'cmpthese';

use sort '_quicksort';

cmpthese( none => sub {
             my @a = map { int rand 100 } 1..1000000;
          },
          sort_num => sub {
             my @a = map { int rand 100 } 1..1000000;
             @a = sort { $a<=> $b } @a
          },
          sort_str => sub {
             my @a = map { int rand 100 } 1..1000000;
             @a = sort { $a cmp $b } @a
          },
          sort_str_workaround => sub {
             my @a = map { int rand 100 } 1..1000000;
             my ($c, $d);
             @a = sort { ($c, $d) = ($a,$b); $c cmp $d } @a
          }
        );

__END__

test: none,                memory used:  3997696 bytes
test: sort_num,            memory used:  3997696 bytes
test: sort_str,            memory used: 68202496 bytes
test: sort_str_workaround, memory used:  3997696 bytes

      [download]
      ... and as you can see, there is a simple workaround (though it would make the sorting much slower).
[reply]
[d/l]
[select]
        If you need 1.4GB to sort 400MB that means that you are probably converting numbers to strings inside the sort.

        Yep! That's the problem.

        { ($c, $d) = ($a,$b); $c cmp $d

        I wonder if that optimisation could be worked into the internal sort?

        I started to suggest that if both had the (same) IOK or NOK flags set, then use a numeric comparison, but I guess it would only take one string embeded at the wrong place for the entire sort to have to be done over.

        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal?
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]

In Section Seekers of Perl Wisdom