BrowserUk has asked for the wisdom of the Perl Monks concerning the following question:

Can anyone see why this highly reduced (sic) code would leak memory like sieve?

Can it be reproduced upon other perls? 

#! perl -slw
use strict;
use List::Util qw[ reduce ];

sub s1 {
    my $r = shift;
    reduce{ $r->[0] } 1 .. 10;
}

while( 1 ) {
    my $t = map s1( $_ ), [1], [2], [3];
}

[download]
Examine what is said, not who speaks.
Silence betokens consent.
Love the truth but pardon error.

Replies are listed 'Best First'.
Re: Massive memory leak (AS 810)
by PodMaster (Abbot) on Feb 25, 2005 at 16:52 UTC
    Looks like a bug in the XS version of reduce. The perl version doesn't seem to leak 
    #! perl -slw

sub reduce (&@) {
  my $code = shift;

  return shift unless @_ > 1;

  my $caller = caller;
  local(*{$caller."::a"}) = \my $a;
  local(*{$caller."::b"}) = \my $b;

  $a = shift;
  foreach (@_) {
    $b = $_;
    $a = &{$code}();
  }

  $a;
}


use strict;
#use List::Util qw[ reduce ];

sub s1 {
    my $r = shift;
    reduce{ $r->[0] } 1 .. 10;
}

while( 1 ) {
    my $t = map s1( $_ ), [1], [2], [3];
}

    [download]

    MJD says "you can't just make shit up and expect the computer to know what you mean, retardo!"
    I run a Win32 PPM repository for perl 5.6.x and 5.8.x -- I take requests (README).
    ** The third rule of perl club is a statement of fact: pod is sexy.

[reply]
[d/l]
Re: Massive memory leak (AS 810)
by bmann (Priest) on Feb 25, 2005 at 16:58 UTC
    From the List::Util 1.14 changelog:

    1.14 -- Sat May 22 08:01:19 BST 2004

Bug Fixes
  * Fixed memory leak in reduce()

    [download]

    Looks like it was previously discussed at Uncollected Garbage ... and reported to gbarr here

    Think I'll upgrade...

[reply]
[d/l]
      Except my version of List::Util is 1.14 and I have the memory leak ...

      Being right, does not endow the right to be rude; politeness costs nothing.
      Being unknowing, is not the same as being stupid.
      Expressing a contrary opinion, whether to the individual or the group, is more often a sign of deeper thought than of cantankerous belligerence.
      Do not mistake your goals as the only goals; your opinion as the only opinion; your confidence as correctness. Saying you know better is not the same as explaining you know better.

[reply]
        Confirmed here too. Just installed 1.14, still eats ram on both Win2K and Debian.
[reply]
Re: Massive memory leak (AS 810) (also with 5.8.5 / 1.14)
by Corion (Patriarch) on Feb 25, 2005 at 17:17 UTC

    Leaks too on 5.8.5 and List::Util 1.14, so things seem to point in the general direction of the List::Util XS code...

    C:\Projekte\Mailfetch>perl -MList::Util -le "print $List::Util::VERSIO
+N"
1.14

C:\Projekte\Mailfetch>perl -v

This is perl, v5.8.5 built for MSWin32-x86-multi-thread

Copyright 1987-2004, Larry Wall

Perl may be copied only under the terms of either the Artistic License
+ or the
GNU General Public License, which may be found in the Perl 5 source ki
+t.

Complete documentation for Perl, including FAQ lists, should be found 
+on
this system using `man perl' or `perldoc perl'.  If you have access to
+ the
Internet, point your browser at http://www.perl.com/, the Perl Home Pa
+ge.

    [download]
[reply]
[d/l]
Re: Massive memory leak (AS 810)
by bmann (Priest) on Feb 25, 2005 at 16:01 UTC
    List::Util v1.13, perl 5.8.4 on Debian leaks like crazy. Over 500M in under 30 seconds!

    Update - Not Scalar::Util, List::Util!
    Update 2 - s/under seconds/under 30 seconds/

[reply]
Re: Massive memory leak (AS 810)
by dragonchild (Archbishop) on Feb 25, 2005 at 15:51 UTC
    I dunno why, but it leaks 8K every 3 iterations on SunOS 5.8, Perl 5.8.4.

    Being right, does not endow the right to be rude; politeness costs nothing.
    Being unknowing, is not the same as being stupid.
    Expressing a contrary opinion, whether to the individual or the group, is more often a sign of deeper thought than of cantankerous belligerence.
    Do not mistake your goals as the only goals; your opinion as the only opinion; your confidence as correctness. Saying you know better is not the same as explaining you know better.

[reply]
Re: Massive memory leak (AS 810)
by ww (Archbishop) on Feb 25, 2005 at 16:04 UTC
    Joost recently wrote (in Memory leak )

    "Devel::LeakTrace or one of the other Devel::Leak* modules might help."

    WARNING: I have no competence here, but found several threads re Memory Leak, while trying to discover if your "reduced" code was effectively parallel to that which was the subject of yet another discussion (which I have NOT re-found).
    HTH
[reply]
Re: Massive memory leak (AS 810)
by RazorbladeBidet (Friar) on Feb 25, 2005 at 16:43 UTC
    I get no memory leak on AIX 5.2 using ps v PID
    --------------
    It's sad that a family can be torn apart by such a such a simple thing as a pack of wild dogs
[reply]
      Curious. On my AIX 5.2 box with perl 5.8.5 it leaks about 70MB a second and segfaults after about 4 seconds.
[reply]
        Hmm... my perl is 5.8.0 with Util version 1.07.00
        --------------
        It's sad that a family can be torn apart by such a such a simple thing as a pack of wild dogs
[reply]
Re: Massive memory leak (AS 810)
by cosimo (Hermit) on Feb 28, 2005 at 07:25 UTC

    As Corion and Podmaster already pointed out, it seems something in the XS code. I'm no XS expert, but I can confirm that commenting out the SvREFCNT_inc(cv) statement solved the problem, as adding a SvREFCNT_dec(cv) at the end of the main loop does. Either way, the List::Util extension builds and executes correctly.
    The first() function seems to have the same problem...

    Looks like the CPAN RT bug is still open.

[reply]
[d/l]
[select]

      Update: Corion provided this working link. Thanks, Corion.

      You may like to add your findings to the thread at [news://nntp.perl.org/perl.perl5.porters/98596]

      I realise that that link doesn't work. They used to, but they stopped it because apparently some people can't use newsreaders?

      Examine what is said, not who speaks.
      Silence betokens consent.
      Love the truth but pardon error.
[reply]

        It looks like this bug has been fixed with List::Util 1.14_1.

        Interestingly, it was fixed by removing the call to SvREFCNT_inc(cv), and adding a LEAVESUB(cv) at the end of first() and reduce() subs. Can anyone shed some light on the reason why this works?

        https://rt.perl.org/rt3/Ticket/Display.html?id=34267
        (perl.org credentials required)

[reply]
[d/l]
[select]
Re: Massive memory leak (AS 810)
by Anonymous Monk on Mar 01, 2005 at 03:28 UTC
    http://rt.cpan.org/NoAuth/Bug.html?id=5466
[reply]

      If you take a look at the XS code referred to in that thread via the link that Corion posted in Re: Uncollected Garbage ..., you'll see that the version was 1.13, and included the following snippet that he identified as the possible cause: 

           if (!CvDEPTH(cv))
       (void)SvREFCNT_inc(cv);

      [download]

      If you then take a look at the latest version, you'll see that snippet is no longer present.

      If you then look at the p5p list, you'll see that dave_the_m identified a new and different problem which also affected first(), and offered a patch to fix both.

      My interpretation, having recently attempted to use the reduce() XS code as a template for optimal callback performance from Inline::C, is that trying to bypass work out what the various macros do--and why--is very, very hard.

      Examine what is said, not who speaks.
      Silence betokens consent.
      Love the truth but pardon error.
[reply]
[d/l]
[select]

Back to Seekers of Perl Wisdom