mpaduano has asked for the wisdom of the Perl Monks concerning the following question:

Hello.

I have a small test program (below) that behaves in a way that I can't quite explain. As it is, this program will run fast and steadily until memory is exhausted. If the IO::Handle->new() is commented out and lines 3,4 are uncommented, again, the program speedily runs out of memory. BUT, if IO::Handle->new() is uncommented and lines 1,2 are uncommented, this program very quickly gets bogged down. Some sort of extra overhead must be involved with managing these IO::Handle references. Furthermore, I plotted the slowdown vs number of handles and it does not scale linearly, but worse than linearly.

What is going on here? Is there some special interaction of IO::Handle objects with the GC? Why do IO::Handle objects start to bog down the program even after they are closed? CPU cycles shoot up but the program actually does less (visible) work. Eventually, the CPU pegs at 99% and the program can't get through the loop more than a few times per second. I originally noticed this in a program that was leaking socket handles (but there was plenty of memory to leak into... the slowdown is what we noticed).


use IO::Handle;


@ary = ( );
$j = 0;
for($i=1; $i<8000001; $i++) {
     $j += $i;
     $z = IO::Handle->new();
#1#     $z->fdopen(fileno(STDERR), 'w');
#2#     $z->close();
#3#     $z = { };
#4#     $z->{X} = 1;
     push( @ary, $z );
     if( $i % 1000 == 0 ) {
         $t = time();
         STDOUT->autoflush(1);
         print "$t    $i\n";
     }
}

[download]
thanks,

matt

Replies are listed 'Best First'.
Re: IO::Handle slowdowns?
by Zaxo (Archbishop) on Jan 31, 2004 at 01:48 UTC

    In all cases, you are building a global array, @ary, that gets up to eight million elements, each a reference to some perl type. That eats lots of memory, probably forcing you into swap and thrashing. The bigger the structure, the sooner. Swap is notorious for bad scaling.

    To open and close IO::Handles inside the loop naturally takes more time than setting up an anonymous hash. In many cases you are contending against swap for the disk spindle.

    After Compline,
    Zaxo

[reply]
      hi.

      actually, no. The CPU starts to bog down immediately and the CPU nearly pegs before the process image is even 15% of available RAM. No swaping, no paging.

      Try running the program and notice the slowdown with the Handles but not the hashes... Sockets will slow the program down too, but be sure to send() at least one character (if you don't call send() with at least one character, it doesn't happen... similar to the open/close effect for the FD).

      It isn't about the Handle objects being slower than ordinary hashes, it is about the wall time to process each Handle going to infinity while all system resources seem to be in ample supply. For regular hashes, the wall time behaves.

      This one will require a real guru I am afraid...

      matt
[reply]
        Update: none of below applies, since Windows doesn't use glibc.

        What OS is this? I wonder if you are hitting the pathological-free bug with an older glibc.

        Update: the bug I'm thinking of is in glibc 2.2.x where free calls will suddenly start taking an enormous amount of time and cpu. If I understand correctly, the malloc library was rewritten for glibc 2.3 and the problem no longer exists there. If you have a glibc 2.2.x, make sure perl is built to use it's own malloc (Configure with -Dusemymalloc) or upgrade to a newer glibc.

[reply]

Back to Seekers of Perl Wisdom