in reply to Re^4: Strange memory leak using just threads
in thread Strange memory leak using just threads

Now that's interesting. I didn't use use_perl_malloc (as you can see in the later comment from 860733 ), but I did use a bunch of -D's that I really have no idea what they do, they were just what was in the config for the stock 5.10 that came with deb lenny.

I'm recompiling 5.12.2 now with just -des -Dprefix=$HOME/perl512 -Dusethreads.

And bonk. It still seems to be leaking memory.

Replies are listed 'Best First'.
Re^6: Strange memory leak using just threads
by BrowserUk (Patriarch) on Sep 20, 2010 at 21:37 UTC
    I didn't use use_perl_malloc ...

    I'm recompiling 5.12.2 now with just -des -Dprefix=$HOME/perl512 -Dusethreads .

    Then perhaps the thing to do would be to try it with use_perl_malloc?

    I'd also try adding usemultiplicity, but this is all just speculation on my behalf.

    First thing you need to do really is get in contact with someone who has a *nix perl that doesn't exhibit the problem.

    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
[d/l]
[select]

      AFAIK, you can't have threaded perl without MULTIPLICITY.

[reply]
[d/l]
Re^6: Strange memory leak using just threads
by zwon (Abbot) on Sep 21, 2010 at 02:32 UTC

    If you search you will find that this subject was discussed here before a lot of times. Yes, perl threads leaking memory on Linux, and yet it seems nobody especially concerned about this. If your Perl program supposed to work on Linux there's no much sense to use threads. Perl threads is not the same thing as POSIX threads. Actually on Linux creating a new process with fork is usually cheaper than creating a new Perl thread both in terms of speed and memory. If you need to handle multiple IO streams, you should use POE, AnyEvent (this one BrowserUk wouldn't recommend I guess), or IO::Select in simple cases, and it will be more efficient than using threads. If you want to perform some CPU intensive operations, than multiprocess model will allow you to use several CPUs, and it is more scalable than multithread model. So what is the reason why are you want to use threads?

[reply]

      Hence the existence of forks.pm. It has the improved efficiency of creating a new "thread" because it uses fork() to do that (clearly not something to use with a native-Windows Perl, of course). (It offers the same interface as threads.pm.)

      I haven't used forks.pm myself. But I have worked with POE and I recently evaluated work done by some very smart co-workers who were using similar systems. People who work with something like POE frequently convince themselves that it isn't much of a burden (as I had).

      Having a chance to look at it with hindsight and not from the middle of it, I was very convinced that the POE way of providing cooperative multi-tasking is indeed quite a burden, even after one has gotten used to the awkwardness. Even the simplest algorithm, if it can involve a pause (such as reading from a network socket) must be broken into pieces. And any routine that calls such a fractured routine must itself be broken apart. And anything that calls that, etc.

      This fracturing also means you don't have much of a call stack and can't easily throw and catch exceptions across the fractures. It keeps getting in your way every day in small and large ways.

      So, I don't consider using POE to be an untenable solution. But I believe it incurs enough of a cost that I wouldn't make it my first choice.

      I believe that forks.pm and Coro are worth considering first, as they allow unfractured coding. Coro will not use multiple "cores", but in most cases I don't mind because I'm usually either using a system that has plenty of other things to do or I have $N servers dedicated to the task and can just run 4 Coro-using instances on each quad-core server (for example). But the limitation is worth keeping in mind as there are certainly situations where it is a drawback.

      - tye        

[reply]
        Hence the existence of forks.pm. It has the improved efficiency of creating a new "thread" because it uses fork() to do that
        1. How long does it take to start one thread per core? Answer: 0.0334138870239258 / 4.

          Sure, fork will be faster. But on any process long-running enough to be worth multi-threading, 0.01 per core is but a drop in the ocean.

        2. How long does it take to communicate 1000 work items to each of those work threads? Answer: 4 cores; 4 threads; 4000 items; 0.113806962966919 seconds.

          How long will it take to

          1. freeze those work items;
          2. send them via sockets;
          3. receive them at the other end;
          4. unfreeze them?

          And I bet that establishing those socket connections more than negates the lower start-up cost of fork.

        Which timing is the more important?

        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]

In Section Seekers of Perl Wisdom