in reply to Re: Re: threads and RAM-Usage
in thread threads and RAM-Usage

You're right, 120 threads is quite excessive, and I can't think of a situation where >120 threads would be needed either.

I was using the above code to write a custom SMTP server for spam prevention and we anticipated it to be heavily hit. During my benchmarks I wanted to simulate the unlikley event of a few hundred simultaneous connections to see how well it performed under this kind of pressure. The entire process was shutting down, no errors, no messages and it wasted many hours of my life trying to figure out why.

I suppose it's not really a *huge* problem in theory, but when you don't know why something happens, it could become a huge problem.

Replies are listed 'Best First'.
Re: Re: Re: Re: threads and RAM-Usage
by BrowserUk (Patriarch) on Jun 15, 2003 at 01:46 UTC

    Testing the limits is fair enough, and exiting without any error is also not freindly, though I think I've seen perl do that when I've stretched the memory beyond available swap space. The OS yells at me that I'm running out of resources, but perl dies quietly. This only seems to happen in extreme cases though.

    Out of interest, I just tried 

    perl58 -Mthreads -e"@t=map{threads->new(sub{sleep 60;})}1..120; sleep 
+10; $_->join for @t"

    [download]

    Which works fine, exactly as you'd expect for 121 threads, 120 + the initial thread, but if you push the number to 122 (or greater), and monitor the number threads in the task manager, it gets to 121 and stops. It executes pretty much as designed, but when it exits, it declares that there are still some number of threads (the number I tried to start - 120) still running.

    It seemed strange that you are hitting the same limit of 120 threads under XP as I am under NT4, so I did a little research. According to MSDN, if Win32 threads are created with a default stack size of 1MB, the limit is 2028 threads, and more if smaller stacks are used. (They also caution against creating more than "around 20":). So it appeared that the limit on the number of threads is a perl limit rather than an OS one.

    To test this I wrote a small C prog that created 500 threads and it ran clean. So the limit definately appears to be in the perl implementation.

    Then I took a browse through threads.c, and found that a linked list is used to store the thread data, and I can't see what would be limiting it.

    I did notice the the return from CreateThread() is never checked for validity, but that the counts of $known_threads and $active_threads is incremented regardless, which explains why I get the "A thread exited when nn threads were still active" warning if I try (and fail) to create more than 120 threads from perl.

    I'm not sure yet whether the 120 limit this is a bug or an undocumented limitation of ithreads, but I am certain that the non-checking of the return from CreateThread() is a bug and I'll report that tomorrow once I've had more time to try and track down the 120 thing.

    Examine what is said, not who speaks.
    "Efficiency is intelligent laziness." -David Dunham
    "When I'm working on a problem, I never think about beauty. I think only how to solve the problem. But when I have finished, if the solution is not beautiful, I know it is wrong." -Richard Buckminster Fuller


[reply]
[d/l]
      Wow, I wish I had a more of your patience and tenacity.

      Anyhows, on a FreeBSD box I have a threaded perl compiled and I'm able to have at leat 1000 threads running simultaneously with no problems.

[reply]

        Interesting. So the limitation isn't Win32 itself, nor threads.pm in general, but some intereaction between the two. Thanks for the info. Now I know that I only need look at the #ifdef WIN32... bits of threads.pm to find out where this limit of 120 is coming from.... at least theoretically.

        Examine what is said, not who speaks.
        "Efficiency is intelligent laziness." -David Dunham
        "When I'm working on a problem, I never think about beauty. I think only how to solve the problem. But when I have finished, if the solution is not beautiful, I know it is wrong." -Richard Buckminster Fuller


[reply]
[d/l]

In Section Seekers of Perl Wisdom