in reply to Use more threads.

> editbin /stack:0x00100000 \yourperl\bin\perl.exe

Thank you so much for this!!! I just did it on Perl-5.12.3 and it allowed my script to use more of the available memory in the system. I couldn't pass a limit of around 80 threads, although on Perl-5.16.3 I could run over 120 threads with no problem. So I figured the versions must have been compiled with different limits (both ActivePerl), and so I found this post.

Unfortunately for now I must stick with 5.12.3, because 5.16.3 can't play with DBI connects in threads very well, producing tons of warnings about redefined subroutines. But with your "hack" I can still use 5.12.3 for a while.

Replies are listed 'Best First'.
Re^2: Use more threads.
by BrowserUk (Patriarch) on Apr 28, 2013 at 06:53 UTC

    Note also the stack_size use line option added to threads as a result of this thread.

    I quite routinely set this as low as 4096 without problems, and doing so I have had a proof of concept code with 3000 threads running in 4GB of memory.

    Not that there are many good uses for using more than 3 or 4 times as many threads as you have cores. So unless you are running your code on a system with 32 cores or more, seeking to run so many threads is usually a sign of naive coding.

    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
[d/l]

      I agree completely with what you said about using many threads (or maybe using any threads at all) but what I've written is a proxy checker (with GUI), so most of the time my threads are waiting for a response or a timeout.

      My next idea is to see if I can create 100 threads any faster, so I'll tighten it up as much as it is possible, and try that stack_size option. Although my coding is a bit naive and far from professional, my only fear is that I'm inventing the wheel sometimes. he-he.

[reply]
        t I've written is a proxy checker (with GUI), so most of the time my threads are waiting for a response or a timeout.

        Starting new threads for each connection is naive. And wasteful.

        A pool of threads and a queue of requests is more efficient (of time and resources), and actually easier to program and reason about.

        With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]

In Section Meditations