Re: Looking for advice on how to tune stack size for threads

I'd set it to: stack_size => 4096 and then try it.

Actually. I've modified my perl binary so that the 'default' value used, when no stack size is explicitly stated. is 4096, this being the minimum that can be used on my system. So far, I've never encountered a problem due to stack.

The reason you can get away with this, is because for the most part, Perl doesn't use the C/processor stack.

Perl manages it own stack (actually:stacks), and these are allocated from the heap. Until recently, very complex regex could consume prodigious amount of the C stack, but then p5p (I think: dave_the_m?) converted the then recursive regex engine to iterative, thus removing perl's last big stack dependency.

However, if you use modules with badly written XS or C components that either allocate large items/quantities of items on their stacks; or enter unbridled recursion, they may cause your threads to run out of (thread) stack if you set it too low.

Personally, I've never encountered such a problem, even with my extreme minimalism, but I'm also quite choosy about the modules I use.

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

Comment on Re: Looking for advice on how to tune stack size for threads Download Code

Replies are listed 'Best First'.
Re^2: Looking for advice on how to tune stack size for threads by fx (Pilgrim) on Dec 03, 2010 at 12:37 UTC
Yes, trial and error seems to be the way here!....Never mind... My original code had lots of different modules so I was thinking that the stack size needed to be quite high. Then, as so often is the way with Perl, I found that as I increased the number of threads the more randomly the program crashed. I've ran into so many thread-unsafe (or perhaps thread-unsure!) modules that a lot of my code is now system calls out to Unix command line utilities that do the same thing. Yes, performance takes a hit but, in this case at least, I'm more interested in running many slow running processes concurrently so it's not all bad.... Thanks for your reply! fx, Infinity is Colourless	[reply]
Re^3: Looking for advice on how to tune stack size for threads by BrowserUk (Patriarch) on Dec 03, 2010 at 13:24 UTC
Yes, trial and error seems to be the way here!....Never mind... Actually, you missed my point completely. Perhaps I didn't make it clear enough. The value of the stack size parameter will almost never be the source of, or the cure for, program crashes with threaded perl code. The default value (on windows, it may vary on other platforms) is 16MB. This is a (ludicrously) oversized allocation as attested to by the fact that--as I suggested above--when I've set this to be 4000 times smaller at just 4096 bytes, I've yet (on 5.10+) to find a piece of Perl code that would cause it to require more stack. I did succeed in making a thread crash through stack exhaustion on 5.8.something by using a carefully constructed regex; but that doesn't work any more. For some of the background to this please see Use more threads.. It was my explorations detailed in that thread that caused the stack_size parameter to be added to threads. The motivation being to allow the programmer to reduce the size of the default in order to save memory. I've never seen nor heard of an occasion when it needed to be increased. My original code had lots of different modules so I was thinking that the stack size needed to be quite high. That's a quite common misperception. Even the most memory hungry pure Perl module--Date::Manip is an old favourite for this purpose--requires almost no processor stack. And the same is true for the vast majority of well-written modules with XS/C components. Then, as so often is the way with Perl, I found that as I increased the number of threads the more randomly the program crashed. No disrespect meant, but this is almost always down to programming errors as a result of programmer misunderstanding. Just as when many monks get a little tetchy when people post statements that their program: "crashes", "fails", "prints errors", "doesn't work"; stories of such failures without detailed errors and code to demonstrate them are frustrating, because they leave no avenue through which to address them. I've ran into so many thread-unsafe (or perhaps thread-unsure!) modules that a lot of my code is now system calls out to Unix command line utilities that do the same thing. Yes, performance takes a hit but, in this case at least, I'm more interested in running many slow running processes concurrently so it's not all bad. Hm. Well, if it works for you that great, but it would be a whole lot better for everyone if you would post code that demonstrates the problems. That way, we can either work out what you are doing wrong; or detail the problems sufficiently that a bug report can be raised against it so they get fixed. My conservative estimate is that >80% of "threads problems" raised here are actually quickly identified as programmer errors that usually have relatively simple fixes; though it does often mean correcting some misconceptions and encouraging somewhat different working practices. Especially for those programmers used to using threading of different flavours in other languages. The upside of that is that the working code is usually far simpler than the buggy code originally posted. Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice.	[reply]