Beefy Boxes and Bandwidth Generously Provided by pair Networks
Clear questions and runnable code
get the best and fastest answer
 
PerlMonks  

Re^2: Looking for advice on how to tune stack size for threads

by fx (Pilgrim)
on Dec 03, 2010 at 12:37 UTC ( [id://875163] : note . print w/replies, xml ) Need Help??


in reply to Re: Looking for advice on how to tune stack size for threads
in thread Looking for advice on how to tune stack size for threads

Yes, trial and error seems to be the way here!....Never mind...

My original code had lots of different modules so I was thinking that the stack size needed to be quite high. Then, as so often is the way with Perl, I found that as I increased the number of threads the more randomly the program crashed. I've ran into so many thread-unsafe (or perhaps thread-unsure!) modules that a lot of my code is now system calls out to Unix command line utilities that do the same thing.

Yes, performance takes a hit but, in this case at least, I'm more interested in running many slow running processes concurrently so it's not all bad....

Thanks for your reply!

fx, Infinity is Colourless

  • Comment on Re^2: Looking for advice on how to tune stack size for threads

Replies are listed 'Best First'.
Re^3: Looking for advice on how to tune stack size for threads
by BrowserUk (Patriarch) on Dec 03, 2010 at 13:24 UTC
    Yes, trial and error seems to be the way here!....Never mind...

    Actually, you missed my point completely. Perhaps I didn't make it clear enough.

    The value of the stack size parameter will almost never be the source of, or the cure for, program crashes with threaded perl code.

    The default value (on windows, it may vary on other platforms) is 16MB. This is a (ludicrously) oversized allocation as attested to by the fact that--as I suggested above--when I've set this to be 4000 times smaller at just 4096 bytes, I've yet (on 5.10+) to find a piece of Perl code that would cause it to require more stack.

    I did succeed in making a thread crash through stack exhaustion on 5.8.something by using a carefully constructed regex; but that doesn't work any more.

    For some of the background to this please see Use more threads.. It was my explorations detailed in that thread that caused the stack_size parameter to be added to threads. The motivation being to allow the programmer to reduce the size of the default in order to save memory. I've never seen nor heard of an occasion when it needed to be increased.

    My original code had lots of different modules so I was thinking that the stack size needed to be quite high.

    That's a quite common misperception.

    Even the most memory hungry pure Perl module--Date::Manip is an old favourite for this purpose--requires almost no processor stack.

    And the same is true for the vast majority of well-written modules with XS/C components.

    Then, as so often is the way with Perl, I found that as I increased the number of threads the more randomly the program crashed.

    No disrespect meant, but this is almost always down to programming errors as a result of programmer misunderstanding.

    Just as when many monks get a little tetchy when people post statements that their program: "crashes", "fails", "prints errors", "doesn't work"; stories of such failures without detailed errors and code to demonstrate them are frustrating, because they leave no avenue through which to address them.

    I've ran into so many thread-unsafe (or perhaps thread-unsure!) modules that a lot of my code is now system calls out to Unix command line utilities that do the same thing.

    Yes, performance takes a hit but, in this case at least, I'm more interested in running many slow running processes concurrently so it's not all bad.

    Hm. Well, if it works for you that great, but it would be a whole lot better for everyone if you would post code that demonstrates the problems. That way, we can either work out what you are doing wrong; or detail the problems sufficiently that a bug report can be raised against it so they get fixed.

    My conservative estimate is that >80% of "threads problems" raised here are actually quickly identified as programmer errors that usually have relatively simple fixes; though it does often mean correcting some misconceptions and encouraging somewhat different working practices. Especially for those programmers used to using threading of different flavours in other languages.

    The upside of that is that the working code is usually far simpler than the buggy code originally posted.


    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.