I haven't looked at your code yet. I might later. It's pretty massive ;-)

Perl threads have that problem, yes. The problem is mainly caused by the fact that all data structures of Perl are copied to each new thread, shared and unshared. Every time you start a new thread. Only opcode trees appear are shared (although I have to take Arthur Bergman's word for that as I never dared go into the actual Perl core to see what's going on there).

This means that you should try to reduce your data-structures before firing off threads. There are several ways to do that: one of them is Thread::Use which allows you to use modules only in threads. Or you could use Thread::Pool to start up a number of worker threads way at the beginning of your program. And you might want to look at monitored queues, either Thread::Queue::Monitored or Thread::Conveyor::Monitored.

Arthur is now working on using a different approach to shared variables which involves some COW (Copy On Write) techniques. But this is for Ponie so it will take some time before that becomes mainstream.

Finally, there is one other alternative that I can recommend if you're not on Win32: forks. Instead of threading, it forks so the OS is taking care of all the COWing. It does this at the expense of using a TCP connection for communication between threads. YMMV

Hope this helps.

Liz

In reply to Re: Question about threads by liz
in thread Question about threads by Gwalchmai

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.