Hello!

I have a script working with large data structures and several threads. The script works fine and so far has scaled up nicely enough. Recently, however, I have been hitting some limits when running 100+ threads and hashes with close to a million entries.

The script uses standard threads and threads::shared. The large data structure is needed by all threads. Since it is :shared, one copy of all the data exists for each thread. This makes memory consumption of my script very high.

I am looking for ways to optimize sharing the data structure. The data should exist only once in memory and I should be able to access it from all threads. It is OK if I need to lock all acccesses or modify the data access code somehow.

So far I have found Thread::Tie, which looks promising, but wouldn't work with my complex structures (hash containing array refs and those arrays containing more refs). If this is the only way to go, I'd be prepared to rewrite my code to use one-dimensional hashes, perhaps pack()'ing data as strings. But first I'd like to know if there are some easier/better alternatives.

All ideas about sharing a single copy of data across threads are welcome. Please, no suggestions about the number of threads or the size of my structures. They have already been thoroughly evaluated.

I am working on 32-bit Windows with Perl 5.12.2.

In reply to Sharing large data structures between threads by Anonymous Monk

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.