mattdeans has asked for the wisdom of the Perl Monks concerning the following question:

Hi,all

I have to use multi-process or multi-thread to process log files. Every single process or thread will use a complex and huge memory use data structure( a hash with a simple key to very complex structure value. In thread or process, I send a key and respect this complex structure data ).

So, I have a problem: first, this hash is very very big, I cannot create this hash before I create process or thread, I have not enough memory. then, this hash is very complex, I've tried threads::shared, not sucessed.

How can I solve this?

Thanks.
  • Comment on How to share data structures between processes or threads?

Replies are listed 'Best First'.
Re: How to share data structures between processes or threads?
by BrowserUk (Patriarch) on Sep 13, 2011 at 02:29 UTC

    Could you show us:

    • Where does the data come from?

      Is it all loaded up from a file at start-up? Or is it built-up over time as the process(es) run? Other?

    • Could you post a few (2 or 3) examples of the keys + associated values?
    • Is the entire dataset required for every iteration?
    • What do you do with the data when the program(s) end?

      Do you save it some where? Discard it and re-build it next time the program runs?

      Or is the data-structure built once as you process each log file, some statistics derived from it and then discarded never to be used again?

    Beware of responders that don't wait for answers to these questions first.

    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
      Thanks
      Where does the data come from?
      Is it all loaded up from a file at start-up? Or is it built-up over time as the process(es) run? Other?
      ====>From database, do some transform. Once done, never change.

      Could you post a few (2 or 3) examples of the keys + associated values? ====>Dont have it for now. value is mixed combination of hash/array/refs of hash,of array, of refs.
      Is the entire dataset required for every iteration?
      ====>yes.
      What do you do with the data when the program(s) end?
      ====>discards. after the program end , i will get a new data, and insert/update to database.
      Do you save it some where? Discard it and re-build it next time the program runs? ====>mysql
      Or is the data-structure built once as you process each log file, some statistics derived from it and then discarded never to be used again?
      ====>see above.
      Thanks.~
[reply]
        Where does the data come from? Is it all loaded up from a file at start-up? Or is it built-up over time as the process(es) run? Other? ====>From database, do some transform. Once done, never change.

        Given the scant description of the data and processing, my only response is that rather than filling (more than) your memory capacity with data read from an RDBMS in order to do processing in Perl, you should be leaving the data in the DB and formulating SQL queries that allow the RDMBS to perform the processing for you.

        Maybe if you explained why that is not possible, then an alternative solution might suggest itself, but it would require much more detail.

        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
Re: How to share data structures between processes or threads?
by zentara (Cardinal) on Sep 13, 2011 at 11:36 UTC
[reply]

Back to Seekers of Perl Wisdom