in reply to Is it possible to create a single Hash-of-Hash.. with multiple threads..

G'day gsat,

Welcome to the monastery.

Is this the sort of thing you were after:

#!/usr/bin/env perl

use strict;
use warnings;
use autodie;
use threads;
use threads::shared;

use Storable;

my $storable_file = './pm_1078249_stored_hash';

{
    my $hoh_to_store = { A => { B => 2, C => 3 }, D => { E => 5, F => 
+6 } };
    store $hoh_to_store => $storable_file;
} 

my @threads;
my %hohoh :shared;

for (1 .. 3) {
    push @threads, threads->create(sub {
        $hohoh{threads->tid} = shared_clone(retrieve $storable_file);
    });
} 

$_->join for @threads;

# Test result
use Data::Dumper;
print Dumper \%hohoh;

# My housekeeping
unlink $storable_file;

[download]

Output:

$VAR1 = {
          '1' => {
                   'D' => {
                            'E' => 5,
                            'F' => 6
                          },
                   'A' => {
                            'B' => 2,
                            'C' => 3
                          }
                 },
          '2' => {
                   'D' => {
                            'E' => 5,
                            'F' => 6
                          },
                   'A' => {
                            'B' => 2,
                            'C' => 3
                          }
                 },
          '3' => {
                   'A' => {
                            'B' => 2,
                            'C' => 3
                          },
                   'D' => {
                            'E' => 5,
                            'F' => 6
                          }
                 }
        };

[download]

[You appear to have put square brackets around several pieces of text in your OP. This generates links here. See "Writeup Formatting Tips" and "What shortcuts can I use for linking to other information?" for details of how to fix your post.]

-- Ken

Replies are listed 'Best First'.
Re^2: Is it possible to create a single Hash-of-Hash.. with multiple threads..
by gsat (Initiate) on Apr 25, 2014 at 01:17 UTC

    Thanks much for your time. Your code is simple & works fine - I guess because it is simple 2 level deep & the subroutine is not increasing the depth while working.

    The data structure - I am handling with, will increase the structure depth. Basically branch-out, like a growing tree, every thread working on a unique branch is OK.

    As other suggestions came like - Storing output in a new structure - and finally merging it - is fine - how to return different sized depth of every branch structure varying between 9 to 14, number of branches would few tens of thousands - and join them to form the final tree.

    The starting structure that I need to share across threads - will be having depths like 4-6 level, when threads finished working every branch will have 9-14 levels.

    BTW, in my experience, the %hash kind of usage are not very convenient, rather - I prefer reference of hash-ref-of-hash-ref - right from the root. I used to like..

    $root->{$branch1}->{$branch2}..

    [download]

    Overall - so far in my understanding - perl threads is not cool, no matter what - It really cannot do big job with complex data structure in shared way - absolutely no references shared between threads. Internally every thread is creating another copy, so with GBs of data-structure I cannot really launch many threads.

    My single-thread serialized code is working great though - it can do serious work on massive data-structures. It lacks true parallelism..

    Perl Threads implementation so far, seems very inadequate - it is good for primary school student exhibiting how parallel programming works.

    I wish next gen threads will be able to do what I am asking - seamless work on the shared structure over GPUs too [why not - being Interpreted, won't stop it running in GPU I guess. :) - anyway Thanks a lot again!!

[reply]
[d/l]
      Perl Threads implementation so far, seems very inadequate - it is good for primary school student exhibiting how parallel programming works.

      That is a wrong conclusion drawn through a lack of understanding.

      If you would care to describe your application in detail -- not just generics, but rather scales and reasons; with examples of data and processing requirements -- then you would (probably) get help with finding definitive, practical and efficient solutions to those requirements.

      Your OP asks a simplistic question with an obvious answer: Yes, of course a multi-level hash can be constructed using multiple threads. It is simple and trivial:

      #! perl -slw
use strict;
use threads;
use threads::shared;

sub displayHash {
    my( $ref, $pad ) = @_;
    return "\n" unless keys %{ $ref };
    my $buf = '';
    for my $key ( sort keys %{ $ref } ) {
        $buf .= "$pad" if length $buf;
        $buf .= "{$key}" . displayHash( $ref->{ $key }, $pad . '   ' )
+;
    }
    return $buf;
}

sub worker {
    my( $ref, $reps ) = @_;

    for( 1 .. $reps ) {
        my $copyRef = $ref;
        for my $step ( map chr( 97 + rand 26 ), 1 .. int( rand 10 ) ) 
+{
            if( exists $copyRef->{ $step } ) {
                $copyRef = $copyRef->{ $step };
            }
            else {
                lock %{ $copyRef };
                $copyRef = $copyRef->{ $step } = &share( {} );
            }
        }
    }
}

our $REPS //= 20;
our $THREADS //= 4;

my %HoHos : shared;

my @threads = map threads->create( \&worker, \%HoHos, $REPS ), 1 .. $T
+HREADS;

$_->join for @threads;

print displayHash( \%HoHos, '' );
__END__
C:\test>junk91 -REPS=7
{b}{l}{w}{g}
{d}{t}{k}{f}{g}{g}{g}{i}{j}
{f}{j}{v}
{g}{s}{g}{h}{g}
{i}{f}{j}{r}{m}{v}{u}{v}{b}
{k}{d}{f}{q}{k}
   {z}{t}{m}{h}{e}{i}
{m}{b}{l}
   {l}{u}{p}{t}{d}{h}
   {q}{k}{y}
{n}{f}{u}{v}{z}{z}{l}
{o}{f}
{p}{u}{s}{w}{n}{t}
{q}{h}
   {y}{k}{d}{l}{a}{n}{a}{i}
{r}{q}{t}{g}
{v}{o}{n}{x}{b}{g}{c}
{w}{a}
{y}{a}
   {d}{y}{j}
{z}{a}{p}{t}{i}{r}{t}
   {y}{i}{g}{c}{v}{z}{z}{k}

      [download]

      But, it is also probably not very helpful to your real application.

      With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
[d/l]

        Dear Monk BrowserUk! Please take a bow!!

        Thank You so much - for enlightening me :) Indeed - it was my lack of understanding.

        I have found this entire discussion from everybody - very helpful.

        -=-=-=-=-=-=-=-

        Now, I need more :) It seems - me and my code is tired of doing so many .. if exists.. else share.. kinda additives before I do any structure change - and I think no way to avoid that. Now a couple of Questions here:

        Q1. Is there any easier method?

        Q2. Is it possible to grow a local tree or branch data-structure, then create a shared_clone - and graft the new tree to s sub-branch of main shared tree?

        Q3. I have few subroutines to be called many times - based on data - from inside my thread_worker. So far I am lacking depth in understanding or visualizing - how/when multiple threads work on different set of data, while calling a subroutine - does the code-inside-subroutine-block gets a replicated inside the thread uniquely? - else how the sanity is retained? - sorry if I am asking too basic question - maybe it is the case, I am confused - and not clear enough - where is my confusion too..

        Q4. Inside subroutine - do I have to use every variable as local and not-shared?

        Thanks a lot! Again & Again!

[reply]

In Section Seekers of Perl Wisdom