Re^5: Is this a job for a fork?

Actually guys, I wonder if I could borrow your brains again. I'm happy to start a new thread if needs be but this is relevant to my previous query.

I didn't realise that sharing hashes would be different from sharing more complex data structures like a hash of hashes.

At the point of making a change to the shared object, is there a way of doing the following:

In thundergnat's code above:

# this
$count{$param1} = $param1 * $param2;
# to this for example
$count{$param1}{$param2}{'count'} = $param1 * $param2;
[download]

Or in BrowserUk's code above:

# this
++$hashref->{ $key };
# to this for example
++$hashref->{ $key }->{'count'}
[download]

Is it possible to get around this apparent limitation in threads::shared as stipulated in their docs?

Once again I very much appreciate any help you can give me

Cheers,

Rich

Comment on Re^5: Is this a job for a fork? Select or Download Code

Replies are listed 'Best First'.
Re^6: Is this a job for a fork? by thundergnat (Deacon) on Jul 13, 2010 at 20:00 UTC
It would be impractical and very difficult to build a shared multilevel hash on the fly, but you don't really need to. Do your expensive processing in your threads then build your multilevel hash in the main thread. Pass the parameters through a shared buffer. Here's my earlier script updated to use queues and a buffer. Note: a bunch of code stolen from BrowserUks example above. #!/usr/bin/perl use warnings; use strict; use Data::Dumper; use threads ( stack_size => 4096 ); use threads::shared; use Thread::Queue; my %count; my @buffer : shared; my $maxthreads = 5; my $queue = Thread::Queue->new; my @threads = map threads->create( \&inc, $queue, \@buffer ), 1 .. $ma +xthreads; # there are way more lines than this but anyway... for my $i ( 1 .. 500 ) { # this just represents that I've found a pair and want to run the +sub if ( $i % 50 == 0 ) { # push the pair in the queue $queue->enqueue("$i:2"); } empty_buffer( \@buffer ) if @buffer; } $queue->enqueue( (undef) x $maxthreads ); $_->join for @threads; empty_buffer( \@buffer ) while @buffer; print Dumper \%count; sub inc { my ( $queue, $arrayref ) = @_; my $tid = threads->tid; while ( my $param = $queue->dequeue ) { # long subroutine using passed params # pausing for effect! # this simulates that the sub takes a while to run sleep( 2 + rand 5 ); # let you know something is happening print "Thread# $tid \| Parameter: $param\n"; my ( $p1, $p2 ) = split /:/, $param; # add it to the buffer lock $arrayref; push @{$arrayref}, "$p1:$p2:" . $p1 * $p2; } } sub empty_buffer { my $buffref = shift; my ( $p1, $p2, $result ) = split /:/, shift @{$buffref}; $count{$p1}{$p2} = $result; } [download]	[reply] [d/l]

Replies are listed 'Best First'.

Re^6: Is this a job for a fork?
by thundergnat (Deacon) on Jul 13, 2010 at 20:00 UTC

It would be impractical and very difficult to build a shared multilevel hash on the fly, but you don't really need to. Do your expensive processing in your threads then build your multilevel hash in the main thread. Pass the parameters through a shared buffer.

Here's my earlier script updated to use queues and a buffer.

Note: a bunch of code stolen from BrowserUks example above.

#!/usr/bin/perl

use warnings;
use strict;
use Data::Dumper;

use threads ( stack_size => 4096 );
use threads::shared;
use Thread::Queue;

my %count;
my @buffer : shared;
my $maxthreads = 5;

my $queue = Thread::Queue->new;

my @threads = map threads->create( \&inc, $queue, \@buffer ), 1 .. $ma
+xthreads;

# there are way more lines than this but anyway...
for my $i ( 1 .. 500 ) {

    # this just represents that I've found a pair and want to run the 
+sub
    if ( $i % 50 == 0 ) {

        # push the pair in the queue
        $queue->enqueue("$i:2");
    }
    empty_buffer( \@buffer ) if @buffer;
}

$queue->enqueue( (undef) x $maxthreads );
$_->join for @threads;

empty_buffer( \@buffer ) while @buffer;

print Dumper \%count;

sub inc {
    my ( $queue, $arrayref ) = @_;
    my $tid = threads->tid;
    while ( my $param = $queue->dequeue ) {

        # long subroutine using passed params
        # pausing for effect!
        # this simulates that the sub takes a while to run
        sleep( 2 + rand 5 );

        # let you know something is happening
        print "Thread# $tid | Parameter: $param\n";

        my ( $p1, $p2 ) = split /:/, $param;

        # add it to the buffer
        lock $arrayref;
        push @{$arrayref}, "$p1:$p2:" . $p1 * $p2;
    }
}

sub empty_buffer {
    my $buffref = shift;
    my ( $p1, $p2, $result ) = split /:/, shift @{$buffref};
    $count{$p1}{$p2} = $result;
}
[download]

[reply]
[d/l]