Beefy Boxes and Bandwidth Generously Provided by pair Networks
Don't ask to ask, just ask
 
PerlMonks  

Using clone/dclone with threads

by Anonymous Monk
on Nov 22, 2004 at 03:27 UTC ( [id://409471]=perlquestion: print w/replies, xml ) Need Help??

Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

Guys,

I'm trying to copy hash record structures from a global shared hash to a local copy, so the threads won't have to lock the global all the time.
Basically I have

{ package cfg; %cfg::sessions = (); threads::shared::share(%cfg::sessions); }
with data like this:
{A}{1}{C} = 1
{A}{1}{D} = 2
.
.
{B}{1}{C} = 3
{B}{1}{D} = 4
Later in a thread I want to do a deep copy of a given record eg
$myhash{A} = $cfg::sessions{A};
I've tried eg
use Storable qw(dclone); $myhash{A} = dclone($cfg::sessions{A});
but get 'Not a reference value' error.

I tried

use Clone qw(clone); $myhash{A} = clone($cfg::sessions{A});
but get a 'Segementation violation'.

In a simple (non-thread) test prog, both of the above work perfectly.
Is it possible to make these work under threads, or should I write my own fn?
Cheers
Chris

Replies are listed 'Best First'.
Re: Using clone/dclone with threads
by BrowserUk (Patriarch) on Nov 22, 2004 at 07:21 UTC
    I'm trying to copy hash record structures from a global shared hash to a local copy, so the threads won't have to lock the global all the time.

    Basically, what you are trying to do is the completely wrong way to go about things when using iThreads with shared data.

    Shared data is already copied to every thread that uses it. Making further copies is just dangerous, unnecessary, and impractical. You would still need to lock the shared data to copy it to your local copy, and again when you put it back. The costs in terms of both time and memory to create duplicate local copies of shared data totally outway any percieved benefit from avoiding locking.

    If you modify your local copy on two threads, and then need to update the shared copy with the changes, you have no way to resolve the conflicts of merging the changes.

    Considerable effort by some very clever people has gone into making sharing work. It was difficult for them to get it to the point where it is now pretty safe, transparent and usable. For you to try and replicate that in user-level Perl code is fraught with dangers and ultimately, a complete waste of time and effort.

    In case that isn't clear :) Don't do that!


    Examine what is said, not who speaks.
    "But you should never overestimate the ingenuity of the sceptics to come up with a counter-argument." -Myles Allen
    "Think for yourself!" - Abigail        "Time is a poor substitute for thought"--theorbtwo         "Efficiency is intelligent laziness." -David Dunham
    "Memory, processor, disk in that order on the hardware side. Algorithm, algorithm, algorithm on the code side." - tachyon
Re: Using clone/dclone with threads
by castaway (Parson) on Nov 22, 2004 at 06:09 UTC
    Can you write a runnable example with actual threads? Then it would be easier to see if its your mistake, and if not, maybe we can play with fixing it..

    Have you tried checking that $cfg::sessions{A} is indeed a reference? try printing ref($cfg::sessions{A}) there and see what it says.

    C.

      FYI, here's some code. Uncomment/comment Clone/Storable, clone/dclone as reqd:

      #!/usr/bin/perl -w + use strict; use threads; use threads::shared; use Clone qw(clone); #use Storable qw(dclone); + my ( $thr1, # thread handle $key, $v1, $v2, $v3, $tmp1, $tmp2, $key1, $key2, $key3 ); + # pkg test { + package cfg; + %cfg::thash = (); threads::shared::share(%cfg::thash); } + { lock(%cfg::thash); $tmp1 = {}; $tmp2 = {}; share($tmp1); share($tmp2); + + $v1="B"; $v2="2"; $cfg::thash{$v1} = $tmp1; $cfg::thash{$v1}{$v2} = $tmp2; $cfg::thash{$v1}{$v2}{server} ='b.a.com'; print "$cfg::thash{$v1}{$v2}{server}\n"; + $tmp1 = {}; share($tmp1); $tmp2 = {}; share($tmp2); + $v1='A'; $v2='1'; $cfg::thash{$v1} = $tmp1; $cfg::thash{$v1}{$v2} = $tmp2; $cfg::thash{$v1}{$v2}{server} ='a.a.com'; print "$cfg::thash{$v1}{$v2}{server}\n"; print "$cfg::thash{'B'}{'2'}{server}\n"; + $tmp1 = {}; share($tmp1); $tmp2 = {}; share($tmp2); $v1='C'; $v2='3'; $cfg::thash{$v1} = $tmp1; $cfg::thash{$v1}{$v2} = $tmp2; $cfg::thash{$v1}{$v2}{server} ='c.a.com'; + } for $key1 (sort keys %cfg::thash) { print "ISP:$key1\t"; for $key2 (sort keys %{$cfg::thash{$key1}} ) { print "RANK:$key2\t"; for $key3 (sort keys %{$cfg::thash{$key1}{$key2}}) { print "$key3 $cfg::thash{$key1}{$key2}{$key3}\n"; } } } + + $thr1 = threads->new(\&sub5, "one"); exit; + sub sub5 { my $thr = $_[0]; my %thash=(); my $key ='test'; while (1) { { lock(%cfg::thash); print "Before: thread $thr: thash = $cfg::thash{'A'}{'1'}{ser +ver}\n"; print "After: thread $thr: thash = $cfg::thash{'B'}{'2'}{serv +er}\n"; # $thash{$key} = dclone($cfg::thash{'B'}); $thash{$key} = clone($cfg::thash{'B'}); } } }
Re: Using clone/dclone with threads
by exussum0 (Vicar) on Nov 23, 2004 at 12:12 UTC
    BrowserUK said everything, 'cept, you are trying to imitate fork. :) (though most if not all implementations of fork do a copy on write)

    ----
    Then B.I. said, "Hov' remind yourself nobody built like you, you designed yourself"

      There is another difference.

      You can easily code your programs using threads in (almost) exactly the same way as you would using fork. Simply not sharing the data will cause it to be duplicated(without the COW obviously) and each thread will be able to use it's copy of the data saftly without performing any locking, or even giving it any consideration.

      The advantage comes when each of the 'processes' has reach a result. At that point, it becomes extremely easy for the 'parent process' (originating thread) to gather all the results from its children together and perform further processing -- eg. collating those results -- without requiring a tiresome and dangerous adventure holiday to the mysterious world of Eyepee Sea.


      Examine what is said, not who speaks.
      "But you should never overestimate the ingenuity of the sceptics to come up with a counter-argument." -Myles Allen
      "Think for yourself!" - Abigail        "Time is a poor substitute for thought"--theorbtwo         "Efficiency is intelligent laziness." -David Dunham
      "Memory, processor, disk in that order on the hardware side. Algorithm, algorithm, algorithm on the code side." - tachyon
Re: Using clone/dclone with threads
by Anonymous Monk on Nov 23, 2004 at 23:05 UTC
    I'll explain why I was trying to make local copies.

    Basically, I've got an HoHoH which contains lots of records, unique key at top-level, so each record appears once & only once in the struct.
    There are multiple threads processing these records, but at any given time a given record will only be being processed by exactly 1 thread, except when I need to pass a record to another thread, which is rare.

    Having read the Perl docs etc, it seems that
    1. you can only lock the entire HoHoH, not sub-levels
    2. you have to lock the entire HoHoH to make an update

    For me, the records will be being updated frequently, but as I say, there will be no clash of trying to update the same record from 2 different threads. If I have to lock the HoHoH for each update, it'll bottleneck badly.
    OTOH, if non-clashing updates are safe without locking, that would be great.

      Replying to myself: Actually, having just re-read perlthrtut, I noticed this one line in the middle:

      Note that a shared variable guarantees that if two or more threads try to modify it at the same time, the internal state of the variable will not become corrupted. However, there are no guarantees beyond this, as explained in the next section.

      The thing I'm still worried about is whether this applies completely to multi-level struct like my HoHoH, as BrowserUK and I have discussed limitations on this in another thread recently.

Re: Using clone/dclone with threads
by Anonymous Monk on Sep 20, 2011 at 18:14 UTC
    I think you need to do this $cfg::sessions{A}||={}; before clone/dclone.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://409471]
Approved by ysth
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others making s'mores by the fire in the courtyard of the Monastery: (7)
As of 2024-03-28 13:55 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found