in reply to Re^4: forking with Storable and IPC::ShareLite
in thread forking with Storable and IPC::ShareLite

Well, with your new script, with the "exit" at the end, it exits before any output can be generated. If I change the "exit" to "<>", so my script waits, I do see a large list saying "data is missing". I'm scratching my head right now, and will let you know if I figure something out.

As an afterthought, when I first tested your original script, I tried to redirect the output to a text file with " script > textfile". I did lose some lines that way on repeated attempts, due to the script exiting before all output was printed. Maybe there is some sort of timing problem, between the ParallelForkManager, the IPC, and the main script. Have you tried putting delays in there, to let one part of the script catch-up with the other part? In my experience with other scripts, where one person sees an error, and another dosn't, it often involves timing differences related to the "speeds" of the different machines. I've even seem cases where scripts will fail when the machine is under high load, but run OK otherwise. Usually, slowing the scripts down, helps. Maybe the IPC through shared memory is "too fast" for the big modules to handle in perfect synchronization?


I'm not really a human, but I play one on earth. flash japh
  • Comment on Re^5: forking with Storable and IPC::ShareLite

Replies are listed 'Best First'.
Re^6: forking with Storable and IPC::ShareLite
by andThenThereWasPERL (Monk) on Feb 03, 2006 at 14:50 UTC
    Thanks for the inputs
    Here is the modified code with wait using IPC::ShareLite- still missing data
    use strict; use Parallel::ForkManager; use IPC::ShareLite; use Storable qw( freeze thaw ); my $fork_manager = new Parallel::ForkManager(5); my $share = new IPC::ShareLite( -key => 1971, -create => 'yes', -destroy => 'yes' ) or die $!; my $hash = {'PARENT' => $$}; $share->store( freeze( $hash ) ); foreach my $child ( 1 .. 10 ) { my $pid = $fork_manager->start($child) and next; $share->destroy( 0 ); my $hash = thaw( $share->fetch ); while (1) { if ($share->lock()) { last; } else { print "$$ cant lock share\n"; sleep 1; } }; for my $id (1 .. 20) { my $key = $child . '-' . $id; $$hash{$key} = qq{|Kid $child pushed $id}; } $share->store( freeze( $hash ) ); $share->unlock; $fork_manager->finish($child); } print "Waiting for Children...\n"; $fork_manager->wait_all_children; my %final_parent_hash; my $sleep_ctr = 10; while ($sleep_ctr) { %final_parent_hash = %{ thaw( $share->fetch ) }; print scalar (keys %final_parent_hash) . "\n"; if ((keys %final_parent_hash) == 201) { last; } sleep 1; $sleep_ctr--; } foreach my $child ( 1 .. 10 ) { for my $id (1 .. 20) { my $key = $child . '-' . $id; if (! exists $final_parent_hash{$key} ) { print "Missing data for Kid $child , data $id\n"; } else { print "$key = $final_parent_hash{$key}\n"; } } }
    Here is the modified code with wait using IPC::Shareable - works like a charm :)
    use strict; use Parallel::ForkManager; use IPC::Shareable; my $glue = $$; my %options = ( create => 1, exclusive => 0, mode => 0644, destroy => 1, ); my %final_parent_hash; my $parent_share = tie %final_parent_hash, 'IPC::Shareable', $glue, { +%options } or die "parent : tie failed\n"; my $fork_manager = new Parallel::ForkManager(5); foreach my $child ( 1 .. 10 ) { my $pid = $fork_manager->start($child) and next; my %options = ( create => 0, exclusive => 0, mode => 0644, destroy => 0, ); my %child_hash; my $child_share = tie %child_hash, 'IPC::Shareable', $glue, { %opt +ions } or die "client: tie failed\n"; for my $id (1 .. 20) { my $key = $child . '-' . $id; $child_share->shlock; $final_parent_hash{$key} = qq{|Kid $child pushed $id}; $child_share->shunlock; } $fork_manager->finish($child); } print "Waiting for Children...\n"; $fork_manager->wait_all_children; my $sleep_ctr = 10; while ($sleep_ctr) { if ((keys %final_parent_hash) == 200) { last; } sleep 1; $sleep_ctr--; } foreach my $child ( 1 .. 10 ) { for my $id (1 .. 20) { my $key = $child . '-' . $id; if (! exists $final_parent_hash{$key} ) { print "Missing data for Kid $child , data $id\n"; } else { print "$key = $final_parent_hash{$key}\n"; } } } IPC::Shareable->clean_up_all;
    I think IPC::ShareLite is the culprit
    Thanks Aveek