Hi Bloehdian,

Re: I want to share an array between two processes created by fork()

The following demonstration provides a cross platform solution. MCE::Shared supports Unix (e.g: AIX, Darwin, FreeBSD, Linux, NetBSD, OpenBSD, Solaris, and possibly others), Windows (ActiveState/Strawberry Perl), and Cygwin. Also supported is Perl included with git-shell for Windows.

use strict;
use warnings;

use MCE::Shared;

tie my @data, 'MCE::Shared';

if ( ! defined ( my $pid = fork() ) ) {
   die "Cannot fork!: $!";
}
elsif ( $pid == 0 ) {
   # Child
   my $item;
   sleep 1;

   for ( my $i = 1; $i <= 10; $i++ ) {
      $item = shift @data;
      $item = '' if ( ! defined $item );

      print "Child: $item\n";
   }
}
else {
   # Parent
   for ( my $i = 1; $i <= 10; $i++ ) {
      push @data, $i;
      print "Parent: @data\n";
   }

   waitpid( $pid, 0 );
}
[download]

The snippet produces the following output.

Parent: 1
Parent: 1 2
Parent: 1 2 3
Parent: 1 2 3 4
Parent: 1 2 3 4 5
Parent: 1 2 3 4 5 6
Parent: 1 2 3 4 5 6 7
Parent: 1 2 3 4 5 6 7 8
Parent: 1 2 3 4 5 6 7 8 9
Parent: 1 2 3 4 5 6 7 8 9 10
Child: 1
Child: 2
Child: 3
Child: 4
Child: 5
Child: 6
Child: 7
Child: 8
Child: 9
Child: 10
[download]

Regards, Mario.

Calling MCE::Shared->init() is recommended whenever running many workers simultaneously. This is a one-time call which makes a request to the shared-manager process for a data channel to use for IPC. Init is called automatically by MCE workers so not necessary at the application level.

Init is helpful for extra performance and applies to running with many workers only.

elsif ( $pid == 0 ) {

   # Child
   MCE::Shared->init();

   ...
}
[download]

Regards, Mario.

Hello Mario,

THX for the reply.

I checked the documentation on CPAN, but it is not totally clear to me, how I would implement locking with this module. In the actual application, both processes will manipulate @data.

Cheers

Bloehdian

I was able to run your code++ and replicate your results on Active State Windows, 5.20.
I did find this link: Many-Core Engine for Perl providing parallel processing capabilities. MCE is new to me. IPC::Shareable is not available on Active State Windows.

Hello Marshall,

Supporting the Windows platform took many attempts and am embarrassed to say how many hours spent just so that MCE::Shared works there too. What is cool about MCE::Shared is the ability to share data no matter how workers are spawned (e.g. via threads, via fork, MCE, MCE::Hobo, Parallel::ForkManager, and possibly other parallel modules).

The MCE module supports Perl 5.8 and above whereas MCE::Shared requires Perl 5.10.1. This is the reason the two modules are separated. There are still users out there wanting MCE to run on Perl 5.8 minimally (e.g. App::Netdisco, Test::Perl::Critic).

Regards, Mario.

After writing a solution using MCE::Shared, I wanted to come back and give IPC::Shareable a try including locking. Here it is.

use strict;
use warnings;

use IPC::Shareable;

my ( $glue, $options ) = ( 'data', {
   create    => 1,
   exclusive => 0,
   mode      => 0644,
   destroy   => 1,
});

tie my @data, 'IPC::Shareable', $glue, $options;

if ( ! defined ( my $pid = fork() ) ) {
   die "Cannot fork!: $!";
}
elsif ( $pid == 0 ) {
   # Child
   my $item;
   sleep 1;

   for ( my $i = 1; $i <= 10; $i++ ) {
      ( tied @data )->shlock;
      $item = shift @data;
      ( tied @data )->shunlock;

      $item = '' if ( ! defined $item );
      print "Child: $item\n";
   }
}
else {
   # Parent
   for ( my $i = 1; $i <= 10; $i++ ) {
      ( tied @data )->shlock;
      push @data, $i;
      ( tied @data )->shunlock;

      print "Parent: @data\n";
   }

   waitpid( $pid, 0 );
}
[download]

It produces the following output on FreeBSD, Linux, and Mac OS X.

Parent: 1
Parent: 1 2
Parent: 1 2 3
Parent: 1 2 3 4
Parent: 1 2 3 4 5
Parent: 1 2 3 4 5 6
Parent: 1 2 3 4 5 6 7
Parent: 1 2 3 4 5 6 7 8
Parent: 1 2 3 4 5 6 7 8 9
Parent: 1 2 3 4 5 6 7 8 9 10
Child: 1
Child: 2
Child: 3
Child: 4
Child: 5
Child: 6
Child: 7
Child: 8
Child: 9
Child: 10
[download]

Notice how the construction for the shared variable is done before spawning. Also, the parent waits for the spawned worker to exit. Other than locking (optional for this use-case due to sleep in worker), everything else is similar to the OP's code.

Regards, Mario.

The following is a comparison demonstrating 2 workers incrementing a shared counter variable. The OO interface for MCE::Shared allows for zero-locking at the application level. All 3 snippets involve set and get while incrementing the variable.

1) ipc_shareable.pl

use strict;
use warnings;

use IPC::Shareable;

my $options = {
   create    => 1,
   exclusive => 0,
   mode      => 0644,
   destroy   => 1,
};

tie my $cntr, 'IPC::Shareable', $options;
my $val;

$cntr = 0;

if ( ! defined ( my $pid = fork() ) ) {
   die "Cannot fork!: $!";
}
elsif ( $pid == 0 ) {
   # Child
   for ( 1 .. 10000 ) {
      ( tied $cntr )->shlock;
      $val = $cntr++;
      ( tied $cntr )->shunlock;
   }
}
else {
   # Parent
   for ( 1 .. 10000 ) {
      ( tied $cntr )->shlock;
      $val = $cntr++;
      ( tied $cntr )->shunlock;
   }

   waitpid $pid, 0;

   print "counter: ", $cntr, "\n";
}
[download]

2) mce_shared.pl

use strict;
use warnings;

use MCE::Mutex;
use MCE::Shared;

my $mutex = MCE::Mutex->new();
tie my $cntr, 'MCE::Shared', 0;
my $val;

if ( ! defined ( my $pid = fork() ) ) {
   die "Cannot fork!: $!";
}
elsif ( $pid == 0 ) {
   # Child
   for ( 1 .. 10000 ) {
      $mutex->lock;
      $val = $cntr++;
      $mutex->unlock;
   }
}
else {
   # Parent
   for ( 1 .. 10000 ) {
      $mutex->lock;
      $val = $cntr++;
      $mutex->unlock;
   }

   waitpid( $pid, 0 );

   print "counter: ", $cntr, "\n";
}
[download]

3) mce_shared_oo.pl

use strict;
use warnings;

use MCE::Shared;

my $cntr = MCE::Shared->scalar( 0 );
my $val;

if ( ! defined ( my $pid = fork() ) ) {
   die "Cannot fork!: $!";
}
elsif ( $pid == 0 ) {
   # Child
   for ( 1 .. 10000 ) {
      $val = $cntr->incr;
   }
}
else {
   # Parent
   for ( 1 .. 10000 ) {
      $val = $cntr->incr;
   }

   waitpid( $pid, 0 );

   print "counter: ", $cntr->get, "\n";
}
[download]

Results

time in seconds  :  FreeBSD  Linux  Mac OSX

ipc_shareable.pl :   1.80    1.87    1.79
mce_shared.pl    :   0.68    0.97    0.58
mce_shared_oo.pl :   0.38    0.73    0.41

counter: 20000
[download]

Fetches require more time on Linux than BSD-based OS in regards to sockets. This is seen not only with Perl/MCE::Shared but also with other languages. I'm not sure why.

MCE::Shared is fully wantarray-aware and performs fetches only when requested. Sadly, that is not the case via the TIE interface. Perhaps, this is an unnecessary overhead IMHO in regards to the TIE implementation inside Perl.

The following results were obtained by omitting fetches: e.g. $cntr++, $cntr->incr. Fortunately, stores perform similarly between BSD and Linux via the OO interface.

time in seconds  :  FreeBSD  Linux  Mac OSX

ipc_shareable.pl :   1.77    1.86    1.61
mce_shared.pl    :   0.68    0.95    0.58
mce_shared_oo.pl :   0.19    0.17    0.16

counter: 20000
[download]

The above is only a glimpse of what is possible with MCE::Shared. The use of parallel data-channels (enabled automatically) for MCE, MCE::Hobo, and threads further reduce latency for the shared-manager process. Calling MCE::Shared->init(), inside the worker, enables the optimization for fork and other parallel modules.

This has been an interesting exercise. Thank you Bloehdian for the initial post.

Tests were conducted on a Macbook Pro laptop (Haswell architecture) OS X 10.11.6 running at 2.6 GHz 1600 MHz RAM. FreeBSD and Linux are virtual machines managed by Parallels Desktop 11.2.1.

Regards, Mario.

Anyway, from looking at your code, the first thing that jumps out at me is: where are the locks? If you have shared memory, your processes must implement some form of locking otherwise you have a mess where 2 folks are trying to modify the same structure at once. It is up to the user to implement these locks between processes. IPC:Shareable shows the shlock() and shunlock() functions.  shift @data; is a "read-modify-write" operation.

Update: Oh, I see I guess you are figuring that the sleep(1) in the child is enough?
From looking at the example in the doc's, I think you need different values for "create" and "destroy" between the 2 processes. Why is tie commented out in the child? Shouldn't the sleep 1 go before the tie to allow time for the parent to create this shared array in the first place? I would simplify things and replace the for loop in the child with a simple, print "@data";.