in reply to Perl forked processes and variable sharing

Hi fireblood,

First, a kind correction to perlfan's comment about MCE. All the methods in MCE and MCE::Shared that pertain to communication are IPC-based. That has been the case since version 1.0. There are examples out there where workers write data to /dev/shm or /tmp and pass the path via IPC to the manager process i.e. data being quite large.

I made a tiny change to your demonstration. MCE::Shared::Server handles requests transparently to and from the shared-manager process, where the data resides. This works as one would expect it to.

$ diff ex1.pl ex2.pl 2a3 > use MCE::Shared; 4c5 < my $string = "First value"; --- > tie my $string, "MCE::Shared", "First value";

That's a tiny glimpse of what MCE::Shared is capable.

OOP Usage

If you're having to fetch or store several thousands or millions of times, then OOP usage involves lesser overhead.

use strict; use warnings; use MCE::Shared; my $string = MCE::Shared->scalar("First value"); sub do_subroutine { print "I am the subroutine and my PID is $$\n"; $string->set("Second value"); } print "I am the invoking process and my PID is $$\n"; print "The value of \$string is ", $string->get(), "\n"; unless (fork) { do_subroutine (); exit; } my $waited_pid = wait; print "The PID of the child upon whom I waited was $waited_pid\n"; print "The value of \$string is ", $string->get(), "\n"; __END__ I am the invoking process and my PID is 22725 The value of $string is First value I am the subroutine and my PID is 22727 The PID of the child upon whom I waited was 22727 The value of $string is Second value

String Class demo 1

One is not forced to using various data modules that ship with MCE::Shared. Next is a demonstration using a custom string class. A shared object is created afterwards.

use strict; use warnings; use MCE::Shared; package My::String; sub new { my ($class, $string) = @_; return bless \$string, $class; } sub set { my ($self) = @_; $$self = $_[1]; return 1; } sub get { my ($self) = @_; return $$self; } package main; my $string = MCE::Shared->share( { module => "My::String" }, "First value" ); sub do_subroutine { print "I am the subroutine and my PID is $$\n"; $string->set("Second value"); } print "I am the invoking process and my PID is $$\n"; print "The value of \$string is ", $string->get(), "\n"; unless (fork) { do_subroutine (); exit; } my $waited_pid = wait; print "The PID of the child upon whom I waited was $waited_pid\n"; print "The value of \$string is ", $string->get(), "\n"; __END__ I am the invoking process and my PID is 22978 The value of $string is First value I am the subroutine and my PID is 22980 The PID of the child upon whom I waited was 22980 The value of $string is Second value

String Class demo 2

Here sharing is instantiated inside the string class.

use strict; use warnings; use MCE::Shared; package My::String; sub new { my ($class) = @_; my $string = MCE::Shared->scalar($_[1]); return bless [ $string ], $class; } sub set { my ($self) = @_; $self->[0]->set($_[1]); return 1; } sub get { my ($self) = @_; return $self->[0]->get(); } package main; my $string = My::String->new("First value"); sub do_subroutine { print "I am the subroutine and my PID is $$\n"; $string->set("Second value"); } print "I am the invoking process and my PID is $$\n"; print "The value of \$string is ", $string->get(), "\n"; unless (fork) { do_subroutine (); exit; } my $waited_pid = wait; print "The PID of the child upon whom I waited was $waited_pid\n"; print "The value of \$string is ", $string->get(), "\n"; __END__ I am the invoking process and my PID is 23089 The value of $string is First value I am the subroutine and my PID is 23091 The PID of the child upon whom I waited was 23091 The value of $string is Second value

Replies are listed 'Best First'.
Re^2: Perl forked processes and variable sharing (update)
by marioroy (Prior) on Feb 07, 2022 at 02:58 UTC

    I met to add that MCE::Hobo and MCE::Shared provide capabilities similar to threads and threads::shared respectively. One isn't required to use MCE workers. In fact, MCE::Shared works also with threads, Parallel::ForkManager, and workers spawned with fork.

    If using non-MCE workers, then call MCE::Shared->init inside the child. That will spread IPC communication across 10 data channels. The channel assignment is done automatically for threads and MCE workers (i.e. MCE, MCE::Child, and MCE::Hobo).

    unless (fork) { # This assigns the worker 1 of 10 data channels in a # round-robin fashion for the life of the worker. MCE::Shared->init(); do_subroutine (); exit; }

    Update: Added demonstrations.

    Below, please find two demonstrations which one may run for comparison. Parallel data channels are helpful especially when a reply is not needed i.e. incrementing a value or writing to a shared output object. Each worker increments the value 100,000 times.

    use strict; use warnings; use MCE::Shared; use Time::HiRes 'time'; my $number = MCE::Shared->scalar(0); sub do_subroutine { $number->incr for 1..1e5; } my @pids; my $start = time; for (1..10) { my $pid; unless ($pid = fork) { do_subroutine (); exit; } push @pids, $pid if defined $pid; } waitpid $_, 0 for @pids; printf "number %d\n", $number->get; printf "seconds %.03f\n", time - $start; __END__ number 1000000 seconds 5.234

    That's quite fast considering incrementing the value involves IPC. Faster is possible if you want non-MCE workers to reach levels capable of threads and MCE workers which call MCE::Shared->init automatically.

    MCE::Shared->init

    Calling MCE::Shared->init is beneficial for non-MCE workers.
    use strict; use warnings; use MCE::Shared; use Time::HiRes 'time'; my $number = MCE::Shared->scalar(0); sub do_subroutine { $number->incr for 1..1e5; } my @pids; my $start = time; for (1..10) { my $pid; unless ($pid = fork) { MCE::Shared->init; # enables multi-channel do_subroutine (); exit; } push @pids, $pid if defined $pid; } waitpid $_, 0 for @pids; printf "number %d\n", $number->get; printf "seconds %.03f\n", time - $start; __END__ number 1000000 seconds 2.086

    That is shy of 500k per second on Clear Linux.

    threads demonstration

    For completeness adding threads and MCE::Hobo demonstrations. They benefit from multi-channel IPC automatically and not necessary to call MCE::Shared->init.

    use strict; use warnings; use threads; use MCE::Shared; use Time::HiRes 'time'; my $number = MCE::Shared->scalar(0); my $start = time; sub do_subroutine { $number->incr for 1..1e5; } threads->create('do_subroutine') for 1..10; $_->join for threads->list; printf "number %d\n", $number->get; printf "seconds %.03f\n", time - $start; __END__ number 1000000 seconds 2.142

    MCE::Hobo demonstration

    use strict; use warnings; use MCE::Hobo; use MCE::Shared; use Time::HiRes 'time'; my $number = MCE::Shared->scalar(0); my $start = time; sub do_subroutine { $number->incr for 1..1e5; } MCE::Hobo->create('do_subroutine') for 1..10; $_->join for MCE::Hobo->list; # same as MCE::Hobo->wait_all printf "number %d\n", $number->get; printf "seconds %.03f\n", time - $start; __END__ number 1000000 seconds 2.087