ovedpo15 has asked for the wisdom of the Perl Monks concerning the following question:

Hello,
I'm trying to make my script more efficient so I decided to learn how to use fork.
This question is more on how to write it nice and clean
I have about 13 different calls that create files. Those calls are not depended and they can run parallel:
create_file('A'); create_file('B'); create_file('C'); create_file('D'); create_file('E'); ...
I understand how to create a fork and how to check if there is one and why it is important to use the wait function.
But I don't know how to make it look nice, clean and readable. Should I create a special function which will be called every time I need to create X (for example 5) forks? How should I keep track of those forks? If for example, I have 13 things I can run parallel, should I use 13 forks (running on a strong machine)? Meaning how to determine how many forks to use?
I would really love to see some real example with for example 13 different calls.

Replies are listed 'Best First'.
Re: Learning to use fork()
by markong (Pilgrim) on Jan 14, 2019 at 23:38 UTC
      Thanks for the reply. I prefer to use the fork method. My main question is what to do if the fork was not success? I still want the reports to be created even if not parallel. Let start with 5 functions:
      sub1(); sub2(); sub3(); sub4(); sub5();
      I would like to loop through the functions (is it even possible?) and create a fork for each one. What should I do if the fork failed?

        Here is an example way to do what you want for educational purposes (Parallel::ForkManager is really the way to go in production code).

        It is possible to loop through functions by taking references with the \& syntax:
        sub sub1 { print 'sub1 ran' } sub sub2 { print 'sub2 ran' } my @refs = (\&sub1, \&sub2); $_->() foreach @refs;
        Fork failure is indicated by an undef return value by the fork function. If you want to go on with processing in a non-parallel manner when fork fails, you can just check if it's defined. Full example:
        use strict; use warnings; use POSIX ':sys_wait_h'; my @subs_to_run = ( \&sub1, \&sub2, \&sub3, \&sub4, \&sub5, ); foreach my $sub (@subs_to_run) { my $is_parent = fork; # try to fork next if $is_parent; # both child and fork failure will be false $sub->(); # we're in the child or fork failed, run the sub exit if defined $is_parent; # exit only if actually forked } 1 while waitpid(-1, WNOHANG) > 0; # wait for all children

        Recover or die:

        if ($pid = fork) { # parent } else { die "cannot fork: $!" unless defined $pid; # child }

        I prefer to use the fork method.

        you mean thats what your homework said you should do

Re: Learning to use fork()
by Eily (Monsignor) on Jan 15, 2019 at 09:35 UTC

    Benchmarking is often (always?) the best way to tell which of two implementations is the fastest. Something like:

    sub create_parallel { ... # code here to create files using fork } sub create_loop { ... # code here to create files in a simple for loop } cmpthese(-5, parallel => \&create_parallel, loop => \&create_loop, );
    The -5 means that the subs will be run several times until 5 seconds of execution (each) are reached, so it might be a good idea to include cleanup in the functions.

    I doubt you will find a significant difference though, or the parallel version might be the slowest. Unless you are writing 13 files on 13 different drives, or doing a lot of networking operations in each thread, each thread will basically just wait for the previous one to stop using whatever device you are writing on.

Re: Learning to use fork()
by tybalt89 (Monsignor) on Jan 16, 2019 at 06:54 UTC

    Limits max forks at one time.

    #!/usr/bin/perl # https://perlmonks.org/?node_id=1228560 use strict; use warnings; use Time::HiRes qw( sleep ); my $maxforks = 5; # at any one time my $active = 0; for my $param ( 'A' .. 'M' ) { $active -= wait > 0 while $active >= $maxforks; if( my $pid = fork ) { $active++; print "active forks: $active\n"; } else { create_file($param); defined $pid and exit; } } $active -= wait > 0 while $active; sub create_file { sleep 0.001 + rand 1; my $parameter = shift; warn "pid $$ create_file('$parameter')\n"; # do somethng }
Re: Learning to use fork()
by ovedpo15 (Pilgrim) on Jan 15, 2019 at 01:14 UTC
    Some progress:
    my $forks = 0; foreach my $i (1..3) { if ($i == 1 && fork() == 0) { $forks++; print "Running first with $$ \n"; call_sub1(); call_sub2(); call_sub3(); call_sub4(); call_sub5(); call_sub6(); exit; } elsif ($i == 2 && fork() == 0) { $forks++; print "Running second $$ \n"; call_sub1(); call_sub10(); call_sub3(); call_sub13(); call_sub5(); exit; } else { if (fork() == 0) { $forks++; print "Running third with $$ \n"; call_sub1(); call_sub4(); call_sub5(); call_sub15(); exit; } } } for (1 .. $forks) { my $pid = wait(); print "Parent saw $pid exiting\n"; } print "done\n";
    It really feels messy and not clear. Also, it is not the most efficient because ALL of the 15 calls are not depend one on another. with that logic I can create 15 if-else statements but of course it is not readable and clear. Also, the names are not the same (and all of them gets arguments - I removed them so it will be more easy to see). How should I solve it? Also, should I kill the child process when its done by using exit?
      It really feels messy and not clear.

      fork is fraught in perl without it really being perl's fault. I had recently endeavored to do some multitasking project that way and became convinced that the implementation is not worth the trade-offs, the first being that it must occur in an eval/die pair. My source looks about like yours, and my results were just about as unsatisfying as well. This is about where I dealt with it first: Re^2: using online translation engines with perl.

      Also, should I kill the child process when its done by using exit?

      There were a couple posts on that thread that showed how to kill the processes. I used a different way to get done what I needed.

      It really feels messy and not clear

      11_42

      You're really asking how to clean 11_42? 11_42 you say? What is 11_42? Exactly!

      1 use Proc::Background

      #!/usr/bin/perl -- use strict; use warnings; use Proc::Background; my @jobs = ( [ $^X, 'job1.pl' ], [ $^X, 'job2.pl' ], [ $^X, 'job3.pl' ], [ $^X, 'job4.pl' ], [ $^X, 'job5.pl' ], [ $^X, 'job6.pl' ], [ $^X, 'job7.pl' ], [ $^X, 'job8.pl' ], [ $^X, 'job9.pl' ], [ $^X, 'job10.pl' ], [ $^X, 'job11.pl' ], [ $^X, 'job12.pl' ], [ $^X, 'job13.pl' ], ); my @procs = map { Proc::Background->new( @$_ ) } @jobs; while( @procs ){ @procs = { $_->alive ? $_ : ItsOverMaybeRestart($_) } map @procs; sleep 1; } sub ItsOverMaybeRestart { my( $proc ) = @_; ... return; }
        @procs = { $_->alive ? $_ : ItsOverMaybeRestart($_) } map @procs;

        I don't know anything about Proc::Background, but shouldn't that be something like

        @procs = map { $_->alive ? $_ : ItsOverMaybeRestart($_) } @procs;
        ?


        Give a man a fish:  <%-{-{-{-<