hotel has asked for the wisdom of the Perl Monks concerning the following question:

Hi all,

Lately I've decided to use ForkManager to harvest the whole computing power available to me and to fasten things. But either I got the whole idea wrong or I cannot code it properly.

I have a script...

...parse a large file (a few hundred MBs) and store some stuff in a hash called H

As H gets populated more memory is used.

Next, I want to pass the reference of H to a subroutine where I want to make a system call with each and every key of H. The outputs of the software called with these system calls are independent of each other. They are basically output files. When only one processor is used to do this, it takes a lot of time, so I'd like you use all 16 cores to initiate 16 system calls simultaneously.

Among other things, here's what I recently tried.

parse a file and collect data into H
pass its reference to Sub

sub Sub {
    
    my $Href = @_;
    my $n_processes = 16;
    my $pm = new Parallel::ForkManager($n_processes); 

    foreach my $key (keys %{$Href}) {
        
        my $var = $key->attribute;
        my $cmd = "SomeBinary ".$var;
        $pm->start and next; # do the fork
        system($cmd);
        $pm->finish;
    }
    pm->wait_for_all_children;
}

[download]

When I use top in terminal to have a look at the usage, I see that my Perl script Script started 16 processes of the same name Script, each using the same amount of memory. The amount of memory is the same with what was needed during populating H. So, obviously I'm doing something very wrong here because I was expecting to see one Script and 16 SomeBinary, each SomeBinary consuming as much memory as it requires (which should be very little). But it seems like I'm creating copies of the Script perl process.

What am I getting/doing wrong? Many thanks.

Replies are listed 'Best First'.
Re: Parallel::ForkManager Memory issue
by RichardK (Parson) on Mar 17, 2013 at 00:24 UTC

    Fork makes a copy of the current running perl program and then you make your system call. That child has to wait around for the system call to finish which is why you are seeing 16 copies, and you should also see the 16 commands running.

    BUT there is no reason to panic, as fork is designed to be efficient as possible and will share pages where possible.

    The help says :-

    On most systems supporting fork(), great care has gone into making it 
+extremely efficient 
(for example, using copy-on-write technology on data pages), making it
+ the dominant paradigm for multitasking over the last few decades. )

    [download]

    Look at the total memory usage which is displayed on the first few lines of the top output and you'll see that there's plenty of memory really.

[reply]
[d/l]

      Hi,

      Thanks for the answer. You're right, in addition to 16 Script.pls I also see 16 SomeBinarys running. On the other hand, I do think there's a memory issue going on as I tried to explain in my previous post. The error message I get after running the script for a while is:

      Uncaught exception from user code:
    Cannot fork: Cannot allocate memory at /path/to/Parallel/ForkManag
+er.pm line 476.

      [download]
[reply]
[d/l]
Re: Parallel::ForkManager Memory issue
by BrowserUk (Patriarch) on Mar 17, 2013 at 03:39 UTC

    What is this line of code doing? my $var = $key->attribute;

    I ask, because it looks like it is calling a method attribute on an object $key; but that is impossible.

    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
[d/l]
[select]

      Hi,

      I think it is possible when the hash H is declared as

      tie my %H, 'Tie::RefHash';

      since you are then able store objects as hash keys

[reply]
[d/l]
        H is declared as tie my %H, 'Tie::RefHash';

        Ah! No wonder you are running out of memory.

        With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
Re: Parallel::ForkManager Memory issue
by aitap (Curate) on Mar 17, 2013 at 08:57 UTC
[reply]
[d/l]
[select]

      Changing the system($cmd) to exec($cmd) seems to work! However I get the following warning

      Statement unlikely to be reached at
    /path/to/Script.pl line 653 (#1)
    (W exec) You did an exec() with some statement after it other than
+ a
    die().  This is almost always an error, because exec() never retur
+ns
    unless there was a failure.  You probably wanted to use system()
    instead, which does return.  To suppress this warning, put the exe
+c() in
    a block by itself.
    
    (Maybe you meant system() when you said exec()?)

      [download]

      Also, when I look at the top output, I do not see 16 but around 2-3 SomeBinarys running. I don't really know why I don't see 16 of them, but looking at how fast the corresponding output files are generated, it actually looks like there are 16 running. Maybe they are spawned and finished so fast that I do not see all 16 at the same time..? Also, some of them out of these SomeBinarys are shown as SomeBinary <defunct> . But maybe those are the SomeBinarys that are about to finish or something..?

[reply]
[d/l]
[select]

        Perl doesn't know that you intended to replace the running program with another process, thus the warning. The message says everything itself; you can try something like { exec($cmd) or die "exec: $!" } to supress the it properly.

        <defunct> processes are zombies. Zombie is a kind of process which had finished running, but its parent has not handled SIGCHLD yet. This is strange; Parallel::ForkManager should handle these situations. To kill a zombie you simply need to kill its parent process (or wait for it to finish).

        Sorry if my advice was wrong.
[reply]
[d/l]
[select]
Re: Parallel::ForkManager Memory issue
by Khen1950fx (Canon) on Mar 17, 2013 at 17:06 UTC
    I just noticed a typo: 
    pm->wait_for_all_children

    [download]
    It should be: 
    $pm->wait_all_children

    [download]
[reply]
[d/l]
[select]
      True. I corrected it after I noticed.
[reply]

Back to Seekers of Perl Wisdom