Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

What is the best way to run several commands simultaneously? I have a script that calls many other scripts which load different files into arrays, they run sequentially which takes several minutes, ie:
@getfile1 = qq/getS1config.pl/; open (FILE1, "@getfile1|" ); @file1 = <FILE1>; @getfile2 = qq/getS2config.pl/; open (FILE2, "@getfile2|" ); @file2 = <FILE2>; @getfile3 = qq/getS3config.pl/; open (FILE3, "@getfile3|" ); @file3 = <FILE3>;
I was reviewing fork and threads documentation, but am not sure if and how they should be used. Thank you for help.

Replies are listed 'Best First'.
Re: simultaneous external script execution
by BrowserUk (Patriarch) on Jan 28, 2007 at 06:12 UTC

    Here's one simple way.

    use threads; use threads::shared; my @file1 : shared; my $t1 = async{ my $getfile1 = qq/getS1config.pl/; open (FILE1, "$getfile1 |" ); @file1 = <FILE1>; }; my @file2 : shared; my $t2 = async{ my $getfile2 = qq/getS2config.pl/; open (FILE2, "$getfile2|" ); @file2 = <FILE2>; }; my @file3 : shared; my $t3 = async{ my $getfile3 = qq/getS3config.pl/; open (FILE3, "$getfile3 |" ); @file3 = <FILE3>; }; $_->join for $t1, $t2, $t3; ## Do stuff with the arrays.

    Note that assigning single scalars into arrays in order to interpolate those scalars into strings is a very dubious practice. You should also be checking your opens and much else. You will have to use lexical variables to use :share

    This is a minimal demonstration that could be much improved.


    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
Re: simultaneous external script execution
by blazar (Canon) on Jan 28, 2007 at 11:07 UTC
    What is the best way to run several commands simultaneously? I have a script that calls many other scripts which load different files into arrays, they run sequentially which takes several minutes, ie:

    First of all, are you sure that you will have an advantage parallelizing these tasks? The fact that you write that your other scripts "load different files into arrays" suggests, without further specifications, that these files are local ones, and if so then parallelizing won't buy you much.

    Also, usage of single element arrays for interpolating and unchecked opens apart, your code is not much different from a series of do EXPR statements, and although I'm not particularly fond of this technique, it is at least less verbose, so you may want to go that way instead.

    One particular problem that I see with your code is that you have these getS1config.pl, getS2config.pl, etc. scripts which bear very similar names: I know that a priori this means nothing and that they could actually be wildly different, but isn't it that by any chance they're very similar instead and could be factored away in some code that you may include in your main script, without the overhead of having to start additional processes, and in particular perl interpreters?

Re: simultaneous external script execution
by zentara (Cardinal) on Jan 28, 2007 at 11:32 UTC
    The general rule of thumb is use threads if you need to share data between scripts in realtime, otherwise use fork. Parallel::ForkManager is widely recommended. Google for examples.

    I'm not really a human, but I play one on earth. Cogito ergo sum a bum