Hello all,
I'm new to the forums, so I apologize if this question is a bit simple. I have an array of files stores in the variable @files. For each file, I would like to run a unix-based command line program called FastQC, which generates an output file. I have 5 nodes with 12 cores per node, so I would like to be able to utilize this power to run 50 instances of the FastQC program simultaneously as part of my perl pipeline.

Ideally, I would also like to grab the exit code or output for each instance, since piping the output to a file would add an extra scanning step to ensure that all processes completed successfully - but if I have to do this I will.

The below code runs the processes sequentially, producing the result and an ".out" file with the status of the run: 

for my $file (@files)
{
        my $fastqc_script = "$fastqc_folder/fastqc $input_folder/$file
+ --outdir=$fastqc_output_folder > $fastqc_output_folder/$file.out";
        `$fastqc_script`;
}

[download]
The goal would be primarily to start the second run immediately after starting the first one, etc. The secondary goal would be to collect the exit codes in an array as they complete. Any help would be greatly appreciated. Thank you very much in advance for your help.

In reply to Execution of parallel unix system commands by sousuffer

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.