I agree with Corion, although if you are running on Windows, I strongly suggest avoiding Parallel::ForkManager and instead use threads, along with Thread::Queue to manage your inputs and outputs.

The way I have tackled this type of scenario is to have one queue with all the input files listed, a set of processing threads, which pick from that queue, process the files and "print" the output to another queue. A final thread is responsible for picking from the output queue and writing directly to file without performing any processing. Its a simple model that re-uses existing threads, reducing the overhead of threading your app.

Sample code:

use strict; use warnings; use threads; use Thread::Queue; use constant MAXTHREADS => 1; # Update this to the maximum number of t +hreads you want. Keep it below the number of physical cores available + on your machine. my @inputFiles = qw(); # Do something to populate your list of inputs my $inputQueue = Thread::Queue->new( @inputFiles ); my $printQueue = Thread::Queue->new(); $inputQueue->end(); # Tell everyone that the inputQueue wont have anyt +hing else added to it my $printThread = threads->new( \&printWorker ); my @workerThreads = map { threads->new( \&worker ) } 1..MAXTHREADS; $_->join for @workerThreads; $printQueue->end(); # Tell everyone that the printQueue wont have anyt +hing else added to it $printThread->join(); ###### END OF MAIN ###### sub worker { # Add code to prepare for individual file processing # This function should be the equivalent of your (working) serial +process while ( defined( my $file = $inputQueue->dequeue() ) ) { # Add code here to process individual files $printQueue->enqueue( 'output' ); # Instead of printing, enque +ue output data to the print queue } } sub printWorker { my $outFile = "myOutFile"; # Set name of output file open my $fhOut, ">", $outFile or &{ warn "Failed to open file for +writing: $!"; exit 1; }; # Kill all threads on failure select $fhOut; print $_ while $_ = $printQueue->dequeue(); close $fhOut; }

In reply to Re^2: POE Examples by SimonPratt
in thread POE Examples by kulls

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":



  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.