I agree with Corion, although if you are running on Windows, I strongly suggest avoiding Parallel::ForkManager and instead use threads, along with Thread::Queue to manage your inputs and outputs.
The way I have tackled this type of scenario is to have one queue with all the input files listed, a set of processing threads, which pick from that queue, process the files and "print" the output to another queue. A final thread is responsible for picking from the output queue and writing directly to file without performing any processing. Its a simple model that re-uses existing threads, reducing the overhead of threading your app.
Sample code:
use strict; use warnings; use threads; use Thread::Queue; use constant MAXTHREADS => 1; # Update this to the maximum number of t +hreads you want. Keep it below the number of physical cores available + on your machine. my @inputFiles = qw(); # Do something to populate your list of inputs my $inputQueue = Thread::Queue->new( @inputFiles ); my $printQueue = Thread::Queue->new(); $inputQueue->end(); # Tell everyone that the inputQueue wont have anyt +hing else added to it my $printThread = threads->new( \&printWorker ); my @workerThreads = map { threads->new( \&worker ) } 1..MAXTHREADS; $_->join for @workerThreads; $printQueue->end(); # Tell everyone that the printQueue wont have anyt +hing else added to it $printThread->join(); ###### END OF MAIN ###### sub worker { # Add code to prepare for individual file processing # This function should be the equivalent of your (working) serial +process while ( defined( my $file = $inputQueue->dequeue() ) ) { # Add code here to process individual files $printQueue->enqueue( 'output' ); # Instead of printing, enque +ue output data to the print queue } } sub printWorker { my $outFile = "myOutFile"; # Set name of output file open my $fhOut, ">", $outFile or &{ warn "Failed to open file for +writing: $!"; exit 1; }; # Kill all threads on failure select $fhOut; print $_ while $_ = $printQueue->dequeue(); close $fhOut; }
In reply to Re^2: POE Examples
by SimonPratt
in thread POE Examples
by kulls
| For: | Use: | ||
| & | & | ||
| < | < | ||
| > | > | ||
| [ | [ | ||
| ] | ] |