in reply to Re^4: Wait for individual sub processes
in thread Wait for individual sub processes [SOLVED]

The concern for memory utilization is valid. The file content is not gathered in the MCE example. Only the chunk id and file path are gathered. The parted content remains inside the output directory until ordered items arrive prior to being merged and unlinked.

... $mce->gather($chunk_id, "$part.out"); ...

The upcoming MCE 1.7 release adds an await method to MCE::Queue. I will demonstrate the gathering of $chunk_id and "$part.out" to a queue and have workers block temporarily in a new MCE::Cookbook.pod. The idea is not to go beyond (200 + max_workers) number of files inside the output directory.

... $q->enqueue( [ $chunk_id, "$part.out" ] ); $q->await( 200 ); # blocks until the queue has 200 or less items ...

Well, will ensure an example is included before releasing 1.7.