in reply to Parallel computing with perl?

"we deal with enormous data sets" and "we can load the data into memory" seems not compatible with one another unless you have multiples of "enormous memory" available on your box! That is, unless the data is strictly read-only, all programs running on different cores can have access to the same data in memory (and by doing so does not make it "dirty" so it gets copied to their own storage) and each program does not use lots of working storage to keep (intermediary) results.

If you need the different programs communicate with one another, something like POE is perhaps a good start.

CountZero

A program should be light and agile, its subroutines connected like a string of pearls. The spirit and intent of the program should be retained throughout. There should be neither too little or too much, neither needless loops nor useless variables, neither lack of structure nor overwhelming rigidity." - The Tao of Programming, 4.1 - Geoffrey James

In Section Seekers of Perl Wisdom