Re^3: Parallel computing with perl?

comparing two huge datasets, operating on one or both or several datasets ... and do some operation on these data.

Which is it? One, two or if more, how many more?
Do those operations modify the original data?
If so, do other concurrent processes need to see those changes as they happen?
Each file is about 40 million lines (data points)

And how long are those lines? Ie. What is the total size of the file(s)?

Do the algorithms involved require random access to the entire dataset(s)? Or sequential access? Or random or sequential access to just some small subset for each top level iteration?

All of these questions have a direct influence upon what techniques are applicable to your application (regardless of the language used). And each answer will probably lead to further questions.

Your best hope of getting a good answer to your best way forward, would be to describe the application in some detail, noting volumes of data and times taken for existing serial processing. Making an existing serial processing script (or at least a fairly detailed pseudo-code description if the application is overly large or proprietary), would be better still.

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

"Too many [] have been sedated by an oppressive environment of political correctness and risk aversion."

Comment on Re^3: Parallel computing with perl? Download Code

Replies are listed 'Best First'.
Re^4: Parallel computing with perl? by necroshine (Acolyte) on Dec 16, 2008 at 11:52 UTC
Necroshine Besides, if you have data with interdependency data, AND you want to use multiple processors, or a 'grid', you will have serious problems to treat with the multiprocessing, whatever techinics you will to use. So in this case, I recommend MPI libraries, because you have the possibility of send data between processes(and the control of this by the processors), and this was made in C language. Perl have some support to MPI, but I never used it(RTFM CPAN). So, if you have inderdependency to data, this is not a problem(but is complicated ;) ), with MPI. Now, if you don't have a interdependency, you can just to use forks package(threads clone, but you don't need to activate thread mode), or some library that manipulates forks, or some other package. Check this in CPAN, RTFM and have fun!!	[reply]

Replies are listed 'Best First'.

Re^4: Parallel computing with perl?
by necroshine (Acolyte) on Dec 16, 2008 at 11:52 UTC

Necroshine

Besides, if you have data with interdependency data, AND you want to use multiple processors, or a 'grid', you will have serious problems to treat with the multiprocessing, whatever techinics you will to use.

So in this case, I recommend MPI libraries, because you have the possibility of send data between processes(and the control of this by the processors), and this was made in C language. Perl have some support to MPI, but I never used it(RTFM CPAN). So, if you have inderdependency to data, this is not a problem(but is complicated ;) ), with MPI.

Now, if you don't have a interdependency, you can just to use forks package(threads clone, but you don't need to activate thread mode), or some library that manipulates forks, or some other package. Check this in CPAN, RTFM and have fun!!

[reply]