in reply to How do I run subroutines in parallel?

Unless you are running on a multiprocessor box or a cluster, 'parallel' really just means multiprocessing. If you fork coprocesses, they can get their own time slice whenever they are ready to run, so your job effectively gets more time from the system.

Parallel::ForkManager is very convenient for that sort of thing. The GPL and Artistic licenses both encourage redistribution under sane terms.

(Added): Threads are another possibility, but they do not gain you time slices on *nix. They're mainly useful if the parallelized routines are I/O bound.

Perl 5.003 is terribly old. There were no threads then. On the brighter side, fork is fine for SMP and other multiple CPU systems, and will do true parallel processing for the kids.

After Compline,
Zaxo

  • Comment on Re: How do I run subroutines in parallel?

Replies are listed 'Best First'.
Re: Re: How do I run subroutines in parallel?
by Joost (Canon) on Sep 09, 2002 at 11:40 UTC
    Unless you are running on a multiprocessor box or a cluster, 'parallel' really just means multiprocessing. If you fork coprocesses, they can get their own time slice whenever they are ready to run, so your job effectively gets more time from the system.
    Good point. My first reaction upon reading the question was that forking and threading can improve your performance ONLY when you've got tasks that have a lot of 'waiting time', such as retrieving a bunch of files over a network. Your point about there being more timeslices for the code when spread over more processes proved me wrong before making an ass of myself in public :-)

    I do wonder if this makes a lot of difference when you don't have a lot of other processes running though, and wheter you cannot get the same sort of gain when you up (or lower, however you look at it) the priority on the process. Does anyone have any pointers to benchmarks/articles on this subject? 

    -- 
Joost       downtime n. The period during which a system
            is error-free and immune from user input.

    [download]
[reply]
[d/l]
Re: Re: How do I run subroutines in parallel?
by Anonymous Monk on Sep 09, 2002 at 15:37 UTC
    Yes we are running this on a multiprocessor box. I understood that threads were unstable in perl. Then again we are using perl 5.003.

    I need to look into fork tomorrow. This might be what I need. How do I call the subroutine? Do I need to split the subroutine into it's own perl file and call: fork(perl subroutine.pl)

    Also, I need to save the stdout/stderr. If I run things in parallel I think the output will be "mixed up". Correct?

    Sorry for such dumb questions...I'm not fluent in perl. :) I've rarely ever used modules... and I've never used threads in perl. And I can't find code examples to do this.

    Thanks for all the help!
[reply]
      If I run things in parallel I think the output will be "mixed up". Correct?Yes and no. You may redirect the output into different files.

      You may also use file locking to make sure that the messages are written to a file in a certain order.

      There are other issues that you have not considered. Once you fork, any contact between parent and child will need to use some IPC (InterProcess Communication) method in order to give data to a child, and to retrieve it. These include TCP Sockets, UNIX Sockets, shared memory & semaphores, RDBMS and finally simple files. I wish you good luck in your fight against the daemons of deadlock and concurrency.

[reply]

In Section Seekers of Perl Wisdom