I recently wrote some simple multi-process code at Re^7: Multiprocess - child process cannot be finished successfully. Yes, the "fake" Windows PID from a Perl fork is negative and my wait statement accounts for that. Code shown does run on Windows and should run also on Unix. This demo code worked better than I thought it would - meaning that sleep in each sub process worked and did not interfere with each other. I am not sure how sleep() is implemented on Windows, but it worked better than expected!

However, it sounds like just using threads is the best way for your compute bound project. The less data you share between threads (ideally nothing that is r/w), the better. Threads can get complicated if there is a lot of sharing going on.

This is a code fragment from some code years ago... The program has about 70,000 input strings. For each input string, it is desired to know which of the other input strings are "close enough" according to some complex rules. For each input, a regex is generated that is run against all other inputs. This is a NxN algorithm. For 70K inputs it took ~1.5 hours. I have a 4 core machine. Running 4 threads, execution time was something like 3.8x (can't get to exactly 4.0, but that is a very good result). So anyway execution time went to ~20 minutes and that was "good enough" and I stopping improving things.

Anyway see below for an example of parallelizing a number cruncher job.

### This is a non-runnable code fragment
### only to show a general idea of threads pulling work
### from a common input queue and putting results on a
### common output queue.

use Threads::Queue;

my @all_inputs;  global data for all threads as read only

### Worker threads #####

my $thread_limit = 4;

my @threads;
push @threads, threads->create(sub{DoWork($workQueue, $doneQueue)}) fo
+r 1 .. $thread_limit;


foreach my $input ( @all_inputs )  #init the work queue with all input
+s
{
    $workQueue->enqueue($input);
}

$workQueue->enqueue(undef) for 1 .. $thread_limit;  #"work finished" m
+arkers
                                                    #each thread will 
+give up
                                                    #when it sees an u
+ndef

$workQueue->end();

$_->join() for @threads;  #waits for all threads to finish!

print "END of threading...\n";

####  Get results off of  Queue

my @results;
while ($doneQueue->pending() && ($_ = $doneQueue->dequeue()) )
{
   push (@results, $_); #results are pointers to array (AoA)
}

sub DoWork
{
   my ($workQueue, $doneQueue) = @_;
   
   while (my $input = $workQueue->dequeue()) 
   {   
      return unless defined ($input); #this ends this thread's job!!
      
      if ($input =~ m|/|)
      {
         $doneQueue -> enqueue([$input,'SKIPPING THIS ENTRY!']);
         next;
      }
      
      my $regex = get_regex_patterns ($input); 
      $regex =~ s/\(|\)//g;  #captured values not needed, only yes/no
        
      my @matches = grep{ m/$regex/ and $_ ne $input}@all_inputs;
 
      push (@matches,'') if @matches==0;
      
      $doneQueue -> enqueue([$call,@matches]);
   }
}

[download]

In reply to Re^3: Parallel processing on Windows by Marshall
in thread Parallel processing on Windows by BernieC

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.