I am experimenting with using threading, or forking, or whatever you want to call it, to speed things up that are slow, but where order of execution doesn't matter.

Here we have an artificial example, where I want to concatenate some letters, but the order of concatenation doesn't matter.

I thought I could use Parallel::ForkManager to achieve my aim like in the following, but as the tests demonstrate, this doesn't work.

The sub concatenate_serial does what I want, but is slow. It takes 9 seconds total, because I slowed things artifically with sleep(3). The other method, concatenate_parallel, finishes in 3 seconds, but it doesn't concatenate the letters.

I think I am probably making an error in thinking, or in misunderstanding what threads, or forks, or whatever you call it, is supposed to work.

**************
UPDATE: I see that the problem is that after each fork I am in a separate process, a copy of the parent process. And the variable I want to build the concat string with isn't shared across these processes.

I see that's the problem. What I don't have is a way to share these variables across the memory process. Well, I suppose one way would be to write the concat string to a permanent store on the hard drive, like a file or a db. But I'm wondering if there's a more elegant way, something that works "just with ram".

ANOTHER UPDATE: Okay, so I guess threads works "just with ram". Could someone throw some code my way that I could plug in to convert my serial code into parallel code (well, to be real pedantic, parallel code that can't be relied on execute sequentially)?
**************

Please help me understand this!

I withdraw to my little cell now for further study and meditation.....

use strict;
use warnings;
use Parallel::ForkManager;
use Test::More qw( no_plan );
use Data::Dumper;


my $letters = [ qw(a b c) ];

# the order of the concatenated letters doesn't matter, as long as we 
+get all three letters
my $content;

ok( index( $content = concatenate_parallel($letters), 'b' ) > 0, "it c
+ontains b, total content: $content"  ); # fast, but doesn't work
ok( index( $content = concatenate_serial($letters), 'b' ) > 0, "it con
+tains b, total content: $content"  ); # works, but slow



#works, but slow
sub concatenate_serial {
  my $letters = shift;
  my $content='';

  foreach my $letter ( @$letters ) {
    sleep 3; #something happens that takes noticeable time, like fetch
+ a url
    $content .= $letter;
  }
  print "serial content: $content\n";
  return $content
}

# fast, but doesn't work
sub concatenate_parallel {
  my $letters = shift;
  my $pm=new Parallel::ForkManager(10);

  my $content='';
  foreach my $letter ( @$letters ) {
    $pm->start and next;

    sleep 3; #something happens that takes time, like fetch a url
    $content .= $letter;

    $pm->finish;
  }
  $pm->wait_all_children;
  return $content
}
[download]

UPDATE: I changed the title to "using parallel processing to...." where originally it was "using Parallel::ForkManager to...". Seems like that module doesn't do what I want, but hopefully something with threads does.

UPDATE 2: Now I'm wondering if I could do this with MapReduce, pipin' fresh on cpan... Could there be ThreadedMapReduce (and/or ForkedMapReduce) instead of DistributedMapReduce?

---

---

Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!

Titles consisting of a single word are discouraged, and in most cases are disallowed outright.

Read Where should I post X? if you're not absolutely sure you're posting in the right place.

Please read these before you post! —

• How do I compose an effective node title?
• How do I post a question effectively?
• Markup in the Monastery

Posts may use any of the Perl Monks Approved HTML tags:

a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr

You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)

	For:		Use:
	&		&
	<		&lt;
	>		&gt;
	[		&#91;
	]		&#93;

Link using PerlMonks shortcuts! What shortcuts can I use for linking?

See Writeup Formatting Tips and other pages linked from there for more info.