in reply to using parallel processing to concatenate a string, where order of concatenation doesn't matter

If Parallel::Queue's threading worked for me I'd suggest the following.

use threads::shared;
use Parallel::Queue;

print concatenate_parallel( ['a' .. 'z'], 4 ) . "\n";

sub concatenate_parallel {
    my $result :shared;
    my @input = map {
        my $string = $_;
        sub {
            $result .= $string;
        };
    } @{ shift @_ };
    my $max_threads = shift @_;

    my $mgr = Parallel::Queue->construct( 'thread' );
    $mgr->runqueue( $max_threads, @input );

    return $result;
}

[download]

⠤⠤ ⠙⠊⠕⠞⠁⠇⠑⠧⠊

  • Comment on Re: using parallel processing to concatenate a string, where order of concatenation doesn't matter
  • Download Code

Replies are listed 'Best First'.
Re^2: using parallel processing to concatenate a string, where order of concatenation doesn't matter
by tphyahoo (Vicar) on Oct 18, 2006 at 14:26 UTC
    Thanks, that's intriguing, but my perl doesn't seem to like that code. 
    hartman@ds0207:~/pfmArena> perl test2.pl
Bogus Parallel::Queue: "fork" and "thread" are exclusive at test2.pl l
+ine 17
hartman@ds0207:~/pfmArena> perl -e '@ARGV=("test2.pl"); while (<>) {di
+e $_ if $. == 17}'
    my $mgr = Parallel::Queue->construct( 'thread' );
hartman@ds0207:~/pfmArena>

hartman@ds0207:~/pfmArena> perl -V | grep -i thread
    osname=linux, osvers=2.6.16, archname=i586-linux-thread-multi
    config_args='-ds -e -Dprefix=/usr -Dvendorprefix=/usr -Dinstallusr
+binperl -Dusethreads -Di_db -Di_dbm -Di_ndbm -Di_gdbm -Duseshrplib=tr
+ue -Doptimize=-O2 -march=i586 -mtune=i686 -fmessage-length=0 -Wall -D
+_FORTIFY_SOURCE=2 -g -Wall -pipe'
    usethreads=define use5005threads=undef useithreads=define usemulti
+plicity=define
    cc='cc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS 
+-DDEBUGGING -fno-strict-aliasing -pipe -Wdeclaration-after-statement 
+-D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
    cppflags='-D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS -DDEBUGGI
+NG -fno-strict-aliasing -pipe -Wdeclaration-after-statement'
    libs=-lnsl -ldl -lm -lcrypt -lutil -lpthread -lc
    perllibs=-lnsl -ldl -lm -lcrypt -lutil -lpthread -lc
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E -
+Wl,-rpath,/usr/lib/perl5/5.8.8/i586-linux-thread-multi/CORE'
                        PERL_MALLOC_WRAP THREADS_HAVE_PIDS USE_ITHREAD
+S
    /usr/lib/perl5/5.8.8/i586-linux-thread-multi
    /usr/lib/perl5/site_perl/5.8.8/i586-linux-thread-multi
    /usr/lib/perl5/vendor_perl/5.8.8/i586-linux-thread-multi
hartman@ds0207:~/pfmArena> perl -V | grep -i ithread
    usethreads=define use5005threads=undef useithreads=define usemulti
+plicity=define
                        PERL_MALLOC_WRAP THREADS_HAVE_PIDS USE_ITHREAD
+S
hartman@ds0207:~/pfmArena>

    [download]
    Any ideas?
[reply]
[d/l]

      Tell it to use threading by default: use Parallel::Queue 'thread';.

      ⠤⠤ ⠙⠊⠕⠞⠁⠇⠑⠧⠊

[reply]
[d/l]
        Can't... get it... to work...

        I tried as you suggested... 

        hartman@ds0207:~/Desktop/misc/idealoSubversionArena/parallelListProces
+sing> ./testpq.pl
Can't locate object method "new" via package "Threads" (perhaps you fo
+rgot to load "Threads"?) at /usr/lib/perl5/site_perl/5.8.8/Parallel/Q
+ueue.pm line 197.
hartman@ds0207:~/Desktop/misc/idealoSubversionArena/parallelListProces
+sing> head testpq.pl
#!/usr/bin/perl
use threads::shared;
use Parallel::Queue 'thread';

print concatenate_parallel( ['a' .. 'z'], 4 ) . "\n";

sub concatenate_parallel {
    my $result :shared;
    my @input = map {
        my $string = $_;

        [download]

        Also tried other permutations with spelling variations, but I feel like I'm stumblinb around in the dark. Parallel::Queue indicates that forking works, but threading is "still in progress".

        Are you able to run this code?

        I am really keen to make progress on this, because I am hoping to use this as a basis for threadedreduce which would serve as an abstraction layer for all sorts of list processing that could potentially be speeded up by using threading, but without changing anything in the "meat" of the program (which would be a function builder that uses threadedreduce or nonthreadedreduce, each having the same effect).

        See

        Could there be ThreadedMapReduce (and/or ForkedMapReduce) instead of DistributedMapReduce?

        and my followup, with threadedreduce stub code, at Re^2: Could there be a ThreadedMapReduce (instead of DistributedMapReduce)?

[reply]
[d/l]
[select]

In Section Seekers of Perl Wisdom