in reply to Refactoring: Better Variable Names For Better Understanding? [SOLVED]

A parallel version for the updated code in the first post. Workers must be spawned prior to creating the image. Otherwise, Perl will crash from the Imager module not thread safe. This is also the case for the GD module.

use strict; use warnings; use Imager; use MCE; # based on original http://www.alfrog.com/mandel.html # karlgoethebier: code refactor # marioroy: parallelization my $width = 1280; my $height = 1024; my $iterations = 20; my @palette; my $image; my $mce = MCE->new( use_threads => 0, # MCE defaults to threads on windows max_workers => 'auto', chunk_size => 8, gather => sub { my ( $x, $y, $color ); while ( ($x, $y, $color) = splice(@_, 0, 3) ) { $image->setpixel( x => $x, y => $y, color => $palette[$color] + ); } }, user_func => sub { my ( $mce, $chunk_ref, $chunk_id ) = @_; my ( $re_c, $im_c, $re_z, $im_z, $color, $temp ); my ( @set_data ); for my $x ( $chunk_ref->[0] .. $chunk_ref->[1] ) { for my $y ( 0 .. $height - 1 ) { $re_c = ( $x - 3 * $width / 4 ) / ( $width / 3 ); $im_c = ( $y - $height / 2 ) / ( $width / 3 ); $re_z = $im_z = $color = 0; while ( 1 ) { $temp = $re_z; $re_z = $re_z * $re_z - $im_z * $im_z + $re_c; $im_z = 2 * $temp * $im_z + $im_c; ++$color; last if $re_z * $re_z + $im_z * $im_z > 4; if ( $color == $iterations ) { $color = 0; last; } } push @set_data, $x, $y, $color; } } MCE->gather( @set_data ); } ); # spawn MCE workers $mce->spawn; # init image and color palette $image = Imager->new( xsize => $width, ysize => $height ); push @palette, Imager::Color->new( 0, 0, 0 ); # black for ( 1 .. $iterations ) { my ( $r, $g, $b ) = map { int rand 255 } 1 .. 3; push @palette, Imager::Color->new( $r, $g, $b ); } # compute mandelbrot $mce->process({ bounds_only => 1, sequence => [ 0, $width - 1 ] }); $mce->shutdown; # save image my $file = qq(mandelbrot.png); $image->write( file => $file );
  • Comment on Re: Refactoring: Better Variable Names For Better Understanding? [SOLVED]
  • Download Code

Replies are listed 'Best First'.
Re^2: Refactoring: Better Variable Names For Better Understanding? [SOLVED]
by marioroy (Prior) on Jun 06, 2015 at 01:39 UTC

    Results: The time needed to compute a Mandelbrot from a Linux VM configured with 4 cores at 2.6 GHz (Haswell Core i7).

    # width => 1280, height => 1024   (  1.3 million pixels )
    
      Original       10.622 secs  eval statements slow it down
      Imager          7.219 secs
      GD              3.689 secs
      MCE + Imager    3.331 secs
      MCE + GD        0.565 secs
      MCE + GD + C    0.139 secs
    
    # width => 3840, height => 3072   ( 11.8 million pixels )
    
      Original       96.129 secs  eval, ditto
      Imager         64.229 secs
      GD             33.190 secs
      MCE + Imager   29.029 secs
      MCE + GD        4.344 secs
      MCE + GD + C    0.436 secs
    
    # width => 6400, height => 5120   ( 32.8 million pixels )
    
      Original      259.215 secs  ditto
      Imager        178.817 secs
      GD             93.168 secs
      MCE + Imager   79.986 secs
      MCE + GD       12.296 secs
      MCE + GD + C    0.997 secs  <-- fast is possible :)
      MCE + GD + C    1.002 secs  <-- repeatable
      MCE + GD + C    0.989 secs  <--
    
    

    Update: Both MCE + GD and MCE + GD + Inline::C compute the upper half only.

    Update: The writer is no longer the hold up for MCE + GD and MCE + GD + Inline::C. The logic was updated to minimize GD calls by 56x.

    Update: Reduced IPC for the MCE + GD + Inline::C demonstration. Wanted to see if possible to reach below 1.0 second. Btw, the time is the unix time which includes the time to launch Perl, load modules, spawn workers, compute the Mandelbrot 6400 x 5120 (upper half only), copy/flip/copy into bottom half, write the image, and shutdown workers. It is mind-boggling.

    Thank you karlgoethebier and perlmonks. Thank you also to the original author wherever you are.

    Kind regards, Mario

      "Thank you karlgoethebier...

      Definitely too much honor for me. I need to thank you! What should i say else? Great work!

      My very best regards, Karl

      BrowserUK should also be mentioned in dispatches because he suggested at least three optimizations: omitting unnecessary variables, drawing by line and flip.

      «The Crux of the Biscuit is the Apostrophe»

        You left out, " and perlmonks". I did not want to leave anybody out. There were other monks who answered this thread. I searched for the bits on BrowserUk suggesting drawing by line and flip. Very interesting and cool.

        I came to realization as well. The motivation came from wanting the writer to do as little work as possible so to keep up with many workers. In fact, the bottom half is drawn using just 3 GD calls, copy upper half/flip vertically/copy into bottom half. Also, am drawing lines and setting pixels, not just lines.

        Thank you for introducing us to Mandel and particularly creating the Threads From Hell #3 and series. Thank you BrowserUk. Thank you all. The original code does not mention the author's name unfortunately. The two eval statements are not necessary and wanting to let the author know.

        Kind regards, Mario

        The solution is posted here including MCE + Inline::C demonstration.

        $ time perl mandelbrot1.pl 16000 > m1.pbm real 8m38.383s ( Perl code by Mykola Zubach, threads only ) $ time perl mandelbrot2.pl 16000 > m2.pbm real 3m52.245s ( MCE supporting threads and processes ) $ time perl mandelbrot3.pl 16000 > m3.pbm real 1.974s ( MCE + Inline::C, mind boggling )

        Kind regards, Mario

Re^2: Refactoring: Better Variable Names For Better Understanding? [SOLVED]
by marioroy (Prior) on Jun 06, 2015 at 01:31 UTC

    The GD module is nearly 2x faster than Imager. Not sure why. Anyway, here is the parallel version using GD. Workers are spawned prior to the image being created by the manager process.

    use strict; use warnings; use GD; use MCE; # based on original http://www.alfrog.com/mandel.html # karlgoethebier: code refactor # marioroy: parallelization my $width = 1280; my $height = 1024; my $iterations = 20; my @palette; my $image; my $mce = MCE->new( use_threads => 0, # MCE defaults to threads on Windows max_workers => 'auto', chunk_size => 8, gather => sub { my ( $x, $y, $color ); while ( ($x, $y, $color) = splice(@_, 0, 3) ) { $image->setPixel( $x, $y, $palette[ $color ] ); } }, user_func => sub { my ( $mce, $chunk_ref, $chunk_id ) = @_; my ( $re_c, $im_c, $re_z, $im_z, $color, $temp ); my ( @set_data ); for my $x ( $chunk_ref->[0] .. $chunk_ref->[1] ) { for my $y ( 0 .. $height - 1 ) { $re_c = ( $x - 3 * $width / 4 ) / ( $width / 3 ); $im_c = ( $y - $height / 2 ) / ( $width / 3 ); $re_z = $im_z = $color = 0; while ( 1 ) { $temp = $re_z; $re_z = $re_z * $re_z - $im_z * $im_z + $re_c; $im_z = 2 * $temp * $im_z + $im_c; ++$color; last if $re_z * $re_z + $im_z * $im_z > 4; if ( $color == $iterations ) { $color = 0; last; } } push @set_data, $x, $y, $color; } } MCE->gather( @set_data ); } ); # spawn MCE workers $mce->spawn; # init image and color palette $image = new GD::Image( $width, $height ); push @palette, $image->colorAllocate( 0, 0, 0 ); # black for ( 1 .. $iterations ) { my ( $r, $g, $b ) = map { int rand 255 } 1 .. 3; push @palette, $image->colorAllocate( $r, $g, $b ); } # compute mandelbrot $mce->process({ bounds_only => 1, sequence => [ 0, $width - 1 ] }); $mce->shutdown; # save image open my $fh, '>mandelbrot.png' || exit 1; binmode $fh; print $fh $image->png; close $fh;

      Update: Corrected a couple 1 off errors.

      Update: Compute upper half only. Makes a copy afterwards, flips vertically, and copies into bottom half. There is a one time delay from Inline::C compiling the C code. Remember to run again.

      Update: Draw lines when possible.

      Update: Reduced IPC.

      Finally, providing a demonstration combining MCE::Flow + GD + Inline::C.

      use strict; use warnings; use MCE::Flow Sereal => 1; use MCE::Queue Fast => 1; use GD; # based on original code at http://www.alfrog.com/mandel.html # karlgoethebier: initial code refactor, thanks perlmonks # marioroy: parallelization via MCE::Flow (2 user tasks) + Inline::C # process upper half only (reduces work for writer by 50%) # draw lines when possible (reduces work for writer by 56x) # reduced # of times to enqueue/dequeue to lower IPC # also see tips by BrowserUk at: # http://www.perlmonks.org/?node_id=1128885 use Inline 'C' => Config => CCFLAGSEX => '-O2'; use Inline 'C' => <<'END_C'; int width, height, iterations, middle; void c_init( int _width, int _height, int _iterations ) { width = _width; height = _height; iterations = _iterations; middle = _height / 2; } AV * c_mandel( int x1, int x2 ) { AV *ret = newAV(); SV *line_size; double re_c, im_c, re_z, im_z, temp; int x, y, color, last_color; for (x = x1; x <= x2; x++) { for (y = 0; y <= middle; y++) { re_c = (double) ( x - 3 * width / 4 ) / ( width / 3 ); im_c = (double) ( y - height / 2 ) / ( width / 3 ); re_z = im_z = (double) 0.0; color = 0; while ( 1 ) { temp = (double) re_z; re_z = (double) re_z * re_z - im_z * im_z + re_c; im_z = (double) 2 * temp * im_z + im_c; ++color; if ( re_z * re_z + im_z * im_z > 4.0 ) break; if ( color == iterations ) { color = 0; break; } } if ( y && color == last_color ) { // increment line size, writer draws a line when > 0 line_size = *av_fetch( ret, -1, TRUE ); sv_setiv( line_size, SvIV( line_size ) + 1 ); } else { av_push( ret, newSViv( x ) ); av_push( ret, newSViv( y ) ); av_push( ret, newSViv( color ) ); av_push( ret, newSViv( 0 ) ); last_color = color; } } } return sv_2mortal( ret ); } END_C # fasten your seat belt, enjoy the ride :) my $width = 1280 * 1; my $height = 1024 * 1; my $iterations = 20; my $Q = MCE::Queue->new(); my $num_mandels = 3; my $num_writers = 1; # must be 1 # init C and MCE c_init( $width, $height, $iterations ); MCE::Flow::init { bounds_only => 1, chunk_size => 16, max_workers => [ $num_mandels, $num_writers ], task_name => [ 'tsk_mandel', 'tsk_writer' ], user_begin => sub { my ( $mce, $task_id, $task_name ) = @_; $mce->{ret} = [] if $task_name eq 'tsk_mandel'; }, user_end => sub { my ( $mce, $task_id, $task_name ) = @_; if ( $task_name eq 'tsk_mandel' ) { $Q->enqueue( MCE->freeze( $mce->{ret} ) ) if @{ $mce->{ret} } +; $Q->enqueue( undef ); } } }; # compute mandelbrot (user tasks; sequence begin, end ) MCE::Flow::run_seq( \&mandel, \&writer, 0, $width - 1 ); MCE::Flow::finish; # for MCE providers sub mandel { my ( $mce, $chunk_ref, $chunk_id ) = @_; push @{ $mce->{ret} }, @{ c_mandel( @{ $chunk_ref } ) }; if ( @{ $mce->{ret} } > 12000 ) { $Q->enqueue( MCE->freeze( $mce->{ret} ) ); $mce->{ret} = []; } } # for MCE consumer sub writer { my ( $mce ) = @_; # init image and color palette my $image = new GD::Image( $width, $height ); my @palette = $image->colorAllocate( 0, 0, 0 ); # black for ( 1 .. $iterations ) { my ( $r, $g, $b ) = map { int rand 255 } 1 .. 3; push @palette, $image->colorAllocate( $r, $g, $b ); } # process draw requests while (1) { my $ret = $Q->dequeue; if (!defined $ret) { last unless --$num_mandels; next; } my $data = MCE->thaw( $ret ); my $size = @{ $data }; for ( my $i = 0; $i < $size; $i += 4 ) { if ( $data->[$i+3] ) { # draw line if line_size > 0 $image->line( $data->[$i], # x1 $data->[$i+1], # y1 $data->[$i], # x2 $data->[$i+1] + $data->[$i+3], # y2 $palette[ $data->[$i+2] ] # color ); } else { # otherwise, set pixel $image->setPixel( $data->[$i], # x $data->[$i+1], # y $palette[ $data->[$i+2] ] # color ); } } } # copy upper half, flip vertically, then copy into bottom half my $middle = int( $height / 2 ); my $temp = new GD::Image( $width, $middle ); $temp->copy( $image, 0, 0, 0, 0, $width, $middle ); $temp->flipVertical(); $image->copy( $temp, 0, $middle + 1, 0, 0, $width, $middle ); # save image open my $fh, '>mandelbrot.png' || return 1; binmode $fh; print $fh $image->png; close $fh; }

      Update: Corrected a couple 1 off errors.

      Update: Computes the upper half only. Makes a copy afterwards, flips vertically, and copies into bottom half.

      Update: Draw lines when possible.

      The following is constructed using MCE::Flow with 2 tasks (many providers and one writer). Spawning workers early is not necessary for this demonstration. I went with GD due to running faster than Imager. However, one can easily change to Imager inside the writer function.

      use strict; use warnings; use MCE::Flow Sereal => 1; use MCE::Queue Fast => 1; use GD; # based on original code at http://www.alfrog.com/mandel.html # karlgoethebier: initial code refactor, thanks perlmonks # marioroy: parallelization via MCE::Flow (2 user tasks) # process upper half only (reduces work for writer by 50%) # draw lines when possible (reduces work for writer by 56x) # also see tips by BrowserUk at: # http://www.perlmonks.org/?node_id=1128885 # fasten your seat belt, enjoy the ride :) my $width = 1280 * 1; my $height = 1024 * 1; my $iterations = 20; my $Q = MCE::Queue->new(); my $num_mandels = 3; my $num_writers = 1; # must be 1 # init MCE MCE::Flow::init { bounds_only => 1, chunk_size => 16, max_workers => [ $num_mandels, $num_writers ], task_name => [ 'tsk_mandel', 'tsk_writer' ], user_end => sub { my ( $mce, $task_id, $task_name ) = @_; $Q->enqueue(undef) if ( $task_name eq 'tsk_mandel' ); } }; # compute mandelbrot (user tasks; sequence begin, end ) MCE::Flow::run_seq( \&mandel, \&writer, 0, $width - 1 ); MCE::Flow::finish; # for MCE providers sub mandel { my ( $mce, $chunk_ref, $chunk_id ) = @_; my ( $re_c, $im_c, $re_z, $im_z, $color, $last_color, $temp ); my ( @ret, $middle ); $middle = int( $height / 2 ); for my $x ( $chunk_ref->[0] .. $chunk_ref->[1] ) { for my $y ( 0 .. $middle ) { $re_c = ( $x - 3 * $width / 4 ) / ( $width / 3 ); $im_c = ( $y - $height / 2 ) / ( $width / 3 ); $re_z = $im_z = $color = 0; while ( 1 ) { $temp = $re_z; $re_z = $re_z * $re_z - $im_z * $im_z + $re_c; $im_z = 2 * $temp * $im_z + $im_c; ++$color; last if $re_z * $re_z + $im_z * $im_z > 4; if ( $color == $iterations ) { $color = 0; last; } } if ( $y && $color == $last_color ) { # increment line size, writer draws a line when > 0 $ret[-1]++; } else { push @ret, $x, $y, $color, 0; $last_color = $color; } } } # freezing here to prevent double freezing/thawing $Q->enqueue( MCE->freeze( \@ret ) ); } # for MCE consumer sub writer { my ( $mce ) = @_; # init image and color palette my $image = new GD::Image( $width, $height ); my @palette = $image->colorAllocate( 0, 0, 0 ); # black for ( 1 .. $iterations ) { my ( $r, $g, $b ) = map { int rand 255 } 1 .. 3; push @palette, $image->colorAllocate( $r, $g, $b ); } # process draw requests while (1) { my $ret = $Q->dequeue; if (!defined $ret) { last unless --$num_mandels; next; } my $data = MCE->thaw( $ret ); my $size = @{ $data }; for ( my $i = 0; $i < $size; $i += 4 ) { if ( $data->[$i+3] ) { # draw line if line_size > 0 $image->line( $data->[$i], # x1 $data->[$i+1], # y1 $data->[$i], # x2 $data->[$i+1] + $data->[$i+3], # y2 $palette[ $data->[$i+2] ] # color ); } else { # otherwise, set pixel $image->setPixel( $data->[$i], # x $data->[$i+1], # y $palette[ $data->[$i+2] ] # color ); } } } # copy upper half, flip vertically, then copy into bottom half my $middle = int( $height / 2 ); my $temp = new GD::Image( $width, $middle ); $temp->copy( $image, 0, 0, 0, 0, $width, $middle ); $temp->flipVertical(); $image->copy( $temp, 0, $middle + 1, 0, 0, $width, $middle ); # save image open my $fh, '>mandelbrot.png' || return 1; binmode $fh; print $fh $image->png; close $fh; }

      The examples demonstrate using the Core MCE API and MCE::Flow.