Beefy Boxes and Bandwidth Generously Provided by pair Networks
Do you know where your variables are?
 
PerlMonks  

Re: Threads From Hell #3: Missing Some Basic Prerequisites

by BrowserUk (Patriarch)
on May 30, 2015 at 15:18 UTC ( [id://1128409]=note: print w/replies, xml ) Need Help??


in reply to Threads From Hell #3: Missing Some Basic Prerequisites [Solved]

Other than the title; there doesn't seem to be any mention of "threads" in your code.

And the thing you've named "queue" is just a bog standard perl array that you push things into and then empty.

And the second part of your title doesn't seem to have any relevance to the code you posted.

You ask:

How can i improve this code?

But as I can't work out what it does (by inspection); and you don't post any results; and don't say what you think might be wrong with it; its hard to begin to work out what you are looking for?

Basically, I cannot make any connection between this post and the earlier 2 in the series?


With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority". I'm with torvalds on this
In the absence of evidence, opinion is indistinguishable from prejudice. Agile (and TDD) debunked

Replies are listed 'Best First'.
Re^2: Threads From Hell #3: Missing Some Basic Prerequisites
by karlgoethebier (Abbot) on May 30, 2015 at 17:30 UTC
    "I cannot make any connection between this post and the earlier 2 in the series"

    Sorry BrowserUK for being imprecise, i'll try to give some more information:

    Short time ago i wrote a script for learning purposes and fun that renders a huge image (20 MP) in ~13m on my box

    I'd like to write a multithreaded version of this script.

    The basic idea is to do it "row by row".

    Enqueue some rows of the image and process them...

    * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x

    ...next step...

    x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x

    ...proceed until finished.

    The snippet in my OP calculates the coordinates for the rows.

    For $width = 4; $height = 4; $qsize  = 4; the output is:

    [ [[0, 0], [0, 1], [0, 2], [0, 3]], [[1, 0], [1, 1], [1, 2], [1, 3]], [[2, 0], [2, 1], [2, 2], [2, 3]], [[3, 0], [3, 1], [3, 2], [3, 3]], ]

    @queue should perhaps better been named @rows.

    Then:

    my $queue = Thread::Queue->new(); $queue->enqueue(\@rows); my @threads = map { threads->create( \&process, $queue ); } 1 .. 4; ...

    Somewhere in the code:

    if ( $div == 0 ) { $image->setpixel( x => $x, y => $y, color => $black ); } else { $image->setpixel( x => $x, y => $y, color => $palette[$color] ); }

    OK, busted. It's a Mandelbrot. My first try.

    I hope very much that i could clarify the connection of my OP to the series.

    Best regards, Karl

    «The Crux of the Biscuit is the Apostrophe»

      Sorry BrowserUK for being imprecise,

      It's not so much a matter of imprecision as omission; but no matter. To the problem.

      You haven't show what where $image in $image->setpixel() comes from. Its not GD (which has setPixel()) so I'm assuming it comes from Imager or Image::Magick or a similar package.

      The first question is: are the C/C++ libraries that underly that module thread-safe.

      To explain. If I run this short script that uses GD, it displays a 100x100 png with the left half red and the right half blue:

      #! perl -slw use strict; use threads; use GD; sub rgb2n{ local $^W; unpack 'N', pack 'CCCC', 0, @_ } my $i = GD::Image->new( 100, 100, 1 ); # async{ $i->filledRectangle( 0,0, 50, 100, rgb2n( 255, 0, 0 ) ); # }->join; $i->filledRectangle( 51,0, 100, 100, rgb2n( 0,0, 255 ) ); open O, '>:raw', 'junk.png' or die $!; print O $i->png; close O; system 'junk.png';

      However, If I uncomment the two commented lines, thus the two rectangles are drawn to the image by different threads, I get this when I run it:

      C:\test>junk999 gd-png: fatal libpng error: No IDATs written into file gd-png error: setjmp returns error condition

      Ie. The libgd that underlies GD isn't threadsafe; and whilst the drawing to the image in the two threads executes without errors; when it comes to writing the image out to a file, a fatal error occurs. This isn't a limitation of perl's threading, but the underlying library.

      But you're using a different graphics library

      Let's explore if it makes any sense to write and image from multiple concurrent threads.

      Let's assume that one thread draws horizontal lines across the image in red, whilst another thread draws vertical bars down the image in blue. What should the resultant image look like?

      Like this? (You'll have to use your imagination here (#is red * is blue):

      ** ** ** ** ** ** ##**###**###**###**###**###**# ** ** ** ** ** ** ##**###**###**###**###**###**# ** ** ** ** ** ** ##**###**###**###**###**###**# ** ** ** ** ** ** ##**###**###**###**###**###**# ** ** ** ** ** **

      Or like this?:

      ** ** ** ** ** ** ############################## ** ** ** ** ** ** ############################## ** ** ** ** ** ** ############################## ** ** ** ** ** ** ############################## ** ** ** ** ** **

      Or some variation on this?:

      ** ** ** ** ** ** #######**###**###**###**###**# ** ** ** ** ** ** ############**###**###**###**# ** ** ** ** ** ** #################**###**###**# ** ** ** ** ** ** ######################**###**# ** ** ** ** ** **

      Now you're probably thinking that as you're only using setPixel, and never writing to the same pixel twice, it doesn't matter what order things occur; but most graphics libraries retain a whole bunch of state between drawing calls -- eg. current clip region; current line pattern; current alpha setting; current .... -- and if two threads start modifying that internal state without using locking, then the internals of the image; and even the entire library can become corrupted.

      That's what happened with my GD example.

      Enough for one reply, more in the next.


      With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority". I'm with torvalds on this
      In the absence of evidence, opinion is indistinguishable from prejudice. Agile (and TDD) debunked
      The snippet in my OP calculates the coordinates for the rows.

      For $width = 4; $height = 4; $qsize = 4; the output is:

      01 [ 02 [[0, 0], [0, 1], [0, 2], [0, 3]], 03 [[1, 0], [1, 1], [1, 2], [1, 3]], 04 [[2, 0], [2, 1], [2, 2], [2, 3]], 05 [[3, 0], [3, 1], [3, 2], [3, 3]], 06 ] <code> @queue should perhaps better been named @rows. Then: <code> 01 my $queue = Thread::Queue->new(); 02 $queue->enqueue(\@rows); 03 my @threads = map { threads->create( \&process, $queue ); } 1 .. 4; 04 ...

      So, you're generating an AoAoA of pixel coorinates and enqueueing it. Let's compare doing that with just setting the pixels in the nested loops you use to generate the AoAoAs:

      The result is:

      c:\test>junk999 Filling a 1000x1000 pixel image 1-pixel at a time took: 6.491771936s Populating 1000x1000 AoAoAs and tranferring to another thread took: 30 +.876082897s

      It takes five times longer to build the AoAoAs and pass it to another thread for drawing, than it does to do that drawing in the main thread!

      But what is the purpose of generating the coordinates and wrapping them up in the AoAoAs, only for the other thread to have to use nested loops to access them, when you could just pass 1000x1000 and have the other thread do the iteration itself.

      And what is the point of passing the entire data structure as a single entity? Only one thread will be able to receive it; so there's not even the possibility of using concurrency.

      Perhaps your intention is to do: $Q->enqueue( @cols ); (Ie. enqueue the contents of the array, not a reference to it.)

      That way, the row arrays would be pushed onto the queue as separate entities; which means that different threads could dequeue individual rows and operate upon them concurrently. Assuming a thread-safe graphics library.

      But even then what is the point in generating zillions of iddy-biddy arrays containing pairs of coordinates; when you could (say) push just the information required to construct them in the target thread:

      # main thread my( $X, $Y ) = ...; for my $y ( 0 .. $Y-1 ) { $Q->enqueue( [ $X, $y ] ); } ## drawing thread while( my( $x, $y ) = @{ $Q->dequeue } ) { $image->setpixel( $_, $y, ... ) for 0 .. $X-1; }

      You need the same loop code in the thread as you would to unpack the AoAs; but you only need to transfer a 1000th or less of the information between threads.

      Of course, the Devil is in the detail. Those ... above represent the color; and I assume that although you haven't shown it, the intent is to calculate the color for your Mandelbrot pixels in the main thread and farm off the drawing to the threads. And your (unshown) intention is to actually enqueue [ x, y, color ], [ x, y, color ] .... But that still doesn't make sense.

      Why transfer the same y-coordinate a thousand times (or however wide your image is) for each row?

      And why transfer a thousand X-coordinates when they can be inferred.

      Ie. Do this:

      ## main for my $y ( 0 .. $Y ) { my @colors; for my $x ( 0 .. $X ) { push @colors, Mandelbrot( $x, $y ); } $Q->enqueue( [ $Y, @colors ] ); } ### thread while( my( $y, @colors ) = @{ $Q->dequeue } ) { $image->setpixel( $_, $y, $colors[ $_ ] ) for 0 .. $#colors; }

      The Y-coordinate once, and a list of colors for the pixels is transfered, and the X-coordinates are inferred. (The problem of thread-safety remains.)


      With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority". I'm with torvalds on this
      In the absence of evidence, opinion is indistinguishable from prejudice. Agile (and TDD) debunked

        This goes to the bone. I need a break for reflection.

        Update:

        A dialogue is an old but established didactic device:

        Socrates: A man is a good adviser about anything, not because he has riches, but because he has knowledge?
        Alcibiades: Assuredly.

        Thank you very much for this excellent advice and best regards, Karl

        «The Crux of the Biscuit is the Apostrophe»

        "Φοβοῦ τοὺς Δαναοὺς καὶ δῶρα φέροντας."

        I tried to make your idea my own, simplifying it as much i could:

        #!/usr/bin/env perl use strict; use warnings; use Thread::Queue; use Data::Dump; my $width = 10; my $height = 10; my $queue = Thread::Queue->new(); for my $y ( 0 .. $width - 1 ) { my @colors = (); for my $x ( 0 .. $height - 1 ) { push @colors, mandelbrot(); } $queue->enqueue( [ $y, @colors ] ); } $queue->end; while ( defined( my $item = $queue->dequeue ) ) { dd $item; } # this is really a fake! sub mandelbrot { int rand 20; } __END__ karls-mac-mini:monks karl$ ./queue2.pl [0, 1, 16, 19, 13, 18, 18, 7, 10, 8, 7] [1, 2, 10, 0, 3, 2, 2, 2, 8, 3, 10] [2, 12, 5, 10, 5, 0, 14, 10, 10, 18, 5] [3, 13, 2, 11, 4, 16, 9, 19, 11, 5, 0] [4, 14, 3, 7, 18, 0, 17, 17, 16, 0, 8] [5, 4, 13, 3, 3, 10, 17, 13, 16, 11, 17] [6, 5, 2, 0, 17, 10, 1, 13, 10, 12, 3] [7, 18, 8, 1, 12, 13, 16, 0, 11, 14, 17] [8, 5, 10, 5, 19, 6, 17, 3, 10, 18, 16] [9, 1, 15, 10, 17, 8, 11, 18, 2, 18, 6]

        I think $Q->enqueue( [ $Y, @colors ] ); is a typo - it should be $Q->enqueue( [ $y, @colors ] );.

        In your while loop i ran into some undefinedness which i tried to fix.

        I was quite surprised that $image->setpixel( $_, $y, $colors[ $_ ] ) for 0 .. $#colors; works. Learning never stops.

        Here is how i create my image and my palette:

        use Imager; open( URANDOM, "</dev/urandom" ) # NTSC ;-); read( URANDOM, $_, 4 ); close URANDOM; srand( unpack( "L", $_ ) ); my $image = Imager->new( xsize => $width, ysize => $height ); my $white = Imager::Color->new( 255, 255, 255 ); my $black = Imager::Color->new( 0, 0, 0 ); my @palette; for ( 1 .. 20 ) { my ( $r, $g, $b ) = map { int rand 255 } 1 .. 3; push @palette, Imager::Color->new( $r, $g, $b ); } # dd \@palette;

        Well, still some omissions but perhaps step-by-step i get the idea.

        Update:

        ... use Devel::Size qw( total_size ); use Time::HiRes qw ( time ); ... $width = 1280*4; $height = 1024*4; ... __END__ Image Size 20 MP Queue Size 640.627 MByte Took 8.150 seconds

        Best regards, Karl

        «The Crux of the Biscuit is the Apostrophe»

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://1128409]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others about the Monastery: (6)
As of 2024-03-28 11:26 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found