in reply to Queuing in multithread context

From what I understand / can see your current multi-threading implementation is flawed. It appears you want to multi-thread sending data to servers, however for each server you want to send the data serially. At the moment, it looks like your implementation is to work through the servers serially and multi-thread the data being copied on each server

This is Not Good and using Thread::Queue won't solve your problem. You need to re-address your threading model. In fact, I would suggest you rebuild your application without using threading - Once its working, then add Thread::Queue to queue your list of servers, then add threads (with a static pool of threads) to do the work for each server.

Finally, you get the Free to wrong pool error because something you are doing (probably a library you are loading) is not thread safe. This can cause all sorts of problems, though there are potential workarounds (such as require'ing affected libraries after threads have been created, or only within the threads that need them).

Replies are listed 'Best First'.
Re^2: Queuing in multithread context
by Hardin (Novice) on Jan 20, 2015 at 12:03 UTC

    Thanks !!

    I was considering my approach wasn't the proper or best one. You perfectly understood what I was trying to achieve thus you made a important point.

    I've already been able to work without thread in a sequential way and ended implementing threads to hurry the stuff and not depending on tasks between separated servers.

    I'm still figuring out how I could use the queue as you pointed but at least it sounds like a more relevant direction...

[reply]

      This is a very basic representation of how your threading model might look:

      use 5.16.2;
use threads;
use Thread::Queue;

my @serverList = qw(one two three four five six);

my $serverQueue = Thread::Queue->new( @serverList );
$serverQueue->end;

my @threads = map { threads->new( \&worker ) } 1..4;
$_->join for @threads;

say 'Back to main - Finished!';

sub worker {
    my $tid = threads->tid;
    while ( my $server = $serverQueue->dequeue_nb ) {
        say "I'm thread ID $tid and I'm processing server $server to c
+opy a whole bunch of files";
        sleep 1;
    }
    say "Thread ID $tid finished";
}

      [download]
[reply]
[d/l]

        Thank you again for your guidance,

        problem is to me that I'm unable to find a pattern where I can achieve a working implementation of your sample. This is a great sample of queueing proved working but I just don't get it, maybe I just completely mistaken the purpose of the Thread::Queue; ... I understood it could hold threads (like tasks) waiting for the completion of one to dequeue a second..

        To be honest what is losing me in your example is that you never "queue" any item in $serverQueue... there is something I don't get in here, maybe this is a bit too advanced for my understanding

[reply]

In Section Seekers of Perl Wisdom