anshumangoyal has asked for the wisdom of the Perl Monks concerning the following question:

I am Forking 100 Threads (5 Processes are forking 100 threads). The Problem I am facing is Hash Variables are not getting the values. Here is my code snippet: 
use threads;
use threads::shared;
my %CallInfoHash : shared = ();

my $processToFork = 5;
my $threads = 100;
my %threadsHash;

for ($jj = 0; $jj <  $processToFork; $jj++) {
...fork a process here:
for ($ii = 0; $ii <  $threads; $jj++) {
my $callNum = $jj;
$UrlToDownload = <Some URL>;
push (@argList, $callNum );
push (@argList, $UrlToDownload);
$threadsHash{$CallNumber} = threads->create(\&tsub, @argList);
}
}

sub tsub {
.. Here some operations are done and values are generated.
These values are to be stored in %CallInfoHash  Hash.
my $value1 = xx;
my $value2 = xx;
$CallInfoHash{$CallNumber}{'value1'} = $value1;
$CallInfoHash{$CallNumber}{'value2'} = $value2;
.... There are about 10 values.
}

[download]
Now when I do foreach(keys %CallInfoHash) after the process has been completed, there is no value in %CallInfoHash. I am sure there is some issue with the shared variables but I am unable to find the solution. Please help.

Replies are listed 'Best First'.
Re: Forking Multiple Threads
by Corion (Patriarch) on Feb 08, 2012 at 11:48 UTC

    Do you understand the difference between fork and threads?

    fork creates a copy of the complete process, the "child". All changes made in the child stay with the child. Once the child exits, all the changes are lost.

    You might want to read some of the recent posts about fork, such as Fork Sharing Globals or Forks (not waiting for children).

    If you are just confused about your terminology, and say "fork" when you mean "start a thread", then maybe your problem stems from parts of your code that you did not show. Maybe you are not ->joining your threads.

    As to your general structure, I highly recommend feeding a few worker threads from a Thread::Queue instead of launching 100 or more threads. If your threads are mostly downloading data, it could also be interesting to use one of the asynchronous frameworks, like AnyEvent::HTTP or a dedicated framework like WWW::Curl.

[reply]
[d/l]
      I do understand difference between thread an forks. First I am using WWW::Curl for downloading file, but the requirement says I need to use Threads and Forks. I have been using Parallel::Forkmanager as well in past but here I want Fork to fork a process with 100 threads and what ever operation thread is doing is stored in a global variable (Hash), so that at the end of program I can see what all has happened (if anything goes wrong). I hope I am able to answer your Query!
[reply]

        That's not how fork works. There is nothing I can do to help you further. I recommend you learn about how fork works, and then find out a way how to store your data before the child program finishes to communicate with the parent. Maybe a logfile for each child works for you.

        Mixing threads and fork is something I highly advise against, and mixing database access and threads or fork also something to avoid.

        If you have "outside" (that is, homework) requirements to use both, threads and fork, the assignment likely was given to you that way so you learn about fork and threads and the difference and advantages and disadvantages of both. I recommend that you review your course material for more information or ask the person who gave you the requirement.

[reply]
Re: Forking Multiple Threads
by locked_user sundialsvc4 (Abbot) on Feb 08, 2012 at 14:04 UTC

    Wince!   500 of anything?!

    Kindly consider that “adding more threads” to any soup does not speed things up:   it slows it down by some amount, except to the extent that the hardware (on both ends of a communication link) actually can overlap I/O and computation.   (And if you thereby overload resources, especially such as memory, the whole thing goes to hall in a hendbasket, very quickly.)

    I suggest that you add some measurements to your request-queues.   Measure the time that a request actually sits in the queue before being sent to the host; then, measure the time the request takes to be returned.   Now, experiment with what happens as you reduce ... I suggest that you drastically reduce ... the number of processes and/or threads; the so-called “multiprogramming level” of your system.   Now, you can objectively measure the result.   Let me predict what you will find...

    I suggest that what you will see is that the actual performance curve follows a “bent knee” pattern that is typical of most such situations that are subject to “thrashing.”   Processing time will rise more-or-less linearly until it hits the wall and the curve goes straight-up into a pattern of exponential(ly bad...) increase.   You are probably already there.

    (Notice that I am talking about “request completion time” every bit as much as, if not more so than simply, “how much smoke is coming out of the ventilation vents of your CPUs.”)

    I/O requests of this type are typically asynchronous:   you can start a lot of them but you don’t have to dedicate a thread to wait for the completion of each one.   You can use a select() type of mechanism and perhaps use only one thread for the whole shebang.   Networks run in terms of milliseconds; CPUs in terms of nano.

      Now, experiment with what happens as you reduce ... I suggest that you drastically reduce ...

      How about you back up your useless diatribe by posting code that supports your contention?

      Put your money fingers where your mouth is, and demonstrate you have even the vaguest notion of what you're talking about.

      Bet you don't!

      With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.

      The start of some sanity?
[reply]

        Aww, knock it off, willya?   You’re just being a pest.   If you care to, why don’t you argue the opposite case?   Demonstrate for your fellow Monks, who I am sure will be quite interested, how throwing 500 threads at this problem is justifiable and necessary.   Otherwise, good sir, please mind your own business.   Snarking the threads of fellow Monks just because you don’t happen to agree with them does not win you the XP that you seem to crave.   Good day to you.

Back to Seekers of Perl Wisdom