nickos has asked for the wisdom of the Perl Monks concerning the following question:

Hi monks!

I'm pretty new to Perl and am having problems with threads. I'm writing a benchmarking script that needs to create many different threads where each thread will run an external program (a script interpreter (not Perl) running different scripts) multiple times.

The script ends prematurely with the following errors: 

Thread already joined at pm_spawn.pl line 72.
A thread exited while x threads were running.
and sometimes it seems to just hang randomly (and in different places). Any ideas what I'm doing wrong? I don't think the problem is from the external program, as the scripts that they're running simply print a character to the command line (for the moment anyway). I should probably add that I'm running Windows 2000.

Here's the script in it's entirety:

----------------BOF----------------
use Acme::Comment type => 'C++', own_line => 1, one_line => 1;
use Cwd;
use Run;
use Config;
use Data::Dumper;
use threads;
//use Async;
use Benchmark qw(:all);

use strict;
use warnings;

$Config{useithreads} or die("Recompile Perl with threads to run this p
+rogram.");

# Change this structure accordingly...
my %allScripts =
(
    submitters =>
    {
        numRuns => 4,
        steps => ["tssubmitter_1.script", "tssubmitter_2.script"]
    },
    reviewers =>
    {
        numRuns => 1,
        steps => ["tsreviewer_1.script", "tsreviewer_2.script"]
    }
);

my $numCompleteTestRuns = 100;

my $scriptInterpreter = "ScriptRunner.exe";
my $currentDir = getcwd();
my @threadList = ();

timethis($numCompleteTestRuns, 'runTests');

sub runTests
{
    print "Waiting to spawn processes.\n";
    
    foreach my $userType (sort {$allScripts{$a} <=> $allScripts{$b}} k
+eys %allScripts)
    {
//        print Dumper($allScripts{$userType}{steps});
        print "\nThe $userType will run $allScripts{$userType}{numRuns
+} times. The steps are:\n";
        foreach my $i (0 .. $#{ $allScripts{$userType}{steps}})
        {
            print "$i = $allScripts{$userType}{steps}[$i]\n";
        }
    } 
    
    print("\n\n");
    
    # key-value pairs in a Perl hash are not always stored in the orde
+r in which they were defined!
//    foreach my $userType (keys %allScripts)
    foreach my $userType (sort {$allScripts{$a} <=> $allScripts{$b}} k
+eys %allScripts)
    {
        for my $i (0 .. $allScripts{$userType}{numRuns})
        {
            print "\nAbout to create $userType thread $i:\n";
            my $thread = threads->new(\&stepThread, \$allScripts{$user
+Type});
            push (@threadList, $thread);
        }
    } 
    
    my $numThreads = $#threadList + 1;
    print("There are $numThreads threads\n\n");
    
    # wait for each running thread to end
    foreach my $thread (@threadList)
    {
        $thread->join;
        print("A thread joined\n\n");
    }
    
    print "Processes ended - Goodbye.\n";
}

# thread runs a type of users steps once only
sub stepThread
{
    print("starting thread\n\n");
    my $currentUsertype = shift;
//    print Dumper(${$currentUsertype}->{steps});
    foreach my $i (0 .. $#{${$currentUsertype}->{steps}})
    {
        my $fileName = ${$currentUsertype}->{steps}[$i];
        my $command = $scriptInterpreter . " \"" . $currentDir . "/" .
+ $fileName . "\"\n";
        print("About to run: $fileName\n");
        my $time0 = new Benchmark;
        my $output = `$command`;
        my $time1 = new Benchmark;
        my $timeTaken = timestr(timediff($time1, $time0));
        my $printOutput = "******** BEGIN $fileName output ********\n$
+output\n******** END $fileName output (time taken: $timeTaken) ******
+**\n";
        print($printOutput);
    }
    print("Ending thread\n\n");
}

[download]
----------------EOF----------------
  • Comment on "Thread already joined at..." and "A thread exited while x threads were running" errors
  • Download Code

Replies are listed 'Best First'.
Re: "Thread already joined at..." and "A thread exited while x threads were running" errors
by PodMaster (Abbot) on Sep 06, 2004 at 13:29 UTC
    Trying to join twice is fatal. 
    C:\>perl -Mthreads -e "my $t = threads->new( sub {} ); $t->join; $t->j
+oin;die 2"
Thread already joined at -e line 1.

    [download]
    You can trap fatal errors with eval. perldoc -f eval 
    C:\>perl -Mthreads -e "my $t = threads->new( sub {} ); $t->join; eval{
+$t->join};warn $@;die 2"
Thread already joined at -e line 1.
2 at -e line 1.

    [download]

     
    thread exited while x threads were running
    That is a warning (not-fatal), not an error, and you can largely ignore it.

    perldoc perldiag
    perldoc diagnostics

    update:
    Also, $Config{useithreads} or die("Recompile Perl with threads to run this program."); is useless as use threads; will fail if useithreads isn't defined.

    MJD says "you can't just make shit up and expect the computer to know what you mean, retardo!"
    I run a Win32 PPM repository for perl 5.6.x and 5.8.x -- I take requests (README).
    ** The third rule of perl club is a statement of fact: pod is sexy.

[reply]
[d/l]
[select]
Re: "Thread already joined at..." and "A thread exited while x threads were running" errors
by BrowserUk (Patriarch) on Sep 06, 2004 at 14:59 UTC

    There are a lot of "anomolies" in your script.

    The primary cause of the errors you are complaining of is becasue you are using a global variable (despite the my applied to it's declaration) to store your thread handles.

    This my @threadList = (); declares the array, but this array is then reused over and over within a subroutine, within nested loops: 

    sub runTests
{
...
    foreach my $userType (...
        for my $i ( ...
            push @threadList, threads->new( ...
        }
    }
...
   foreach my $thread (...
        $thread->join;
   }
}

    [download]

    So, each time through, you are pushing thread handles into that array, and then later joining them--but you never remove them!

    Which means that the second time through, you still have all the (joined) thread handles from the first run, and you attempt to re-join these, and get errors.

    A few others: 

    sort { $allScripts{$a} <=> $allScripts{$b} } keys %allScripts

    [download]

    The keys to this hash are strings, but you are using the numeric comparator (<=>), which means you are also getting a load of Argument "..." isn't numeric in numeric comparison (<=>) at... which you are apparently ignoring.

    Your comment: 

        # key-value pairs in a Perl hash are not always 
    # stored in the order in which they were defined!

    [download]

    is very misleading--mostly because it is so nearly true :) The order in which keys are returned by the keys operator is "Not well defined". In versions prior to 5.8.something, the order was determined by a series of internal storage details that means it was "essentially random". In later versions, the return order is deliberately randomised for security reasons, so it is "actually random".

    If it is important to your application that you can iterate your data structure in a specific order, you should either use an array -or- one of the many "sorted hash" modules from CPAN.

    Finally, I'm not sure what your script is trying to achieve, so this comment may be way off base, but in general, "threads" and "timing" do not go together.

    Threads are non-determanistic, which is to say, the time between your program asking a thread to start, and that thread actually getting a timeslice and running is determined by the OS according to a variety of factors that are entirely beyond prediction--system load, number of cpu's, whether other threads and processes in the system are cpu-bound or IO-bound; and a whole host of other factors.

    Timing things that are running on concurrent threads is about as meaningful as a chocolate teapot.

    Examine what is said, not who speaks.
    "Efficiency is intelligent laziness." -David Dunham
    "Think for yourself!" - Abigail
    "Memory, processor, disk in that order on the hardware side. Algorithm, algorithm, algorithm on the code side." - tachyon
[reply]
[d/l]
[select]
      "This my @threadList = (); declares the array, but this array is then reused over and over within a subroutine, within nested loops:"

      Yeah, I realised that after reading the first reply - moving the declaration into the scope of the runTests() subroutine fixes the initial problem but I'm still seeing the script hang. To check it wasnt the non-Perl interpreter which I'm trying to load-test that's crashing, I changed the script slightly to run DOS batch files instead, but I still get problems.

      "The keys to this hash are strings, but you are using the numeric comparator (<=>), which means you are also getting a load of Argument "..." isn't numeric in numeric comparison (<=>) at... which you are apparently ignoring."

      I don't seem to be getting those warnings. Any ideas?

      "Finally, I'm not sure what your script is trying to achieve, so this comment may be way off base, but in general, "threads" and "timing" do not go together."

      My company sells a proprietary system that sits on top of Apache, and we're trying to profile part of our software so that we can make accurate recommendations to customers when they ask what sort of hardware they should use with our software. My plan is to run the bulk of the code many times to get an average speed (see timethis($numCompleteTestRuns, 'runTests');).

[reply]

        Without sight of the modified script, plus the bat files, I could only guess. A word of caution though, if your not already using 5.8.4 or 5.8.5, upgrade before continuing. Many early bugs in ithreads where fixed in these builds. There is no point in re-discovering them:)

        I don't seem to be getting those warnings. Any ideas?

        No!? 

        use Benchmark qw[ timethis ];
timethis( 10, sub { my $x = 'A' <=> 'B' } );

Argument "B" isn't numeric in numeric comparison (<=>) at (eval 12) li
+ne 1.
Argument "A" isn't numeric in numeric comparison (<=>) at (eval 12) li
+ne 1.
timethis 10:  0 wallclock secs ( 0.00 usr +  0.00 sys =  0.00 CPU)

            (warning: too few iterations for a reliable count)

        [download]
        ... and we're trying to profile part of our software so that we can make accurate...

        I can only re-emphasis. "Timing" + "threads" !== "accurate". In fact, it will not even be consistant. This is not an error (of threads or Perl's ithreads), it's just a fact of life with preemptive multi-processing.

        With the additional information, and another look at your script, it looks like your trying to simulate a server environment and draw conclusions based upon that simulation. This will not work. Your simulation is simply not accurate, nor representative enough to be of any merit.

        If you need to profile, then you should be using a real profiler on the real code in the real environment, or as close to it as is possible to achieve.

        That's a pretty flat judgement based on very little information, so don't be offended. You may know what your doing is good enough for your purpose--it just doesn't look like it from where I am sitting :)

        Examine what is said, not who speaks.
        "Efficiency is intelligent laziness." -David Dunham
        "Think for yourself!" - Abigail
        "Memory, processor, disk in that order on the hardware side. Algorithm, algorithm, algorithm on the code side." - tachyon
[reply]
[d/l]

Back to Seekers of Perl Wisdom