None of that info is particularly helpful to me at least but if you see something I don't, I'm all ears.

What I can see from that info is:

1. At line 3426:

   RXD+ (982) > Command completed with a return code of 0
   RXD+ (0) > _handle() output: 51686792
   thread handle:2d00 thread-id: 4240x
   [download]

   A thread, with a perl thread Id (tid) of 982 completes and Windows thread handle:2d00 has (just prior to joining) a OS thread ID of 4240, and the join completes without error.

2. Later, at line 3775:

   thread handle:2d00 thread-id: 0x
   GetLastError() output: '6'
   Join failed with 'Bad file descriptor' : 'The handle is
   invalid' at rxd.pl line 1128.
   [download]

   Just prior to a join attempt, 'another thread' with the same OS thread handle 2d00, this time does not have an OS thread id, which indicates that the thread handle:2d00 is indeed an invalid handle as the system reports.

What that indicates is that either:

• The OS is reusing the same OS thread handle -- which whilst possible seems unlikely.
• Or this; threads->list(threads::joinable) is returning the handle of an already joined thread. Which also seems unlikely, but could happen if the (Perl) internal linked list got corrupted some how.

The next thing I would try is adding a similar trace line at the end of S_ithread_create(), something like:

S_ithread_create(
...
    printf( "ITCREATE: thread handle:%x thread-id: %dx\n", thread->han
+dle, GetThreadId( thread->handle ) );
    MY_POOL.running_threads++;
    return (thread);
}
[download]

And also in

STATIC void
S_ithread_free(pTHX_ ithread *thread)
{

...

#ifdef WIN32
    printf( "ITFREE: thread handle:%x thread-id: %dx\n", thread->handl
+e, GetThreadId( thread->handle ) );
    if (handle) {
        CloseHandle(handle);
    }
#endif

...

}
[download]

The idea is to isolate whether -- when the error occurs -- the invalid handle is to a thread that has already been freed -- in which case the bug is in threads::list() -- or to a thread that has not yet been freed -- in which case it would mean an OS error of some kind; perhaps resource constraint;

I breifly looked at trying to run your server here and trying to re-create the failure. Whilst the server runs and accepts connections from a telnet seesion, it won't accept input from it because (my) telnet sends character by character and it is expecting entire commands wrapped in your (incredibly complicated) comms protcol.

There is no way I am going to be able to reverse engineer a client that can talk that protocol.

Alrighty, took a bit longer to recreate this time for whatever reason. I made the changes you suggested, but in the line printf( "ITFREE: thread handle:%x thread-id: %dx\n", thread->handle, GetThreadId( thread->handle ) ); i changed thread->handle to just 'handle' since it looks they already freed that pointer at that point.
I attached the log of this run below in the files called serverOutput2.txt, it exited with a different error this time,

Join failed with 'Inappropriate I/O control operation' : 'The handle i
+s invalid'
 at rxd.pl line 1128.
The RXD server has been shutdown.Perl exited with active threads:
        151 running and unjoined
        4 finished and unjoined
        0 running and detached
[download]

Which again appears to be related to the thread handle.
Sorry about the complicated protocol. People I work with did not wish to go through the trouble of having much back and forth communication between the client/server, and rather just send command once, receive response, and still needed a way to transfer a 400MB file. So this is what we (I) came up with (you should have seen the earlier version). I've attached the client (rx.pl) as well as the test script i used to recreate this issue. (i also included the server, rxd.pl, with all other commands besides exec stripped out except for EXEC to shorten the code. And included the exact threads.xs used to compile the threads module)
Thanks again for the help. Files:
https://dl.dropbox.com/u/19686501/perlmonk.zip

Okay. I think I have a handle on what is happening. The short explanation is that you are simply running the OS out of resources.

2000 concurrent threads, each starting a console session doing a dir -- 100 of which are recursive from root -- consumes prodigious amount of resources.

Your use of a VM maybe a contributory factor; I cannot reproduce the error here. My system grinds to a near complete halt for an extended period, but once the 1900 dirs of the current directory finish and 1900 cmd.exe's & 1900 threads & 1900 tcp connections go away, my system returns to a responsive state and it is then just a case of waiting while the 100 dir/s c:\ finish recursing the 212,000 directories and 1.5 million files on my hard drive, and then for all that data to get wrapped up in your protocol, shipped back to the receiving processes, unwrapped and output to the terminal.

But it works. It eventually completes okay; which I find quite remarkable and makes me think Ithreads -- on windows at least -- is in remarkably good fettle.

I do not believe that the problem you are seeing is a Perl issue; but rather an OS issue where -- under extreme resource depletion -- it is dropping/forgetting kernel thread objects that have completed before perl gets the chance to wait for them. I don't believe that should happen under normal circumstances, but these are not normal.

Why "believe" this and "believe" that!

There is a possibility that the trace output we produced is lying to me. For simplicity, the trace I had you add to threads.xs is crude -- and flawed. Using printf from multiple threads in C, is subject to the same problems of buffering and overlapping as print/printf are from multiple threads in Perl. It needs to be serialised. It is possible the symptoms I am seeing in the trace output you supplied -- Ie. A thread created that has become a non-thread by the time Perl tries to join it:

9408: ITCREATE: thread handle:2ca0 thread-id: 3620x
...
thread handle:2ca0 thread-id: 0x
GetLastError output: '6'
[download]

Is a symptom of overwritten buffered IO, rather than an OS "quirk". To counter that possibility, I've re-written the tracing code and wrapped it in a critsec to (attempt to) preclude that possibility. What that means is I am going to ask you to replace your threads.xs with this version:

I was going to post it above, but it is too big; perlmonks won't accept it. You'll need to /msg me an email ID so I can send it to you.

And re-build/install it. Then re-create your problem one more time. If the new install goes well, you should see:

*** CritSec initialised ***

RXD+ had been started on port 1600
...
[download]

when you start rxd.pl.

If you are successful in re-creating the failure, the trace output should be more reliable.

I also suggest the following 1-line change to rxd.pl which whilst it won't cure the problem; should make it less likely to occur -- assuming I've diagnosed the problem correctly. The change is to severely reduce the stack size allocated to each thread:

use warnings;
use threads stack_size => 4096;
use threads::shared;
use IO::Handle;
use IO::Socket::INET;
use File::Find;
use File::Path;
use Digest::SHA;
...
[download]

Note: Anecdotal evidence suggest that this does not work under (some versions of) *nix. If that is one of your targets. (It might work with 64k rather than 4k, but that is a guess! I've never had any feedback to confirm or deny that.)

A long term fix

Finally, I think that the real fix for the problem -- assuming we can confirm my diagnosis -- is to limit the number of concurrent clients to some sane number. On windows, with the stack_size fix above, a moderately specified VM -- say 8GB memory -- should handle 100 concurrent clients okay. You'll need to tweak that number for your target environment.

How I would implement that limiting is in the following code:

    ...
    unless ($client = $lsn->accept) {
        tprint ("Could not connect to socket: " . $!);
        next;
    }
    if( threads::list( threads::running >= 100 ) {
        $client->shutdown( 2 );
        $client->close;
        tprint( "Client $client rejected; too many concurrent clients.
+" );
        next;
    }
    ...
[download]

You might want to defer the rejection until you've accepted and validated the transmitted command and return a rejection/retry notification at that point if there are still too many concurrent clients.

{Thwack!} Balls in your court :)

Update: BTW, I also reduce testrdx.pl to this:

<Reveal this spoiler or all in this thread>

Which both reduced system resource usage (by doing away with the threads waiting on clients) on my single machine tests and control the number of concurrent clients.

---

With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

RIP Neil Armstrong

Sight of the changes made to the OP code and threads.xs would make interpreting that output a possibility :)

Also, what OS/version; Perl/version; threads/version?

Changes made to threads.xs

        /* Join the thread */
#ifdef WIN32
        printf( "thread handle:%x thread-id: %dx\n", thread->handle, G
+etThreadId( thread->handle ) );
        if (WaitForSingleObject(thread->handle, INFINITE) != WAIT_OBJE
+CT_0) {
            printf("GetLastError output: '%d'", GetLastError());
            /* Timeout/abandonment unexpected here; check $^E */
            Perl_croak(aTHX_ "PANIC: underlying join failed");
        };
#else
        if ((rc_join = pthread_join(thread->thr, &retval)) != 0) {
            /* In progress/deadlock/unknown unexpected here; check $! 
+*/
            errno = rc_join;
            Perl_croak(aTHX_ "PANIC: underlying join failed");
        };
#endif
[download]


Changes made to rxd.pl

    # wait to join and delete from hash
    foreach my $join (threads->list(threads::joinable)) {
        my $val = eval{ $join->join() } or die "Join failed with '$!' 
+: '$^E'";
        tprint("_handle() output: " . $join->_handle());
        #my $val = $join->join();
        # tprint("Deleting fd- $val");
        delete $FDcache{$val};
    }
[download]




OS/Version: Microsoft Windows Server 2003 R2 Enterprise x64 Edition Service Pack 2 And this is a VM if that makes any difference.

Perl/version: Active perl v5.16.1

C:\>perl -v
This is perl 5, version 16, subversion 1 (v5.16.1) built for MSWin32-x
+64-multi-t
hread
(with 1 registered patch, see perl -V for more detail)
[download]

threads/version: threads-1.86


Let me know if you need anything else!