In general, the simplest way to debug thread subs, is to do so by getting them right outside of threads first.

Ie. Call them as a standard subroutines until they perform correctly.

Occasionally, you will encounter a bug that only shows up inside a thread, but thanks to the iThreads model, that is remarkably rare.

On those rare occasions when this has happened to me--invariably when calling Inline::C or XS code inside the thread--I've usually found that the problem doesn't happen if I run that code in a single thread at a time. Whether that thread be the main thread, or some other. And that invariable means that some dynamic library called from the XS routines I am calling, is inherently not thread safe and cannot be fixed without having the source available.

The typical scenario for this kind of error, is when the dll/so stores per-client state internally, using the process id as an index into a static data area. Hence, when called from two (or more) threads of the same process, it tries to re-use the same memory for two concurrent clients with inevitable results.

Interesting that your first stack trace had a frame for XS_Net__SSH2__new and this one doesn't...

By the time the warning that a thread terminated abnormally is issued, the tread that terminated abnormally is already terminated - it won't show up in the stack trace to the warning.

Here is another test program that demonstrates how to produce Perl stack trace if a thread dies and in output generally. These might be useful in case you can't duplicate the failure while running under the debugger.

This also shows that a fatal signal is inconsistent with the failure you are seeing (i.e. your process continues to execute after the thread terminates). This limits the possibilities somewhat.

use strict;
use warnings;
use threads;
use Carp qw(cluck);
$Carp::Verbose = 1;
use Inline 'C';

$SIG{__DIE__} = sub { cluck('someone died: ' . $_[0]); };

sub sub_that_dies {
        cluck('starting sub_that_dies');
        sleep(2);
        die "something bad happened";
}


sub start_thread {
        my ($arg) = @_;
        cluck('starting start_thread');
        print "thread started: $arg\n";
        if($arg == 1) {
                sub_that_dies();
        } else {
                sub_that_core_dumps();
        }
}


my $thr1 = threads->create('start_thread', 1);
$thr1->join();

my $thr2 = threads->create('start_thread', 2);
$thr2->join();

__END__
__C__
#include <string.h>

void sub_that_core_dumps() {
        printf("starting sub_that_core_dumps\n");
        sleep(2);
        printf("sub_that_core_dumps awoke\n");
        Inline_Stack_Vars;
        Inline_Stack_Push(newSVpvf("About to segfault in sub_that_core
+_dumps"));
        Inline_Stack_Done;
        perl_call_pv("Carp::cluck",0);
        memcpy(0,"source",1);
        Inline_Stack_Void;
}
[download]

Output is:

starting start_thread at ./test2.pl line 20 thread 1
    main::start_thread(1) called at ./test2.pl line 30 thread 1
    eval {...} called at ./test2.pl line 30 thread 1
thread started: 1
starting sub_that_dies at ./test2.pl line 12 thread 1
    main::sub_that_dies() called at ./test2.pl line 22 thread 1
    main::start_thread(1) called at ./test2.pl line 30 thread 1
    eval {...} called at ./test2.pl line 30 thread 1
someone died: something bad happened at ./test2.pl line 14.
 at ./test2.pl line 9 thread 1
    main::__ANON__('something bad happened at ./test2.pl line 14.\x{a}
+') called at ./test2.pl line 14 thread 1
    main::sub_that_dies() called at ./test2.pl line 22 thread 1
    main::start_thread(1) called at ./test2.pl line 30 thread 1
    eval {...} called at ./test2.pl line 30 thread 1
Thread 1 terminated abnormally: something bad happened at ./test2.pl l
+ine 14.
starting start_thread at ./test2.pl line 20 thread 2
    main::start_thread(2) called at ./test2.pl line 33 thread 2
    eval {...} called at ./test2.pl line 33 thread 2
thread started: 2
starting sub_that_core_dumps
sub_that_core_dumps awoke
About to segfault in sub_that_core_dumps at ./test2.pl line 25 thread 
+2
    main::start_thread(2) called at ./test2.pl line 33 thread 2
    eval {...} called at ./test2.pl line 33 thread 2
Segmentation fault (core dumped)
[download]

Running gdb on the core file yields the following backtrace:

#0  sub_that_core_dumps () at test2_pl_945a.xs:15
#1  0xb7745ad2 in XS_main_sub_that_core_dumps (my_perl=0x93ec518, cv=0
+x947f61c)
    at test2_pl_945a.xs:30
#2  0x08137c50 in Perl_pp_entersub (my_perl=0x93ec518) at pp_hot.c:295
+2
#3  0x081019e9 in Perl_runops_debug (my_perl=0x93ec518) at dump.c:2120
#4  0x08082d5b in Perl_call_sv (my_perl=0x93ec518, sv=0x94f19c4, flags
+=10)
    at perl.c:2611
#5  0xb7416224 in S_ithread_run (arg=0x93a97e8) at threads.xs:509
#6  0xb76b84c0 in start_thread () from /lib/i686/cmov/libpthread.so.0
#7  0xb763784e in clone () from /lib/i686/cmov/libc.so.6
[download]