If you compile perl and your XS code with -g flag, gdb will be able to show you more information, including where and in which file each frame in the stack relates to.

See Building a debugging perl in the INSTALL file that comes with the perl source for guidance on how to do this for perl.

This will make it easier to understand, but nothing will make it easy - between perl, XS, SSH and threads, you have quite a complicated situation.