What XS-based module are you using? Replace it with something that doesn't use XS.

Yes, that is just a guess and one based on very little data. But I've found that particular guess to be correct shockingly often, even when I have less data than this, and sometimes even when I have significant data pointing the finger away from the particular XS module.

The next option is that you have found a bug in Perl. But I've found that to be a significantly less likely case.