Check your checks. Presumably you are allocating/relallocating memory somewhere, using some sort of IPC like shmem or maybe mmap. Check that the APIs called from XS are allocating enough memory, and that you are testing return values. You might also wish to check $!, or whatever mechanism you are returning errors (how do you indicate out-of-memory to the caller?). It could be that this particular tester has a low memory situation (runnning out of swap, for example).

The large negative numbers returned by some of the tests are probably pointers and often indicate uninitialised memory - as you probably know this is common when you run out of bounds.