in reply to mocking or trapping system calls

I don't think this is easily doable. You probably would have to rewrite your code so that it calls your own backtick() function instead of `...`.

Alternatively, I'd look into running your program in some sort of sandbox (a chroot-ed environment or even a virtual machine), so that it can do something like rm -rf / without affecting your system.