I tend to think that it's probably up to me to deal with it privately
I agree. Any pure-perl module, regardless of whether it depends on impure modules at any depth, should go in the architecture-independent lib path by default. How else would you deal with the case where the deep dependencies change either to or from XS? Or vary depending on other factors?
Your particular use case is far enough from the mainstream that it is IMHO up to you to set the necessary flags when installing such modules to meet your specific requirements.