The non-arch-specific files (INST_LIB) will be installed to INSTALLSITELIB, but you want them in the arch-specific site directory (INSTALLSITEARCH). So I think the comments so far (plus those docs) suggest that you could parse out the value of INSTALLSITEARCH from perl -V:installsitearch, then override INSTALLSITELIB to that value: perhaps at make time with something like make install INSTALLSITELIB=`find_archlib`.
It is not clear however why you don't simply put each perl build in a separate hierarchy. That would give you the separation you want in a far cleaner and more obvious manner.
I think the assumption underlying the existing directory layout is that things can and will be tested in isolation: if Module::B passes its tests and gets installed on each of perl-X and perl-Y, then it should not matter which of them the pure-perl Module::A has been tested against. If it does matter, that suggests either that there are tests missing for Module::B, or that there is code in one module that should be in the other.