If you were using EUMM, i'd say try putting ".o" files into OBJECT key name for WriteMakefile, but you are not using EUMM. You can also try #xs room on irc.perl.org.
Comment on Re: Module::Build::WithXSpp does not correctly handle XS files correctly?