...recompiling your Perl doesn't help you any.

Not necessarily true.  If you build a perl that's binary compatible with the perl that was originally used with the modules, you can for sure make use of existing XS modules — provided they do work in principle, i.e. their dependent libs are installed, etc. Otherwise, as any such libs will have to be installed anyway, trying to rebuild/install the modules alone wouldn't help you any.

Of course, the real question is why th OP didn't simply also copy over the perl binary that originally belonged to the modules...