Thanks for your cues.
Yes, I have a suggestion since I frequently have to update just the XS modules on a web host when they change the version of Perl. To find the XS modules you have installed, go to your local library directory where Perl modules are installed, and run the following command:
find . -name "*.so"
I had already thought of something similar, but it's generally quite hard to go back to the distribution name from a single .so file path.
Further research has probably suggested a way to solve also this problem but I need to know if both ExtUtils::MakeMaker or Module::Build write a so called .packlist file when installing the modules. My guess would be yes, but maybe somebody has more on this.