Although not dependent on the system Perl, this approach essentially has the same problems with XS modules that are linked against system libraries.

For example, take Net::SSLeay.  The XS shared lib ".../Net/SSLeay/SSLeay.so" that directly belongs to the module is typically in turn linked against system libraries, such as /usr/lib/libssl.so.0.9.8, which implement the real functionality (the SSLeay.so is just the binding/wrapper). As those system libs are located externally of the local lib tree that you packaged up, you'll run into exactly the same problems, in case some system upgrade should've messed with those libs...

The only way around such issues is to either

1. link the binding statically against a ".a" version of the respective system libs (suitable for advanced users only — in any case definitely not the default)
2. package up the (tested) system libs together with the Perl stuff, and hard-code the dynamic library search path (to your local lib dir) into the shared object of the XS binding (also not the default)
3. as (2), but mess with LD_LIBRARY_PATH, or similar, at runtime