There is no fact, mere "discussions with a lawyer" regarding dynamic linking, and I don't see the linking that Perl does (dynamically loading object code) as "creating a derivative work" in the same sense that linking to a C header creates a "derivative work", because to use the API from Perl does not involve the C headers.

Building the XS component involves the C headers and thus makes the XS component a derivative work, but that cannot infect the Perl program, no matter how much RMS wants this to.