perl 5.42 will install modules to /usr/local/share/perl5/site_perl/5.42, right?.

The package manager shouldn't touch site_perl at all. Non-core modules it installs should be installed in the vendor dirs.

If we're talking about modules you install yourself, then the paths can be configured when perl is built. perl -V:'^install.*' will show the installation dirs used by a perl. That said, /usr/local/share/perl5/site_perl/5.42 seems likely if /usr/local/share/perl5/site_perl/5.40 was previously used.

I just don't grasp why all of the modules under the versioned dir need to be reinstalled

Because someone decided to install them in a versioned dir.

That said, that's a good practice because they might have XS dependencies.