The instance of Perl you use to invoke the Makefile.PL (or other first-step Perl script) sets which instance of Perl the module will be built for. If you built it for the wrong version, then just rebuild it, this time using the proper instance of Perl to run the Makefile.PL.

make clean
$path_to_version/perl Makefile.PL
make
...
[download]

If you do not have a specific need for having two different versions of Perl installed, then it sounds like you would have a much easier time if you were to uninstall the version of Perl that you don't need.

If you do have a need for two versions of Perl (it is rather common to "leave the 'system' Perl alone" and to install your own version of Perl that you can customize for your needs), then I would next uninstall the version of Perl that you installed and then re-install it, this time making sure that your new version does not share any module directories with the other version of Perl.

Having two versions of Perl sharing a directory where modules are installed is usually not worth the resulting complexity.

If the only module directory being shared between the two versions of Perl is your user-specific directory, then we come back to "use the right version of Perl". Make an alias or change your $PATH or something so you stop jumping between Perl versions for different steps of what you are trying to do.