Strictly speaking, there are two ways that modules may be built; there are CPAN configuration variables corresponding to both of them.   In my experience, you need to separately and correctly set both.

In addition, you need to be sure that the PERL5LIB environment variable includes the target path ... before you actually start trying to install anything.   Typically, you do this by editing the .bash_profile file on Unix/Linux and then reading it using the source command.   In Windows, this setting is made (of course) in the Registry by means of the System control panel or otherwise.

(If you don’t do this first, then Perl will attempt to self-test the modules that it has just built, but those modules won’t know where to find themselves or anything else because PERL5LIB isn’t set yet.   Not pretty.   Did that one “bite” me once?   You bet it did!)

You are correct that the “base” library search-path is hard-coded into the Perl executable such that it can be obtained by perl -V with a capital “V” as you did.   However, this is only the base contents of @INC before added-to by PERL5LIB and so on; it does not specify the build-targets for CPAN purposes.