So I do that and discover that now it can't find the modules

Because you installed "the modules" with the other Perl, so they went into that Perl's library. If you are using a perlbrew Perl then that Perl needs to have the modules installed with it.

One thing I do a lot is create a base perlbrew Perl installing either nothing but core Perl, or nothing but core Perl and certain common extensions, and then use perlbrew lib based upon that previously created Perl to house application-specific modules.

Then after that you will either have to use the shebang line, or else do some path wizardry on top of perlbrew, which is a bit silly since that's what it's for.