Maybe then you need to invoke your tests in a different way. I usually run single tests as

perl -Mlib -w t/01-api.t
[download]

... and the whole test suite with

make test
[download]

or, if I want to run the author tests as well, with:

make
prove -bl xt/ t/
[download]

Directly starting a test will get things confused because the kernel will invoke the Perl that is hardcoded in the first line instead of using the Perl that you (may) want.