The directories in PERL5LIB are searched before the standard directories (they are placed on the front of @INC). I don't think there is a way of removing the standard libraries just using the environment variable.

You can do more sophisticated adjustment from with perl by manipulating @INC within a BEGIN block, but that is not for the faint hearted.