and it doesn't work for my "neVer" answer

Yes, it does: $answer =~ m/^v|(?:never)$/i

If you are using a relatively modern Perl (5:10 or later) you can even use the Perl switch construct (which is called given ... when).

use Modern::Perl;

my $answer = <STDIN>;

given ($answer) {
    when (/^y(?:es)?$/i)    { say 'yes'; }
    when (/^no?$/i)         { say 'no'; }
    when (/^a(?:ll)?$/i)    { say 'all'; }
    when (/^v|(?:never)$/i) { say 'never'; }
    default                 { say 'default'; }
}
[download]