As with most code, this can be improved further, yet still keep your own style. You are testing to see if $answer is 'y', and then testing to see if it is 'a'. It can't be both! No point in testing for 'a' or 'v' if it is 'y'. And what if the user mis-types, for example 'Q' (users never do what you want them to)?
Also, you are not reporting if the unlink fails. autodie is an easy way of doing that without hacking your code too much:

use strict;
use warnings;
use autodie;

my $origin = 'somefile';  # inserted for testing
my $removeall = 0;        # inserted for testing

if ($removeall == 1) {
    unlink $origin;
} 
elsif ($removeall == 0) {
    print "Remove '$origin'? ([Y]es, [N]o, [A]ll, ne[V]er; default: N)
+ ";
    chomp(my $answer = lc(<STDIN>));

    #print "answer: |$answer|\n" if (DEBUG);

    if (($answer eq 'y') or ($answer eq 'yes')) {
        unlink $origin;
    }
    elsif (($answer eq 'a') or ($answer eq 'all')) {
        unlink $origin;
        $removeall = 1;
    }
    elsif ($answer eq ('v' or 'never')) {
        $removeall = -1;
    }
    elsif ($answer ne '' and $answer ne 'n') {
        print "Invalid response: '$answer' (taken as N)\n";
    }
}
[download]