A simple and more flexible (for the user) way would be to reverse the args:

if ( "add" =~ /^\Q$answer/ ) {
...
}
[download]

Update: you can even precompile the regex for multiple uses:

my $ans = qr/^\Q$answer/;
if ( "add" =~ /$ans/ ) {
...
} elsif ( "remove" =~ /$ans/ ) {
...
}
[download]