Maybe you should consider a statistical approach.

You could use an absolute tolerance:

my $good_chars = $FORM{'phone'} =~ tr/-()0-9. //;
$good_chars += 3 if $FORM{'phone'} =~ /ext/i;

if (length($FORM{'phone'} - $good_chars) > 4) {  # Accept <= 4 bad
   # Bad phone number!
   ...
}
[download]

or a relative tolerance:

my $good_chars = $FORM{'phone'}) =~ tr/-()0-9. //;
$good_chars += 3 if $FORM{'phone'} =~ /ext/i;

if ($good_chars / length($FORM{'phone'} < 0.80) {  # Accept <= 20% bad
   # Bad phone number!
   ...
}
[download]

It would probably be more reliable than trying to find out all the valid characters.