This might serve as a starting point, but what you are doing is fraught with problems. Even assuming that you are only interested in US phone numbers (4 or 10 digits), there are several conventions used by different groups of people when writing phone numbers, especially when the intended or perceived audience is international (ie. on the web), that will break when cleaned up using your rather simplistic mechanism.

As an example, note the number in the sample code below that begins +44(0).... in which the +44 is the country code for the UK, but the bracketed zero is there to indicate that the remaining digits of the number must be prefixed by zero if dialled from within the UK. Simply stripping the non-digits from the number will result in an invalid number no matter where it is dialled. I am not sure if anyone in the US uses this convention, but it is fairly common in Europe.

#! perl -slw
use strict;

my @fields = (
    '1234',
    '01234567890',
    'abcd1234xy',
    '012 345 6789',
    '+44(0)1234 567890',
);

for (@fields) {
    (my $new = $_) =~ s[\D+][]g;
    print "Old:'$_' New:'$new'??" 
        if length $new != 4 and length $new != 10;
}
[download]