Save a whole lot of effort by not deleting the non-numbers.

Instead, specifiy the acceptable separator characters in the regex.

$_ =~ s/\(?(\d{3})\)?  # area code with optional parentheses
        [-.\s]?        # possibly followed by separator
        (\d{3})        # first part of number
        [-.\s]?        # optional separator
        (\d{4})        # last part of number
       /$1-$2-$3/x;
[download]

You'd be better off using existing modules. If the closest modulkes are not quite good enough, why not modify the module, and possibly send your edits to the module maintainer.

I'm allowing both period '.' or dash '-' or a space as the separator. The area code may or may not have parentheses.