in reply to Re^2: Something strange in the world or Regexes
in thread Something strange in the world or Regexes
0xa0 is an unbreakable space in e.g. latin1. c2 would be LATIN CAPITAL LETTER A WITH CIRCUMFLEX assuming latin1. Some pc charsets use chars in that region for e.g. dos-style line-drawing. Badly done pasting might have added these chars?
Update: just checked UTF-8: Almut's correct: looks like you've submissions in UTF8 which accidentally use the wrong space char. Probably the submitter is preparing his file in word or something similar unsuitable.
One sane approach is whitelisting as already suggested by Silas, e.g. just stripping non-alphanumerics-non-minus with e.g. s![^a-z0-9\-]!!gio. Note that this will also eat up space and line ends in $_. Which works, as we stick to the common subset of ASCII, which is also valid for submissions in UTF-8 and latin1. If you also see other charsets, things like GNU recode might help if enlightening submitters fails.
|
|---|