There are no "M-" characters as such. That's just how cat is displaying the non-ASCII characters. The ones that you say are "trivial" are probably some kind of white space character that you don't notice in the spreadsheet.

Once you know what characters they are, for example as suggested in Re: What's the 'M-' characters and how to filter/correct them?, you can remove them with a regular expression. For example, here's a situation I dealt with recently involving invisible special characters that were causing problems with web browsers:

        # U+2028 ('Line Separator') and U+2029 ('Paragraph Separator')
+ are valid JSON
        # but cause a parse error in the browser. So we remove them.
        $job_xml =~ s/\x{2028}|\x{2029}//sg;
[download]

Thank you for the explaination.

I think I'm able to get my script recognize the non-ascii characters out of the pieces of data.

However when I'm trying to remove/replace the non-ascii characters using the regex, it result still shows some unexpected characters （wrapped in point brackets) left in the position. Examples like following:

 25             $line =~ s/[^:ascii]//g;

 26             print $out_hdl "$line";

Result:

11AM<A0> LONDON

Dep<F3>sito Centralizado

This seems not like what I saw from the various examples on internet.

So, do you have ideas what's left there, and how could it be fully removed by this kind of regex?

 

Thanks

I am beginning to suspect that you have an XY Problem. Why do you have to sanitize your data in the first place? To what end? Is it really useful to just weed out characters and turn Depósito into Depsito?

Then, your character class definition is incomplete. It should be [^:ascii:] - the last colon was missing.

perl -le'print map{pack c,($-++?1:13)+ord}split//,ESEL'