How to remove wide chars (ex: 年 or &#x6216) from a text file?

Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

Greetings Monks,

I have a question for you... How can I remove the below from a text file?

&#x5177;&#x4E09;&#x5E74;&#x6216;&#x4EE5;&#x4E0A;&#x76F8;&#x95DC;&#x5DE
+5;&#x4F5C;&#x7D93;&#x9A57; 

 -  &#x4E3B;&#x8981;&#x8CA0;&#x8CAC;&#x516C;&#x53F8;&#x5167;&#x4E4B;&#
+x8336;&#x6C34;&#x3001;&#x6E05;&#x6F54;&#x3001;&#x96DC;&#x52D9;&#x7B49
+;&#x5DE5;&#x4F5C; 

 -  &#x5FC5;&#x9808;&#x6709;&#x8CAC;&#x4EFB;&#x611F;&#x3001;&#x6574;&#
+x6F54;&#x3001;&#x5F85;&#x4EBA;&#x6709;&#x79AE; 

 -  &#x9700;&#x9577;&#x6642;&#x9593;&#x5DE5;&#x4F5C;
[download]

Thanks in Advance!

Comment on How to remove wide chars (ex: 年 or &#x6216) from a text file? Download Code

Replies are listed 'Best First'.
Re: How to remove wide chars (ex: 年 or &#x6216) from a text file? by davido (Cardinal) on Mar 05, 2013 at 20:52 UTC
Do you want to remove any code point that is outside the range of normal ASCII text? Do you want to remove any HTML entity that translates to a code point beyond the ASCII range? Both of those could leave big holes in your text. Or are you asking how to map code points that fall outside of the ASCII range into code points that fall within? If so, what mapping would you prefer? And how would you gracefully downgrade characters that don't have any obvious ASCII approximation? As you begin to consider and answer (for yourself and for us) those questions, a solution will emerge. Dave	[reply]
Re: How to remove wide chars (ex: 年 or &#x6216) from a text file? by graff (Chancellor) on Mar 06, 2013 at 02:54 UTC
I agree with davido - your question is too vague. What I see in the OP are hexadecimal entity references for Chinese characters. Maybe you have entity references to other non-ASCII characters as well? If you just want to delete all such entity references: `s/\&#x\w+;//g;` [download] (Note that the number of hex digits per character may vary.) In the case of the OP data sample, that regex leaves 7 lines that are either empty or contain just hyphens and/or spaces.	[reply] [d/l]
Re: How to remove wide chars (ex: 年 or &#x6216) from a text file? by Anonymous Monk on Mar 05, 2013 at 15:28 UTC
`my $badstuff = read_file( $file, { binmode => ':raw' } ); $string =~ s/\Q$badstuff\E//;` [download] now $string no longer has $badstuff, whatever it is see File::Slurp, binmode, quotemeta, Data::Dump	[reply] [d/l]