PERL UNIX and strangeness and char conversion

Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

I am trying to replace <tags> in a document. I noticed something interesting:

if I do 'more FOO.txt' I see:

<93> abcde <94>
[download]

if I view the same file with ptked or gedi, I see:

\x{93} abcde \x{94}
[download]

SOOO, is the 'more' filter correct or is the editor written in PERL correct?

I want to convert the stupidness to plain ASCII(?) punctuation marks so the grammar is correct, it looks better, and I can search with a perl/ptk program I am writing. Has someone already written a program to do this?

I have already figgered out how to convert the EOL chars to "/n" .

Comment on PERL UNIX and strangeness and char conversion Select or Download Code

Replies are listed 'Best First'.
Re: PERL UNIX and strangeness and char conversion by hippo (Archbishop) on Jan 18, 2021 at 16:35 UTC
This seems essentially the same as both Substitute and converting to UTF8 and convert tags to punctuation. Have a read of those and let us know if and how your question differs. 🦛	[reply]
Re: PERL UNIX and strangeness and char conversion by ikegami (Patriarch) on Jan 19, 2021 at 12:18 UTC
The file doesn't contain `<93>` or `\x{93}`; it contains a byte with a value of 0x93. This is garbage (where found) when UTF-8 is expected. The two programs simply handle this garbage differently. What you have is probably a file encoded using cp1252 rather than UTF-8. «“» and «”» are encoded as `93` and `94` respectively when using cp1252. Convert the file's encoding from cp1252 to UTF-8 before using it with tools expecting UTF-8. `iconv -f cp1252 -t UTF-8 file.cp1252 >file.utf8` [download]	[reply] [d/l] [select]
Re: PERL UNIX and strangeness and char conversion by tybalt89 (Monsignor) on Jan 18, 2021 at 19:54 UTC
""more" and those editors are all correct. They are just showing a different representation of the same unprintable character. `perl -pe 'tr/\x80-\xff/?/' <oldfile >convertedfile` [download] There. I've converted all that strangeness to a plain ASCII ? just like you asked.	[reply] [d/l]
Re^2: PERL UNIX and strangeness and char conversion by Anonymous Monk on Jan 18, 2021 at 21:55 UTC
OK! I THINK I GET IT `perl -pe 'tr/\x93-\x94/"/' <INFILE >OUTFILE` [download] Inserts the " `perl -pe 'tr/\x97/,/' <INFILE >OUTFILE` [download] Inserts the , `perl -pe 'tr/\x91-\x92/'/' <INFILE >OUTFILE` [download] Croaks. If I insert "\" to escape the second ' IT STILL CROAKS! What is special about \x91 and \x92 in the above statement? At least I made progress this time...	[reply] [d/l] [select]
Re^3: PERL UNIX and strangeness and char conversion by tybalt89 (Monsignor) on Jan 18, 2021 at 22:11 UTC
The bash shell does not take escapes, try `perl -pe 'tr/\x91-\x92/\x27/' <INFILE >OUTFILE` [download] instead.	[reply] [d/l]
Re: PERL UNIX and strangeness and char conversion by jcb (Parson) on Jan 18, 2021 at 23:59 UTC
if I do 'more FOO.txt' I see: <93> abcde <94> if I view the same file with ptked or gedi, I see: \x{93} abcde \x{94} That is the hint that your "<tags>" are not tags at all. The `more` program displays bytes with values greater than 127 as "<xx>" where "xx" is two hex digits. As other monks mentioned on a previous question, this is very likely to be Microsoft "smart quotes" garbage if those are in the proper positions in the text, although standards declare the 0x93 and 0x94 codepoints to be in the C1 group of control characters. (So of course Microsoft would use them as graphic characters...) You probably want the old "demoronizer" tool; it was written specifically to fix this type of stupidity.	[reply]
Re: PERL UNIX and strangeness and char conversion by Anonymous Monk on Jan 18, 2021 at 20:31 UTC
Apps vary considerably as to how they display "non-ASCII characters." And, separately, whether they understand UTF-x (Unicode). Both are probably telling you, each in their own way, that they see bytes with MSB=1.	[reply]
Re: PERL UNIX and strangeness and char conversion by betmatt (Scribe) on Jan 20, 2021 at 10:52 UTC
This is a plea from somebody that maybe needs to know more Computer Science. Can you please rewrite this question giving a bit of background. I feel that I should understand the question, but I can't make head or tail of it. A bit of background would go a long way for the more novice browser.	[reply]