Re: Converting UTF8 to ANSI

\x{EF}\x{BB}\x{BF} is the UTF-8 BOM, but because you're looking for the UTF-8 encoded BOM EF BB BF instead of the Unicode character U+FEFF, that tells me you haven't opened the file with the right encoding, and personally I think decoding afterwards is more of a pain than opening the file with the right encoding in the first place. Also, by "ANSI" I assume you mean Windows-1252. Anyway, if you are certain that all of your UTF-8 encoded files begin with a BOM, you can use File::BOM, the following will open files that have a BOM with the proper encoding, but fall back to CP-1252 if they don't:

use File::BOM qw/open_bom/;
open_bom(my $fh, $filename, ':encoding(cp1252)');
[download]

Otherwise, if you have no sure way of telling the files apart, you may have to use Encode::Guess, with the caveat that it's just a guess. Something like this maybe:

use Encode::Guess;
open my $fh, '<:raw', $filename or die $!;
read $fh, my $buf, 1024; # may need bigger buffer for better guess?
close $fh;
my $enc = guess_encoding($buf, qw/cp1252 utf8 UTF-16/);
ref($enc) or die "Can't guess $filename: $enc";
print "$filename: guessed ",$enc->name,"\n"; #Debug
open $fh, '<:encoding('.$enc->name.')', $filename or die $!;
[download]

In both cases, you may want to strip the BOM off the beginning of the data read from the file via $data =~ s/\A\x{FEFF}//;

Comment on Re: Converting UTF8 to ANSI Select or Download Code

Replies are listed 'Best First'.
Re^2: Converting UTF8 to ANSI by ikegami (Patriarch) on Aug 28, 2017 at 04:26 UTC
by "ANSI" I assume you mean Windows-1252. To find out a machine's actual "ANSI" encoding, you can use the following: `use Win32 qw( ); my $ansi_enc = "cp".Win32::GetACP();` [download]	[reply] [d/l]
Re^3: Converting UTF8 to ANSI by Your Mother (Archbishop) on Aug 29, 2017 at 18:11 UTC
Excellent tip. FWIW, I had to update to use it. `PS C:\Users\moo> perl -MWin32 -E 'say Win32::GetACP()' Undefined subroutine &Win32::GetACP called at -e line 1. PS C:\Users\moo> cpanm Win32 --> Working on Win32 Fetching http://www.cpan.org/authors/id/J/JD/JDB/Win32-0.52.tar.gz ... + O Configuring Win32-0.52 ... OK Building and testing Win32-0.52 ... OK Successfully installed Win32-0.52 (upgraded from 0.44) 1 distribution installed PS C:\Users\moo> perl -MWin32 -E 'say Win32::GetACP()' 1252` [download]	[reply] [d/l]
Re^3: Converting UTF8 to ANSI by palkia (Monk) on Aug 29, 2017 at 18:05 UTC
Thank you. I didn't even know a machine can have, an actual vs other, ANSI encoding. I'll look in to it as soon as I can.	[reply]
Re^2: Converting UTF8 to ANSI by palkia (Monk) on Aug 29, 2017 at 18:02 UTC
Thank you for your replay. You are correct, my understanding of the differences between specific encoding form is strictly theoretical and limited at the moment, especially in the perl context (1st time I ever encounter the term BOM). I hope to learn more about it as soon as I can. As for what I mean by ANSI, I really don't know. All I know is what the encoding line says when I "save as" a file with "Notepad" (Win-xp). Unfortunately I'm currently preoccupied with the fallout of attempting to install File::BOM as you can see here. Any assistance with this bigger issue will be most appreciated.	[reply]
Re^3: Converting UTF8 to ANSI by haukex (Archbishop) on Aug 29, 2017 at 19:45 UTC
You may want to have a look at The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!) As for what I mean by ANSI, I really don't know. All I know is what the encoding line says when I "save as" a file with "Notepad" (Win-xp). I suspect that'll be an encoding like the one I mentioned. You could use the other module I showed, Encode::Guess, to try and find out - it's a core module, no installation necessary. Another trick is to use a character that gets encoded differently in different encodings. I like the Euro symbol € (`"\x{20AC}"` in Perl), other than being easy to type on my keyboard, it is: Not representable in ASCII or Latin1 (aka ISO/IEC 8859-1) In Latin-9 (aka ISO/IEC 8859-15) it is the byte `0xA4` In Windows-1252 (aka CP-1252 and sometimes aka "ANSI") it is the byte `0x80` In MacRoman it is the byte `0xDB` (at least Mac OS >=8.5) In UTF-8 it is the bytes `0xE2, 0x82, 0xAC` In UTF-16 it is `0x20AC` - depending on endianness, bytes `0x20, 0xAC` in UTF-16-BE, or `0xAC, 0x20` in UTF-16-LE So try typing "€" into Notepad, saving it, and using a hex editor to get a clue (not a definite answer) as to what encoding it is. Minor updates.	[reply] [d/l] [select]
Re^3: Converting UTF8 to ANSI by AnomalousMonk (Archbishop) on Aug 29, 2017 at 19:10 UTC
As for what I mean by ANSI, I really don't know. Welcome to the wonderful world of character encodings! What you may mean is the ASCII character encoding. This is an old, 7-bit encoding with the most significant bit (bit 7) always 0. One neat thing about the newer UTF-8 encoding (some people say it's the only neat thing) is that all valid ASCII characters are automatically valid UTF-8 characters. Unfortunately, things quickly go to pieces after that; not all valid UTF-8 characters are valid ASCII, and any mapping of UTF-8 to ASCII is totally arbitrary. Oh, well... Give a man a fish: `<%-{-{-{-<`	[reply] [d/l]