What encoding do you use to save the file? When I save it as as Windows-1252 and change it to actually do something, the string gets replaced:
La Conner….Shelter Bay Fee Simple
Here's how the important bytes are represented:
00000000: 2321 2f75 7372 2f62 696e 2f70 6572 6c0a #!/usr/bin/perl.
00000010: 7573 6520 7374 7269 6374 3b0a 7573 6520 use strict;.use
00000020: 7761 726e 696e 6773 3b0a 0a0a 6d79 2024 warnings;...my $
00000030: 636c 6d6e 4e6d 203d 206d 7920 2463 6c6d clmnNm = my $clm
00000040: 6e56 616c 203d 2027 4c61 2043 6f6e 6e65 nVal = 'La Conne
00000050: 72e2 80a6 2e53 6865 6c74 6572 2042 6179 r....Shelter Bay
00000060: 2046 6565 2053 696d 706c 6527 3b0a 6d79 Fee Simple';.my
map{substr$_->[0],$_->[1]||0,1}[\*||{},3],[[]],[ref qr-1,-,-1],[{}],[sub{}^*ARGV,3]
| [reply]
[d/l]
[select] |
It's UTF8. The data I receive is from a rest api so CP1252 characters are UTF8 encoded. This data is going into a database that will be used for the web with UFT8 encoding so I need to replace these characters accordingly.
| [reply] |
sub convert_to_html_entities {
my $str= shift;
utf8::decode($str);
$str =~ s/[\x{201A}-\x{2122}]/ '&#'.ord($&).';' /ger;
}
You could even just wholesale replace all non-ascii characters to completely sidestep the encoding problem:
sub convert_nonascii_to_html_entities {
my $str= shift;
utf8::decode($str);
$str =~ s/[^\x20-\x7E]/ '&#'.ord($&).';' /ger;
}
| [reply]
[d/l]
[select] |
my %smblCO = (
'xE2' => {
'\xE2\x80\x9A' => '‚',
'\xE2\x80\x9C' => '“',
'\xE2\x80\x9D' => '”',
'\xE2\x80\x9E' => '„',
'\xE2\x80\x93' => '–',
'\xE2\x80\x94' => '—',
'\xE2\x80\x98' => '‘',
'\xE2\x80\x99' => '’',
'\xE2\x80\xA0' => '†',
'\xE2\x80\xA6' => '…',
'\xE2\x84\xA2' => '™'
},
'xC2' => {
'\xC2\xA9' => '©',
'\xC2\xAE' => '®',
'\xC2\xB1' => '±',
'\xC2\xBC' => '¼',
'\xC2\xBD' => '½',
'\xC2\xBE' => '¾',
'\xC2\xB0' => '°'
}
);
You should try putting the keys in double quotes.
my %smblCO = (
'xE2' => {
"\xE2\x80\x9A" => '‚',
"\xE2\x80\x9C" => '“',
"\xE2\x80\x9D" => '”',
"\xE2\x80\x9E" => '„',
"\xE2\x80\x93" => '–',
"\xE2\x80\x94" => '—',
"\xE2\x80\x98" => '‘',
"\xE2\x80\x99" => '’',
"\xE2\x80\xA0" => '†',
"\xE2\x80\xA6" => '…',
"\xE2\x84\xA2" => '™'
},
'xC2' => {
"\xC2\xA9" => '©',
"\xC2\xAE" => '®',
"\xC2\xB1" => '±',
"\xC2\xBC" => '¼',
"\xC2\xBD" => '½',
"\xC2\xBE" => '¾',
"\xC2\xB0" => '°'
}
);
Naked blocks are fun!
-- Randal L. Schwartz, Perl hacker
| [reply]
[d/l]
[select] |
After much study, I determined that what your program does to a file is
replace non-ascii characters with their html equivalents. Your hex
sequences suggest that the external file is encoded as utf8, but you
read it as a string of bytes.
It has already been shown that your approach can be made to work. I prefer the following approach because its purpose is obvious. It lets perl decode the utf8 so it can substitute characters rather than byte strings. Note that all characters are treated the same. It does not matter how many bytes they require. Note also that the hash keys are the actual characters. The code point notation is just a convenient way to specify them without needing utf8 in the Perl code.
use strict;
use warnings;
use feature 'state';
use feature 'say';
#use utf8; #refer to node id=11154836 below
use autodie;
use Encode qw(decode);
while (my $clmnVal = <>) {
$clmnVal = decode('UTF-8', $clmnVal);
chomp $clmnVal;
$clmnVal = smblRpr( $clmnVal );
#say unpack('U0H*', $clmnVal);
say $clmnVal;
}
exit;
sub smblRpr {
my $strng = $_[0];
state %smblCo = (
# code point => HTML description
"\N{U+201A}" => "‚", # SINGLE LOW-9 QUOTATION MARK
"\N{U+201C}" => "“", # DOUBLE LEFT QUOTATION MARK
"\N{U+201D}" => "”", # DOUBLE RIGHT QUOTATION MARK
"\N{U+201E}" => "„", # DOUBLE LOW-9 QUOTATION MARK
"\N{U+2013}" => "–", # EN DASH
"\N{U+2014}" => "—", # EM DASH
"\N{U+2018}" => "‘", # LEFT SINGLE QUOTATION MARK
"\N{U+2019}" => "’", # RIGHT SINGLE QUOTATION MARK
"\N{U+2020}" => "†", # DAGGER
"\N{U+2026}" => "…", # HORIZONTAL ELLIPSIS
"\N{U+2122}" => "™", # TRADE MARK SIGH
"\N{U+00A8}" => "©", # COPYRIGHT SIGN
"\N{U+00AE}" => "®", # REGESTERED SIGN
"\N{U+00B1}" => "±", # Plus-Minus sign
"\N{U+00BC}" => "¼", # Vulgar fraction one quarter
"\N{U+00BD}" => "½", # Vulgar fraction one half
"\N{U+00BE}" => "¾", # Vulgar fraction three quarter
"\N{U+00B0}" => "°", # Degree sign
);
state $regex = do{
my @chars = keys %smblCo;
local $" = '|';
qr/@chars/;
};
return ($strng =~ s/($regex)/$smblCo{$1}/gr);
}
UPDATE: Removed use utf8 from code per hippo's comment.
| [reply]
[d/l]
[select] |
| [reply]
[d/l] |
| [reply] |
I would also work with the decoded strings. But I wouldn't have a table. If you have to use entities because you can't handle UTF-8, it make no sense to encode a subset only a subset of Unicode using entities. You should encode every non-ASCII Code Point.
See HTML::Entities's encode_entities_numeric.
| [reply]
[d/l] |