Re: Locale Responsibilities

What does decode_utf8 do above check for UTF8 compliance and set the utf8 flag? Does it pack 4 octets per 32 bits for binary and one character per 32 bits for utf8 data.

decode_utf8 converts bytes "\xC3\xA9" into character "\xE9".

Internally, the string returned is the utf8 representation (a Perl-specific superset of UTF-8) of the character with the UTF8 flag on. For example, character "\xE9" is stored as the two bytes "\xC3\xA9", UTF8=1.

If the data read in is in binary format, then why did I have to use `use bytes' when searching it with an re (including the searching of binary data).

You don't.

At the time this made sense but now I'm having to convert the data to UTF8 I'm wondering well if it isn't already in utf8 then surely it's binary and then why the need for use bytes

You're unclear as to whether you're talking about the internal or external encoding. Perhaps Re: Decoding, Encoding string, how to? (internal encoding) would help.

Comment on Re: Locale Responsibilities Select or Download Code

Replies are listed 'Best First'.
Re^2: Locale Responsibilities by aecooper (Acolyte) on May 24, 2009 at 15:41 UTC
If I don't use the use bytes pragma I get: Malformed UTF-8 character (unexpected non-continuation byte 0xf4, 1 byte after start byte 0xf1, expected 4 bytes) in pattern match (m//) at /home/aecoope/code/monotone.ca/mtn-browse/lib/perl/FindFiles.pm line 601. when searching binary data with an re. Tony.	[reply]
Re^3: Locale Responsibilities by ikegami (Patriarch) on May 25, 2009 at 17:15 UTC
No, your incorrect use of `_utf8_on` (or equivalent such as the `:utf8` PerlIO layer) is causing that. `use bytes` kinda fixes your earlier bug. $ perl -MEncode=_utf8_on -e'$s = "\xF1\xF4"; _utf8_on($s); "" =~ /$s/' Malformed UTF-8 character (unexpected non-continuation byte 0xf4, imme +diately after start byte 0xf1) in regexp compilation at -e line 1. Malformed UTF-8 character (1 byte, need 4, after start byte 0xf4) in r +egexp compilation at -e line 1. Malformed UTF-8 character (unexpected non-continuation byte 0xf4, imme +diately after start byte 0xf1) in regexp compilation at -e line 1. Malformed UTF-8 character (1 byte, need 4, after start byte 0xf4) in r +egexp compilation at -e line 1. $ perl -MEncode=_utf8_on -e'$s = "\xF1\xF4"; "" =~ /$s/' $ [download]	[reply] [d/l] [select]