in reply to Re: Warning: Unicode bytes!
in thread Warning: Unicode bytes!

With respect, you are wrong! I  know what data my scalars contain, and none of it is, nor could ever be mistakable, for unicode data.

Any determination by perl, that IT know's better than I, is a guess--and a wrong guess! For perl to guess, against my explicit instruction to the contrary, is also wrong.


Examine what is said, not who speaks.
"Efficiency is intelligent laziness." -David Dunham
"Think for yourself!" - Abigail

Replies are listed 'Best First'.
Re: Re: Re: Warning: Unicode bytes!
by dragonchild (Archbishop) on Apr 26, 2004 at 01:18 UTC
    To a human, it couldn't be mistaken. However, bits 1101 1100 1111 0111 (or whatever) may mean AX or it could mean the moon character in Chinese. That's all the contribution I have, regardless of PDF::Template's claim of Unicode compatibility. :-)

    ------
    We are the carpenters and bricklayers of the Information Age.

    Then there are Damian modules.... *sigh* ... that's not about being less-lazy -- that's about being on some really good drugs -- you know, there is no spoon. - flyingmoose

      That's precisely the point. As I said in the OP, the scalars in question contain arbitrary binary data, not text in any form of encoding, hence my use of use bytes, which works admirably well for index, rindex and anything else that doesn't use the regex engine.

      However, the search criteria I need to use lends itself nicely to using a regex, and under most circumstances of data and search values, works perfectly. But every now and again I was getting mismatches, and spent ages investigating both how the data and the search terms were constructed, before the realisation dawned.


      Examine what is said, not who speaks.
      "Efficiency is intelligent laziness." -David Dunham
      "Think for yourself!" - Abigail

        In that case, the question is "why does the scalar have the utf8 flag set". If it came from a filehandle, that question is equvilent to "why is the filehandle set to be in a utf8 encoding". A binmode will probably solve your problems here.

        If it didn't come from a filehandle, or marking the FH as having binary data is not a good thing, you can use $wasutf8 = Encode::_utf8_off($string);

        In case you wondered, my general rule is that giving the runtime more information about what's going (by making sure the utf8 bit is set correctly on scalars, or the encoding is set correctly on filehandles) is better then forcing it to do what you want, when it has other ideas (by using bytes).

        (That's not a hard and fast rule, of course...)


        Warning: Unless otherwise stated, code is untested. Do not use without understanding. Code is posted in the hopes it is useful, but without warranty. All copyrights are relinquished into the public domain unless otherwise stated. I am not an angel. I am capable of error, and err on a fairly regular basis. If I made a mistake, please let me know (such as by replying to this node).

        Perhaps you could provide an actual example? I don't think I understand the situation you refer to. I was assuming, perhaps erroneously, that you are running at least 5.8.1; I don't know as much about the utf8-related bugs that afflicted regex matching before then. Even there, I can't imagine the situation you seem to describe unless you actually are introducing utf8 data.
Re: Re: Re: Warning: Unicode bytes!
by Anomynous Monk (Scribe) on Apr 27, 2004 at 10:50 UTC
    With respect, you are wrong! I know what data my scalars contain, and none of it is, nor could ever be mistakable, for unicode data.
    Then you have no reason to say "use bytes".
    Any determination by perl, that IT know's better than I, is a guess--and a wrong guess! For perl to guess, against my explicit instruction to the contrary, is also wrong.
    Perl shouldn't guess; it should only flag as utf8 what you have (somehow or other) told it is utf8. use bytes doesn't do what you think; if anything, it will (in the presence of utf8 data) make things worse by exposing you not just to unicode characters but to the bytes that make up their encoding.