in reply to Re^19: Interleaving bytes in a string quickly
in thread Interleaving bytes in a string quickly

Gotcha! (at last!).

Gotcha what? I've said the same thing for the beginning.

After utf8::upgrade($byte_string);, $byte_string is no longer a byte string. It's a character string, or a codepoint string.

No, it's not. Just like a number can be stored in an IV, an UV, an NV or a PV, a string of bytes is a string of bytes can be stored in a PV w/ UTF8=0 or a PV w/ UTF8=1. It's the same bytes no matter what internal format is used. Perl does not assign meaning to the values inside the string*.

If you guarantee that you give a string in UTF8=0 format (say by calling utf8::downgrade before calling interleave) you won't have a problem. That hadn't been specified until now. Instead of trying to gotcha me, maybe you should have said what you want to say. Your games aren't fun.

...can silently encode your bytes using UTF-8.", is just plain fiction. Any encoding has to be done, explicitly, by the programmer.

No, it doesn't. I even gave an example.

if (!sv_in || !sv_pad) croak("usage"); is a redundant code path

Ah good. I wasn't sure, so I erred on the safe side. In many places in the core, SV* can be NULL.

STRLEN i = l_in; is never used.

Thanks, Fixed.

It can also segfault! Try this print interleave_bytes( undef, 0 );

Thanks, Fixed by restoring my initial approach on which I had tested that. (The problem is that SvPV(0) is special.)

So, if you're gonna stand on that high horse throwing stones, you really ought to make sure that your mount doesn't have a glass jaw!

Pointing out a problem and explaining it when asked does not fit that description.

And your suggestion that I say I refuse to say anything until I can produce perfect code every time is just ridiculous. I have no problem addressing my problems.

* — Unless you ask it to. For example, uc will treat the values as unicode characters (regardless of the storage format), and vec will treat the values as bit strings (regardless of the storage format).

Update: Additions made to address the second have of the parent. I hadn't read it initially figuring it was just "gotcha!". But there were useful comments.

Replies are listed 'Best First'.
Re^21: Interleaving bytes in a string quickly
by BrowserUk (Patriarch) on Mar 01, 2010 at 13:46 UTC
    ...can silently encode your bytes using UTF-8.", .... Any encoding has to be done, explicitly, by the programmer.

    No, it doesn't.

    Yes. It does!

    I even gave an example.

    No. You did not! And you still have not--given an example of encoding occuring "silently".

    Perl does not assign meaning to the values inside the string*.

    What is PV w/ UTF8=0 or a PV w/ UTF8=1., if it is not "Perl ... assigning a meaning"?

    At the C-level, there is no difference; but at the Perl level there most definitely is. And it is at the C-level I am calling SvPVX(). For the very reason I do not want to make any such distinction. The same would not be true if I used SvPVBytes() as you suggested.

    If you guarantee that you give a string in UTF8=0 format (say by calling utf8::downgrade before calling interleave) you won't have a problem.

    You are still getting it arse backward. I don't need to call downgrade(), because I know I'm never going to call upgrade(), or anything else that might cause perl to assign any other meaning than bytes to my data.

    That hadn't been specified until now.

    Oh, but it has. Over and over:

    1. I could use it. I know the data I'd be passing are byte strings. Nothing else makes sense given the purpose of the code.
    2. But given I'm reading the file (binary data) in raw mode, how would they get "silently encoded"?
    3. Data either originates from within my program, or from without. And in either case, Perl will treat it as bytes unless I do something explicit to indicate that it should do otherwise. And since I know I'm not going to do that, I do not have to consider it.
    Your games aren't fun.

    I'm not playing games.. Far from it.

    You asserted: "It can silently encode your bytes using UTF-8.". Since you have demonstrably more knowledge of unicode than I, despite my strong belief that this was impossible, I was unsure enough to ask: Okay. How could it "silently encode my bytes as utf"?. Because if it was true, I wanted to know how.

    But, despite your protestations to the contrary above, you have still not provided an example of how this can happen. I now know this is because you cannot do so. Because it is impossible.

    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
[d/l]

      What is PV w/ UTF8=0 or a PV w/ UTF8=1., if it is not "Perl ... assigning a meaning"? 

      uint8  i = 2;
uint16 j = i;

      [download]

      Does the second assignment change the meaning of 2? No. It's still the number of cats my friends has. 

      my $i = "\x82";
utf8::upgrade( my $j = $i );

      [download]

      Does the second assignment change the meaning of "\x82"? No. It's still the HDMI code for something or other.

      because I know I'm never going to call upgrade(), or anything else that might cause perl to assign any other meaning than bytes to my data.

      Presumably you mean "anything else that might cause perl to use the 32/64-bit string format", then why didn't you say so?

      It's a very weird assumption to make, but you're free to make all the assumptions you want.

[reply]
[d/l]
[select]
        uint8  i = 2; uint16 j = i; Does the second assignment change the meaning of 2?

        Cheap theatrics. Cos now int i = -1; uint j = i; it does.

        Presumably you mean "anything else that might cause perl to use the 32/64-bit string format",

        Nope! I meant exactly what I said.

        Paraphrase: If I populate the PV of a perl scalar with byte values, and later call SvPVX() to gain access to those byte values, I will get back exactly what I put in them...unless in the interim, I explicitly do something to change them. No presumptions or assumptions. Simply fact.

        And using SvPVX() will NEVER cause it to

        silently encode your bytes using UTF-8.
        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
[d/l]
[select]

In Section Seekers of Perl Wisdom