http://qs1969.pair.com?node_id=1151770


in reply to Re^3: UTF-8 and systemIO are not friends anymore
in thread UTF-8 and systemIO are not friends anymore

I go as low as Perl let's me go (system IO), and don't rely on anything that PerlIO might layer on top. If I do that, I don't have to peel back the layers of every onion and figure out what is happening and then compensate when necessary.
I understand. But, in Perl, sys functions are just not suitable for that. Consider:
use 5.022; use warnings; use Fcntl; use Devel::Peek; sysopen my $fh, 'out', O_WRONLY | O_CREAT; my $buffer = "\xFF\xFF\xFF"; utf8::upgrade($buffer); Dump $buffer; syswrite $fh, $buffer;
Folks, I don't know how to eliminate all unknowns, other than to use system IO.
Bypassing PerlIO is not enough; using system IO doesn't solve anything. And anyway, there are many ways for errors to appear - string concatenation, for instance... It doesn't really have much to do with IO per se.
Can anyone shed more light? Or is the answer to just go with PerlIO, "accept the defaults", trust in the decisions that are built into PerlIO, and let the chips fall where they may?
It seems to me that p5porters are strongly opposed to any explanations about how text in Perl actually works, which is too bad. Perl's "unified" model of text is a particularly leaky abstraction, IMO. Basically, p5porters advise "decode all inputs, encode all output" (using Encode, for example, or open my $fh, '<:encoding(SOME_ENCODING)', or binmode, or some such). Of course, in practice some strings cannot be decoded, or sometimes you don't want to decode/encode anything but some module that you use does that for you anyway (that's relatively recent examples actually posted on Perlmonks).

I'm not sure why they're opposed to document it... it's not like it's something difficult to understand. Why don't you ask them about it? If they're not actually against it, and just don't have time, someone else (maybe even I) can do it (especially if you'll then fix my grammatical, orphographic and other mistakes).

Replies are listed 'Best First'.
Re^5: UTF-8 and systemIO are not friends anymore
by Tommy (Chaplain) on Jan 04, 2016 at 16:15 UTC

    I find that most insightful, and helpful. What I'm seeing here basically is that it's quite easy to 'break' strings. I haven't ever really tried, but I can see that if someone were to deliberately (or inadvertently) work such chicanery with text, it would probably break the 'protections' I've hoped to achieve with syswrite().

    With regard to the idea of your possible assistance in documenting these specific Perl behaviors (once understood/clarified), you probably doesn't need to do so with significant stipulations for linguistic mistakes when you use words like "orthographic" in reference to your own literary introspection ;-) I.E.- you good, bro.

    It would be a worthwhile undertaking, because I can't hope to successfully design working software implementations over undocumented behavior. I have some somewhat-pressing software deployments in the pipeline that are going to hinge on a firm understanding of these undocumented things.

    Tommy
    A mistake can be valuable or costly, depending on how faithfully you pursue correction