in reply to Re^2: UTF-8 and systemIO are not friends anymore
in thread UTF-8 and systemIO are not friends anymore
it's already confusing enough, unfortunately.
...Which is unfortunately what I'm trying to help avoid. I'm getting more confused about it myself! This, sadly, isn't helping the situation.
My feeling is that IO should not try to do fancy things for people that they didn't ask for. Just an opinion, and obviously not shared by all. The core issue is that a lot of people don't know how to do portable IO on their own (not implying that my understanding of it is without flaw), and it takes a large amount of time investment to learn how to do it properly, and so we can end up with a DWIM attitude about it...
And that leads to a DWIM implementation. In an effort to be portable, maybe, and certainly in an effort to make things easy on people who just want to reliably read and write files, PerlIO (as it's referred to by many), effectively layers on more activity than C 'system' open/read/write operations. In fact, the trend toward our current situation feels like it has been progressively more DWIM over the years. Again, some might disagree, and I do confess that I haven't read the full release notes of every Perl either. But this isn't the point...
With multi-byte characters, the situation just snowballs progressively into more confusion. I openly admit that I do not yet know, but am working on obtaining a better understanding of the variances in behaviors of Perl's many actively-used versions on all major many platforms, in addition to pre-baked Perl "distributions" (e.g.- strawberry), and individual compilation options when compiled from source (e.g.- perlbrew). Presently I do not know how many permutations of behavior there are, and how to account for all of them when attempting to establish best-practices for completely portable IO--if not for others, at least for me. In the very least, I want to know more and do better than I currently can for my own endeavors.
So, to eliminate all unknowns, "I go back to the beginning." I go as low as Perl lets me go (system IO), and don't rely on anything that PerlIO might layer on top. If I do that, I don't have to peel back the layers of every onion and figure out what is happening and then compensate when necessary. Honestly how can anyone ensure that what is written is what was intended, when using PerlIO, every time, on every platform, in every Perl? We can't, and NOT exclusively because of Perl. We don't know, and can't know, specifically because we don't always know what people intended to write; we don't know the human element's every intention.
So this is where I sit scratching my head. It's a balancing act of decoding both Perl behavior and human intention.
Some, I'm sure, will say that I'm a fool for overthinking things, or that I'm worrying too much. But even if I have to open myself up to a lambasting because of admittance that I don't completely know how to solve a problem because to me it's nature remains somewhat of mystery, I will hazard the shame <mild sarcasm> if I'm able to gain greater understanding by having posited my questions to the collectively much-more-intelligent community. Can anyone shed more light? Or is the answer to just go with PerlIO, "accept the defaults", trust in the decisions that are built into PerlIO, and let the chips fall where they may?
Folks, I don't know how to eliminate all unknowns, other than to use system IO. When I do so, I do it under the assumption that I'm probably going to be able to safely write out all content, all the time, on all Perls, on all systems, without creating filename.mojibake when I wanted to create filename.txt -- so long as my output stream wasn't mojibake already. If input was garbage, the output to a file will be garbage, surely...but it won't be because of me.
I do understand that if a stream is already mangled, that it's going to still be mangled when written via syswrite. I do expect that some will say that using PerlIO might have corrected the issue in some situations (maybe). But for now I am operating under the assumption that if I don't do anything behind the curtains (DWIM), I won't later have anything to explain. I can honestly say, hey, my software wrote out what you fed it.
A mistake can be valuable or costly, depending on how faithfully you pursue correction