http://qs1969.pair.com?node_id=555948


in reply to Re^5: How is perl able to handle the null byte?
in thread How is perl able to handle the null byte?

This is the passage from the ('a', possibly superceded), ANSI/ISO C spec that worries me:

From Wikipedia (my emphasis).

The Unicode standard 4.0 says that "ANSI/ISO C leaves the semantics of the wide character set to the specific implementation but requires that the characters from the portable C execution set correspond to their wide character equivalents by zero extension."
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal?
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.
  • Comment on Re^6: How is perl able to handle the null byte?

Replies are listed 'Best First'.
Re^7: How is perl able to handle the null byte?
by graff (Chancellor) on Jun 18, 2006 at 00:42 UTC
    That quotation comes from section 5.2 of the Unicode standard (in the current 4.1 version), and it is about the "wchar_t" data type in C. If/when the Perl-to-C interface includes conversions from (say) utf8 Perl strings to C wchar_t*, this detail would need to be taken into account.

    I don't actually know whether XS covers conversion from SV to wchar_t at present, but I believe this is not relevant to the SV-to-char* issue being discussed in this thread.

[reply]

      My point is that when you encode strings to formats other than Perl's internal utf8 format, you end up with a scalar without the utf flag set, in which multiple bytes are used to represent single characters. In some of these encodings, some individual bytes of the multi-byte characters can be null. Eg. As you pointed out the UTF-16 variants.

      These encoded values often need to be passed to system apis. At that point, warning or dieing because the scalar contains embedded nulls would be wrong and if Perl was to try and implement such warnings or traps for embedded null bytes, it would need to detect the above situations.

      Example. The often questioned, but (to my knowledge), yet to be resolved problem of globbing the windows filesystem for files with unicode filenames. Win32 has (for a long time now), a very functional set of APIs for dealing with Unicode (wide) filenames. In order to use them, it is necessary to convert the path and/or wildcard inputs to the FindFirstFileW() call, to Window's internal Unicode representation, UTF-16(LE). This can be done using Encode::encode(). Having used this call (say) 

      ## Why oh why does it try to modify the input!?
my $uPath = encode( 'UTF-16LE', $_ = "\\\\?\\c:\\some\\path", 1 );

      [download]

      The scalar $uPath will contain the UTF-16LE representation of the input. 

      use Encode;

print encode( 'UTF-16LE', $_ = "\\\\?\\C:\\some\\path", 1 );;
\ \ ? \ C : \ s o m e \ p a t h

$uPath = encode( 'UTF-16LE', $_ = "\\\\?\\C:\\some\\path", 1 );;

print unpack 'H*', $uPath;;
5c005c003f005c0043003a005c0073006f006d0065005c007000610074006800

      [download]

      Now I need to pass this as the first parameter to FindFirstFileW()--which is defined as 

      HANDLE FindFirstFileW( LPCTSTR lpFileName, LPWIN32_FIND_DATA lpFindFil
+eData )

LPTCSTR => long pointer to (array of) TCHAR
TCHAR   => wchar_t

      [download]

      either directly through Win32::API, or indirectly through glob or opendir; but if embedded null detection was implemented, it would warn or die because every other byte of the PV contains a null--but that is intentional and correct.

      So, not only would Perl need to detect embedded nulls, it would also need to detect cases where embedded nulls are legitimate. The only way I can see that it could do that is if an additional field where added to the SV to record the encoding contained by the SV--a not inconsiderable expense.

      And that's completely ignoring the fact that SVs can and often do contain arbitrary binary data that can legitimately contain nulls and for which there would be no generic mechanism for flagging as exempt from embedded null byte detection.

      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal?
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
[d/l]
[select]

In Section Seekers of Perl Wisdom