Esteemed Monks,

I am searching for unicode-filenames on disk with File::Find::Rule in this way:

use utf8;

use Test::More;
use Test2::Plugin::UTF8;
use Devel::Peek;
use Encode;
use File::Find::Rule;

# the file's 1st letter is greek kappa,
# 2nd letter is iota accented (with tonos)
my $expected = 'test/κί.png';
my @gots = map { Encode::decode_utf8($_) }
  File::Find::Rule->file()->name(qr/.*\.png/)->in('test')
;
is(scalar(@gots), 1, "got 1 item");
is($gots[0], $expected, "name is as expected");

Dump($expected);
Dump($gots[0]);

done_testing();

[download]

Dir 'test' contains one file: κί.png (κί.png, kappa, iota accented dot png).

In Linux, the above succeeds and prints the filenames as:

  FLAGS = (POK,IsCOW,pPOK,UTF8)
  PV = 0x561260048d00 "test/\xCE\xBA\xCE\xAF.png"\0 [UTF8 "test/\x{3ba
+}\x{3af}.png"]
  CUR = 13
  LEN = 15

  FLAGS = (POK,IsCOW,pPOK,UTF8)
  PV = 0x561260249060 "test/\xCE\xBA\xCE\xAF.png"\0 [UTF8 "test/\x{3ba
+}\x{3af}.png"]
  CUR = 13
  LEN = 15

[download]

In OSX (10.13 high sierra whatever), it fails and it prints:

  FLAGS = (POK,IsCOW,pPOK,UTF8)
  PV = 0x7fd35e7d8b60 "test/\316\272\316\257.png"\0 [UTF8 "test/\x{3ba
+}\x{3af}.png"]
  CUR = 13
  LEN = 15

  FLAGS = (POK,IsCOW,pPOK,UTF8)
  PV = 0x7fd35e7ee4e0 "test/\316\272\316\271\314\201.png"\0 [UTF8 "tes
+t/\x{3ba}\x{3b9}\x{301}.png"]
  CUR = 15
  LEN = 17

[download]

The difference is that in Linux the "greek iota with tonos (accented)" is just 1 character: U+03AF (hex \x{3af}). And this comes out both when I typed it in the script (the $expected) and when File::Find::Rule found the corresponding file on disk.

In OSX however, the expected value is just as in linux. But File::Find::Rule finds the file on disk and its name is now : kappa, iota and *separate* accent (\x{3ba}\x{3b9}\x{301}). It looks the same but test fails.

Additional info:

The raw output of File::Find::Rule is:

OSX: PV = 0x7f8724885690 "test/\316\272\316\271\314\201.png"\0 <<<< se
+parate accent
Linux: PV = 0x5566b6bc05d0 "test/\xCE\xBA\xCE\xAF.png"\0

[download]

So, it seems the culprit is File::Find::Rule who adds the separate accent.

The program works find I guess because the OS treats these filenames with accented unicode chars correctly. What's bothering me is that my tests fail.

Am I doing things wrong? I need some advice.

bw, bliako

5min Edit: changed the title

In reply to File::Find::Rule returns different filenames if they have chars with accents: OSX vs Linux by bliako

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.