Re^8: treat files with umlauts (utf)

The OP's posted code does not include an assignment to $scandir

It does not, but the OP does mention UTF-8, so $scandir being UTF-8 is a possibility, to say the least.

The OP's posted code does not contain umlauts

It doesn't have to be umlauts, what matters is whether $scandir is UTF-8. Make this change to your code and see what happens:

use utf8; # just for utf8::upgrade

# bytewise, this is already UTF-8...
my $scandir = './pm_1080490_utf8_readdir';

#... but we need to flag it as such for
# the problem to manifest itself:
utf8::upgrade $scandir;
# now on to readdir
[download]

the OP's posted code does not require use utf8

Right, it does not. And use utf8 is not absolutely necessary in the above test code - use -CS/binmode or -CA to initialize $scandir.

If you're referring to the output from that containing: FLAGS = (PADMY,POK,pPOK,UTF8) Then the UTF8 part of that is caused by the umlaut in 'für'.

Does that code not answer your:

[use utf8] has nothing to do with the:... flagging variables

# under 'use utf8'
    FLAGS = (PADMY,POK,pPOK,UTF8)
    ... "f\303\274r"\0 [UTF8 "f\x{fc}r"]
# no utf8
    FLAGS = (PADMY,POK,pPOK)
    ... "f\303\274r"\0
[download]

Does the first variable have the UTF8 flag or does it not? What about the second variable? Aren't those two strings exactly the same?
I'm out of this thread.

Comment on Re^8: treat files with umlauts (utf) Select or Download Code

Replies are listed 'Best First'.
Re^9: treat files with umlauts (utf) by mike.scharnow (Initiate) on Apr 07, 2014 at 08:25 UTC
Hello all! Thank you all for your ideas and for your discussion, which taught me some more internals abuot UTF-Handling. I hope I will be able to work in the Umlaut field without further problems. as to the cause of the problem (as I understand it now): You gave me the correct hints: it was not the problem of readdir but the problem of $scandir. I have a configuration xml file, which I read in using XML::Simple. $scandir is read from this file using something like `my $scandir = $config->{external_systems}->{filesIN}`. Now, the config file is stored in ISO-8859-1. It seems that in this construction, $scandir is not stored as UTF, but as ISO-8859-1, although there are no umlauts in the directory name! Now, when I concat $scandir with the result of readdir, it seems that a non-utf-value (from xml file) is concatenated with an utf value (from readdir). And as soon as there is an umlaut in the filename, the resulting string is invalid, causing "-f" to say "this is not a file". I solved it by writing `my $scandir = …; utf::downgrade($scandir);` [download] Then I could successfully read, copy and move the files. Hoping that this is the "correct" way of dealing with the problem and again thanks very much Mike	[reply] [d/l] [select]

Replies are listed 'Best First'.

Re^9: treat files with umlauts (utf)
by mike.scharnow (Initiate) on Apr 07, 2014 at 08:25 UTC

Hello all!

Thank you all for your ideas and for your discussion, which taught me some more internals abuot UTF-Handling. I hope I will be able to work in the Umlaut field without further problems.

as to the cause of the problem (as I understand it now): You gave me the correct hints: it was not the problem of readdir but the problem of $scandir.

I have a configuration xml file, which I read in using XML::Simple. $scandir is read from this file using something like my $scandir = $config->{external_systems}->{filesIN}.

Now, the config file is stored in ISO-8859-1. It seems that in this construction, $scandir is not stored as UTF, but as ISO-8859-1, although there are no umlauts in the directory name!

Now, when I concat $scandir with the result of readdir, it seems that a non-utf-value (from xml file) is concatenated with an utf value (from readdir). And as soon as there is an umlaut in the filename, the resulting string is invalid, causing "-f" to say "this is not a file".

I solved it by writing

my $scandir = …;
utf::downgrade($scandir);
[download]

Hoping that this is the "correct" way of dealing with the problem and again thanks very much

Mike

[reply]
[d/l]
[select]