I't getting more and more complex... (putty was somehow reset to something other than UTF8 which made it all more tricky
1) the directory itself
root@slarti:/media/usbdisk/music_org/scripts/test# ls /media/usbdisk/music/checkit/Crosby\,\ Stills\,\ Nash\ \&\ Young/
Déjà Vu
So... that seems to be Ok (assuming this correctly being viewd while at UTF8)
2) The source file's content (part of it at least)
root@slarti:/media/usbdisk/music_org/scripts/test# grep 'Crosby\,\ Stills\,\ Nash\ \&\ Young' mp3_v4.csv
/media/usbdisk/music/checkit/Crosby, Stills, Nash & Young/D▒j▒ Vu/01 - Carry On.mp3;Crosby, Stills, Nash & Young;D?j?▒ Vu;Carry On;1;Rock;1970
Now, still being in UTF8 mode this does not look good
3) running a check-script of mine...
#!/usr/bin/perl
# INIT ----
use strict;
use warnings;
# VARS ----
# SUBS ----
# MAIN ----
print "Opening list...\n";
my $file = shift;
open (F, $file) or die "$0\n";
while (my @fields = split /;/, <F>, 8) {
unless (-f $fields[0]) {
print $fields[0] . ": not found\n";
}
}
close (F);
print "Ready\n";
Output: (part of it)
root@slarti:/media/usbdisk/music_org/scripts/test# ./check_paths.pl mp3_v4.csv | grep Stills
/media/usbdisk/music/checkit/Crosby, Stills, Nash & Young/D▒j▒ Vu/01 - Carry On.mp3: not found
This means the path in the sourcefile does not match the actual subdirectory
4) The fun part: vi mp3_v2.csv
/media/usbdisk/music/checkit/Crosby, Stills, Nash & Young/Déjà Vu/01 - Carry On.mp3;Crosby, Stills, Nash & Young;D?j?ï Vu;Carry On;1;Rock;1970
Compare this output to step 2 which does a grep on this very same file. Now all of a sudden it's looking good again. The source files has not been edited and all these outputs have been taken in the same putty windows while check before and aft on the encoding. UTF8 all the time.
Now I'm puzzled.
I think I'm gonna rub Buddha's belly - hopefully it'll provide me with some luck :-) | [reply] |
Rather than trying to come up with explanations for why get what
you get... let me just recommend that, when debugging encoding
problems, you start by looking at the raw data1 with as few
tools in between as possible. Every tool (terminal, putty,
grep, vi, browser, whatever) or processing step might have it's own problems or
automagic-isms with one or the other encoding, so what you see may not
necessarily be what you really have.
My preferred tool in cases like these is a classical hexdump, or as
ikegami suggested, a Devel::Peek dump (when in Perl). Together with
some knowledge of how the various encodings represent data, this
usually allows you to figure out what's going wrong, eventually.
In this particular case, I would start by looking at a hexdump of
the CSV file (e.g. with the command line tool hexdump, which is
installed/available on most distros). Then we would reliably know what
the "Déjà" is represented as in the CSV file.
___
1 though Juerd might disagree, saying that
you shouldn't, unless you're an expert already...
| [reply]
[d/l] |
What are the values you get when typing
:set encoding
:set fileencoding
in vim?
What do you get when you simply enter locale on the command line? | [reply]
[d/l]
[select] |
root@slarti:/media/usbdisk/music_org/scripts/test# localev
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=
:set encoding
encoding=utf-8
:set fileencoding
fileencoding=latin1
| [reply] |