in reply to Re^2: encoding problem om Ubuntu Linux
in thread encoding problem om Ubuntu Linux

Ad 1: ... PV = 0x81757e8 "mp3_v4.csv"\0

I think the file name in the CSV file - the one with the "Déjà" - would be more interesting :) — which is what ikegami meant...

Ad 2: ... PV = 0x817dad8 "D\303\251j\303\240 Vu"\0

This verifies that the filesystem encoding of that file is UTF-8. So, presumably, the file name in the CSV file isn't in UTF-8.

Replies are listed 'Best First'.
Re^4: encoding problem om Ubuntu Linux
by j.goor (Acolyte) on Jun 20, 2008 at 11:08 UTC
    Aha - I see.

    SV = PV(0x8154064) at 0x8153594
    REFCNT = 1
    FLAGS = (PADBUSY,PADTMP,POK,READONLY,pPOK)
    PV = 0x8187b58 "/media/usbdisk/music/checkit/Crosby, Stills, Nash & Young/D\303\251 Vu/06 - D\303\251 Vu.mp3"\0
    CUR = 80
    LEN = 84
      The file name you're quoting here doesn't contain a 'j' at all, so even if all your encoding issues are fixed, you won't get a match. The \303\251 sequence is just an UTF-8 encoded é.

      Please post a bit of your script so that we can see what you're actually doing. Missing characters are rather atypical for encoding problems (at least in that form).

      Update: after thinking a bit more about your problem, I think your console/terminal might be b0rked. Can you enter and view accented characters on the command line? Are the entries of the file displayed correctly if you simply cat them to the console?

      PV = 0x8187b58 "/media/usbdisk/music/checkit/Crosby, Stills, Nash & Yo +ung/D\303\251 Vu/06 - D\303\251 Vu.mp3"\0

      Ok, so that's valid UTF-8, too, but a different file name, i.e. ".../Dé Vu/06 - Dé Vu.mp3"... (somehow, the "jà" is missing).

      When you view the CSV file with some other tool/editor (which one are you using?), what do you see displayed?

        I't getting more and more complex... (putty was somehow reset to something other than UTF8 which made it all more tricky

        1) the directory itself
        root@slarti:/media/usbdisk/music_org/scripts/test# ls /media/usbdisk/music/checkit/Crosby\,\ Stills\,\ Nash\ \&\ Young/
        Déjà Vu

        So... that seems to be Ok (assuming this correctly being viewd while at UTF8)

        2) The source file's content (part of it at least)
        root@slarti:/media/usbdisk/music_org/scripts/test# grep 'Crosby\,\ Stills\,\ Nash\ \&\ Young' mp3_v4.csv
        /media/usbdisk/music/checkit/Crosby, Stills, Nash & Young/D▒j▒ Vu/01 - Carry On.mp3;Crosby, Stills, Nash & Young;D?j?▒ Vu;Carry On;1;Rock;1970

        Now, still being in UTF8 mode this does not look good

        3) running a check-script of mine...
        #!/usr/bin/perl

        # INIT ----
        use strict;
        use warnings;

        # VARS ----

        # SUBS ----


        # MAIN ----
        print "Opening list...\n";
        my $file = shift;

        open (F, $file) or die "$0\n";
        while (my @fields = split /;/, <F>, 8) {
        unless (-f $fields[0]) {
        print $fields[0] . ": not found\n";
        }
        }
        close (F);

        print "Ready\n";


        Output: (part of it)
        root@slarti:/media/usbdisk/music_org/scripts/test# ./check_paths.pl mp3_v4.csv | grep Stills /media/usbdisk/music/checkit/Crosby, Stills, Nash & Young/D▒j▒ Vu/01 - Carry On.mp3: not found

        This means the path in the sourcefile does not match the actual subdirectory

        4) The fun part: vi mp3_v2.csv
        /media/usbdisk/music/checkit/Crosby, Stills, Nash & Young/Déjà Vu/01 - Carry On.mp3;Crosby, Stills, Nash & Young;D?j?ï Vu;Carry On;1;Rock;1970

        Compare this output to step 2 which does a grep on this very same file. Now all of a sudden it's looking good again. The source files has not been edited and all these outputs have been taken in the same putty windows while check before and aft on the encoding. UTF8 all the time.

        Now I'm puzzled.
        I think I'm gonna rub Buddha's belly - hopefully it'll provide me with some luck :-)