in reply to Re^3: encoding problem om Ubuntu Linux
in thread encoding problem om Ubuntu Linux

Aha - I see.

SV = PV(0x8154064) at 0x8153594
REFCNT = 1
FLAGS = (PADBUSY,PADTMP,POK,READONLY,pPOK)
PV = 0x8187b58 "/media/usbdisk/music/checkit/Crosby, Stills, Nash & Young/D\303\251 Vu/06 - D\303\251 Vu.mp3"\0
CUR = 80
LEN = 84

Replies are listed 'Best First'.
Re^5: encoding problem om Ubuntu Linux
by moritz (Cardinal) on Jun 20, 2008 at 11:19 UTC
    The file name you're quoting here doesn't contain a 'j' at all, so even if all your encoding issues are fixed, you won't get a match. The \303\251 sequence is just an UTF-8 encoded é.

    Please post a bit of your script so that we can see what you're actually doing. Missing characters are rather atypical for encoding problems (at least in that form).

    Update: after thinking a bit more about your problem, I think your console/terminal might be b0rked. Can you enter and view accented characters on the command line? Are the entries of the file displayed correctly if you simply cat them to the console?

Re^5: encoding problem om Ubuntu Linux
by almut (Canon) on Jun 20, 2008 at 11:21 UTC
    PV = 0x8187b58 "/media/usbdisk/music/checkit/Crosby, Stills, Nash & Yo +ung/D\303\251 Vu/06 - D\303\251 Vu.mp3"\0

    Ok, so that's valid UTF-8, too, but a different file name, i.e. ".../Dé Vu/06 - Dé Vu.mp3"... (somehow, the "jà" is missing).

    When you view the CSV file with some other tool/editor (which one are you using?), what do you see displayed?

      I't getting more and more complex... (putty was somehow reset to something other than UTF8 which made it all more tricky

      1) the directory itself
      root@slarti:/media/usbdisk/music_org/scripts/test# ls /media/usbdisk/music/checkit/Crosby\,\ Stills\,\ Nash\ \&\ Young/
      Déjà Vu

      So... that seems to be Ok (assuming this correctly being viewd while at UTF8)

      2) The source file's content (part of it at least)
      root@slarti:/media/usbdisk/music_org/scripts/test# grep 'Crosby\,\ Stills\,\ Nash\ \&\ Young' mp3_v4.csv
      /media/usbdisk/music/checkit/Crosby, Stills, Nash & Young/D▒j▒ Vu/01 - Carry On.mp3;Crosby, Stills, Nash & Young;D?j?▒ Vu;Carry On;1;Rock;1970

      Now, still being in UTF8 mode this does not look good

      3) running a check-script of mine...
      #!/usr/bin/perl

      # INIT ----
      use strict;
      use warnings;

      # VARS ----

      # SUBS ----


      # MAIN ----
      print "Opening list...\n";
      my $file = shift;

      open (F, $file) or die "$0\n";
      while (my @fields = split /;/, <F>, 8) {
      unless (-f $fields[0]) {
      print $fields[0] . ": not found\n";
      }
      }
      close (F);

      print "Ready\n";


      Output: (part of it)
      root@slarti:/media/usbdisk/music_org/scripts/test# ./check_paths.pl mp3_v4.csv | grep Stills /media/usbdisk/music/checkit/Crosby, Stills, Nash & Young/D▒j▒ Vu/01 - Carry On.mp3: not found

      This means the path in the sourcefile does not match the actual subdirectory

      4) The fun part: vi mp3_v2.csv
      /media/usbdisk/music/checkit/Crosby, Stills, Nash & Young/Déjà Vu/01 - Carry On.mp3;Crosby, Stills, Nash & Young;D?j?ï Vu;Carry On;1;Rock;1970

      Compare this output to step 2 which does a grep on this very same file. Now all of a sudden it's looking good again. The source files has not been edited and all these outputs have been taken in the same putty windows while check before and aft on the encoding. UTF8 all the time.

      Now I'm puzzled.
      I think I'm gonna rub Buddha's belly - hopefully it'll provide me with some luck :-)

        Rather than trying to come up with explanations for why get what you get...  let me just recommend that, when debugging encoding problems, you start by looking at the raw data1 with as few tools in between as possible. Every tool (terminal, putty, grep, vi, browser, whatever) or processing step might have it's own problems or automagic-isms with one or the other encoding, so what you see may not necessarily be what you really have.

        My preferred tool in cases like these is a classical hexdump, or as ikegami suggested, a Devel::Peek dump (when in Perl). Together with some knowledge of how the various encodings represent data, this usually allows you to figure out what's going wrong, eventually.

        In this particular case, I would start by looking at a hexdump of the CSV file (e.g. with the command line tool hexdump, which is installed/available on most distros). Then we would reliably know what the "Déjà" is represented as in the CSV file.

        ___

        1 though Juerd might disagree, saying that you shouldn't, unless you're an expert already...

        What are the values you get when typing
        :set encoding :set fileencoding
        in vim?

        What do you get when you simply enter locale on the command line?