Zapawork has asked for the wisdom of the Perl Monks concerning the following question:

Hello Monks,
I have a strange issue. I'm reading what appears to be a normal text file written by a blackberry server.
When i load in the first line of the log file that defines its columns it appears in memmory in the komodo debugger and in print statements as: 
"ÿþ"\x00N\x00a\x00m\x00e\x00.\x00I\x00D\x00"\x00,\x00"\x00P\x00I\x00N\
+x00"\x00,\x00"\x00E\x00m\x00a\x00i\x00l\x00 \x00A\x00d\x00d\x00r\x00e
+\x00s\x00s\x00"\x00,\x00"\x00T\x00y\x00p\x00e\x00 \x00o\x00f\x00 \x00
+M\x00e\x00s\x00s\x00a\x00g\x00e\x00"\x00,\x00"\x00T\x00o\x00"\x00,\x0
+0"\x00C\x00c\x00"\x00,\x00"\x00B\x00c\x00c\x00"\x00,\x00"\x00F\x00r\x
+00o\x00m\x00"\x00,\x00"\x00S\x00u\x00b\x00j\x00e\x00c\x00t\x00"\x00,\
+x00"\x00B\x00o\x00d\x00y\x00"\x00,\x00"\x00S\x00e\x00n\x00d\x00/\x00R
+\x00e\x00c\x00e\x00i\x00v\x00e\x00d\x00 \x00D\x00a\x00t\x00e\x00"\x00
+,\x00"\x00S\x00e\x00r\x00v\x00e\x00r\x00 \x00L\x00o\x00g\x00 \x00D\x0
+0a\x00t\x00e\x00"\x00,\x00"\x00O\x00v\x00e\x00r\x00a\x00l\x00l\x00 \x
+00M\x00e\x00s\x00s\x00a\x00g\x00e\x00 \x00S\x00t\x00a\x00t\x00u\x00s\
+x00"\x00,\x00"\x00C\x00o\x00m\x00m\x00a\x00n\x00d\x00"\x00,\x00"\x00U
+\x00I\x00D\x00"\x00" <br><br>
for any match that i do and any print that i do.

[download]
However, if i run split and print it out i get: 
"Name.ID","PIN","Email Address","Type of Message","To","Cc","Bcc","Fro
+m","Subject","Body","Send/Received Date","Server Log Date","Overall M
+essage Status","Command","UID"

[download]
What is going on here? I need to do a regex match and print the string normally. 
  open FILE, "<", $file or die "open: $!"; 


   
        while (<FILE>)
        {
            my $line2 = $_;
    
            chomp $line2;
            my @line = split (/","/,$line2);
           
            if ( $line[0] =~ m/Name.ID/)
            {
                print "here \n";
                next;
            }
            else
            {
                print "$line[0] \n";
            }
            
           
            print "$line[0],$line[1],$line[2],$line[3],$line[4],$line[
+5],$line[6],$line[7],$line[8],$line[9],$line[10],$line[11],$line[12],
+$line[13],$line[14]";
 
        }
    }
}

[download]

result:
ÿþ"
"Name.ID","PIN","Email Address","Type of Message","To","Cc","Bcc","Fro
+m","Subject","Body","Send/Received Date","Server Log Date","Overall M
+essage Status","Command","UID"&#11308;&#11308;&#11308;&#11308;&#11308
+;&#11308;&#11308;

[download]
Dave -- Saving the world one node at a time

Replies are listed 'Best First'.
Re: Dealing with strange data encoding issue
by jhourcle (Prior) on Feb 20, 2009 at 17:13 UTC

    Looks like UTF-16 (aka UCS-2) to me.

    See perlunicode

    update: I guess I'm still learning ... but that explains why I don't see references to UCS-2 anymore

[reply]

      UTF-16 is not quite the same thing as UCS-2.

      • UCS-2 uses a single 16-bit word per character. It is therefore unable to serialize characters other than U+0000 .. U+FFFF.
      • UTF-16 uses a variable number of 16-bit words per character, and is thus able to access all UNICODE characters.

      Basically, UCS-2 is to UTF-16 as iso-latin-1 is to UTF-8.

      There's not enough info to know whether the encoding used in the OP is UCS-2 or UTF-16. To play it safe, accept UTF-16 and send UCS-2.

      Update: Noteable flaw in the comparison:

      • Characters supported by UCS-2 are encoded identically in UTF-16.
      • Characters may be encoded differently in iso-latin-1 and UTF-8.
[reply]
      UTF-16 it is!

      Thank you so much
      Dave -- Saving the world one node at a time
[reply]
Re: Dealing with strange data encoding issue
by Bloodnok (Vicar) on Feb 20, 2009 at 18:32 UTC
    Judging by the format you list, I'll bet a pound to a pinch of s#!t that your Blackberry server is a Windoze box - you have my every sympathy.

    In a previous incarnation i.e. contract, I experienced a similar problem to yourself - but I'm afraid to say that, IIRC, our solution to the problem merely involved a global substitution pre-processing step ... until we found an undocumented feature of the application in question actually knew how to interpret Windoze-speak, ostensibly plain text, files!!!

    Arrgghh, imagine the frustration - over and above having to use Windoze in the first place, that is.

    A user level that continues to overstate my experience :-))
[reply]

Back to Seekers of Perl Wisdom