I am running the latest release of ActivePerl on Windows Server 2003 and have run into a problem that has me completly confounded. I have a plain text file that I know is Unicode. I need to open this file and see if one of the lines contains certain data. So, I: 
open FILE, $myfile
while(<FILE>) {
    if(/Information Store/i) {
        $found = 1;
        last;
    }
}

[download]
Not surprisingly, it doesn't work. So I do some reading, peruse perluniintro, perlunicode and various other documentation and find that what I REALLY need to do is: 
open FILE, "<:utf8", $myfile
while(<FILE>) {
    if(/Information Store/i) {
        $found = 1;
        last;
    }
}

[download]
Still doesn't work. Inside the while loop, I place a print "$_\n" and find that each line is printed with what appear to be spaces between each letter. This is the case whether I use the first open method or the second. So, just to see what Perl thinks, I insert utf8::is_unicode($_) in the while loop. Returns 1 every time no matter how I open the file. Obviously Perl believes this to be a Unicode string. I try using utf8::decode on the string to see what it turns into, only to find out that although the function returns 1 for success, the string is completly unchanged. So, in desparation, I attempt to turn the string into an array of bytes using unpack. Both unpack "U*" and unpack "C*" return the exact same character array, and in both, the "space" between characters is reported as having a value of 48. According to the ASCII character charts, this should print as a 0 (zero). Now I'm completly dumbfounded. I don't care if Perl thinks this is a Unicode string, ASCII string, EBCDIC string, or anything else it wants. I just want to be able to use a regular expression to find certain data within this string and I can't. Does anyone have any advice on how to get this accomplished?

In reply to Unicode and text files by dirtdart

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.