use utf8; does more than allow UTF-8 characters to be in the source file (in strings and even for identifier names!). After all, there weren't any, so I didn't include it for that purpose. Dig a little deeper, or try leaving it out and see what happens (after knowing the original works OK (fixed typos, agrees with your data format, etc.)).

The chart: I don't use a chart, but I can convert to/from UTF-8 on a whiteboard. Can you? Look at the reason for the numbers, not just at the numbers, in binary. Go back to the original source document on UTF-8 if it's not explained in the Perl docs. You can find it at unicode.org, or in the back of the book if you own a copy.

Matching multibyte chars explicitly: I did that years ago and wished for better. Now, it's unnecessary. Why would you need to do that?

++$char{$&} while (/[^\0-\x{7f}]/g);
[download]

Unsure of unpack: You're not calling it in your sample, so I don't know what you mean. Leftover from another test-run, I suppose. You might be thinking:

foreach (unpack "U*") {
   ++$chars{$_}  if (ord($_) > 127);
   }

# ... later
while (my ($ch, $count)= each %chars) {
   printf "character U+%04x seen $count times.\n", $ch;
   }
[download]

—John

wow!

This was trivial now that it's done:

#require 5.6;
use strict;
use warnings;
use utf8;

my %chars;
my %descids;
while (<>) {
   while ( /[^\x{1}-\x{7f}]/g)  {
      ++$chars{$&};
      }
   }
foreach my $char (keys %chars){
    print "$char found $chars{$char} times\n";
}
print "found ". keys(%chars) . " distinct non-ascii chars\n";
[download]

I had to comment out the version of perl: I have 5.6.1, and the compiler complained at that (that should be easy to fix). I next checked if any characters lie outside of ISO-8859-1 by changing the regex range to look up to \x{ff}, and got zero. That is the practical result of this whole exercise; this 58k tabbed text file will be easier to import into various DBMS systems if the user knows that the characters lie in the range of Latin-1

The use utf8 directive is absolutely essential; the unicode hex notation is not allowed in the regex without it.

The inner while loop (vs an if statement) around the regex is a little unclear; I guess the match with the /g modifier returns a list, and the if statement would only chec the scalar return. Would something like this capture all the matches in a single line into a list?

while(<>){
  while (my @matches = /[^\x{1}-\x{7f}]/g){
     $conid = /patten-to-find-this-column/;
     $hash_of_lists{$conid} =[@matches];
# linking this with inner hash of found characters is fuzzy but near..
+.
     ++$chars{$&};
  }
}
[download]

My next task is to make some data structures; at the top level are concept_ids (one of the fields in this table). Each concept-id is associated with numerous description_ids (the primary key of this table). Each row of this table (each description_id) could have numerous non-ascii characters, each associated with a frequency.

I intend to collect this all into a hash of lists of hashes of hashes.

The inner hash is the non-ascii characters and their frequency. The list of hashes is the row of the table with its non-ascii characters; each row could have a number of distinct non-ascii characters in it. And the hash of lists is the unique concept_id associated with numerous description_ids. After I have that, I'll want the individual words with the characters also collected and reported somehow, but that will come last.

This will take some thinking; I'm taking a company trip tomorrow and can work this out in the hotel. I might not be able to post for about a week, depending on internet access.

Your help is much appreciated.

John

>> I had to comment out the version of perl: I have 5.6.1, and the compiler complained at that (that should be easy to fix).

That's odd. I put it in to make it clear that you didn't need 5.8, and that older than 5.6 wouldn't work. It's a good idea in posts for that reason, even if you don't use it in your own scripts that you don't share with anyone or put on multiple machines.

Look up the syntax of require and use in the perlfunc manpage, and see what the problem is. Use perl -v on the command line to see what version is actually running.

>> The use utf8 directive is absolutely essential; the unicode hex notation is not allowed in the regex without it.

Correct. That's one reason I used \x{7f} instead of \x7f which would mean the same thing: to make sure it was compiled with Unicode support turned on. Then taking out the use like I suggested would produce a meaningful error and tell you what else use utf8 affects.

The /g modifier doesn't make the regex return a list of all matches. The context controls that, and it's being called in scalar context.

Rather, the /g will return one at a time, and each time through the while loop will get the next one. See, the regex stuff is remembering the state in an object that's created when you use the matching syntax. This is important to understand, and you'll see it in a few other places in Perl, too.

The idea of calling context is important in Perl. I see you wrote print "found ". keys(%chars) . " distinct non-ascii chars\n";. What would happen if you used commas instead of the two dots? Why does it change its meaning?

Your variation using my @matches= will indeed get all the matches at once. So why use a while loop? the rest of it doesn't make sence. Use it this way:

my $_= "sdfdsf,65dfdf**3#ooijoi4asdfdsf.";
my @matches= /[a-z]/g;
print "I found: @matches\n";
[download]

When you get back, try a new top-level thread, and be detailed since it will be picked up by new readers.

Parsing each line in the file into fields? What format is it in? If it's comma-delimited text or something like that, there is a module which does that. If it's a unique character that doesn't appear within the field itself, just use the split function.

—John