This was trivial now that it's done:

#require 5.6;
use strict;
use warnings;
use utf8;

my %chars;
my %descids;
while (<>) {
   while ( /[^\x{1}-\x{7f}]/g)  {
      ++$chars{$&};
      }
   }
foreach my $char (keys %chars){
    print "$char found $chars{$char} times\n";
}
print "found ". keys(%chars) . " distinct non-ascii chars\n";
[download]

The use utf8 directive is absolutely essential; the unicode hex notation is not allowed in the regex without it.

The inner while loop (vs an if statement) around the regex is a little unclear; I guess the match with the /g modifier returns a list, and the if statement would only chec the scalar return. Would something like this capture all the matches in a single line into a list?

while(<>){
  while (my @matches = /[^\x{1}-\x{7f}]/g){
     $conid = /patten-to-find-this-column/;
     $hash_of_lists{$conid} =[@matches];
# linking this with inner hash of found characters is fuzzy but near..
+.
     ++$chars{$&};
  }
}
[download]

I intend to collect this all into a hash of lists of hashes of hashes.

The inner hash is the non-ascii characters and their frequency. The list of hashes is the row of the table with its non-ascii characters; each row could have a number of distinct non-ascii characters in it. And the hash of lists is the unique concept_id associated with numerous description_ids. After I have that, I'll want the individual words with the characters also collected and reported somehow, but that will come last.

This will take some thinking; I'm taking a company trip tomorrow and can work this out in the hotel. I might not be able to post for about a week, depending on internet access.

Your help is much appreciated.

John

>> I had to comment out the version of perl: I have 5.6.1, and the compiler complained at that (that should be easy to fix).

That's odd. I put it in to make it clear that you didn't need 5.8, and that older than 5.6 wouldn't work. It's a good idea in posts for that reason, even if you don't use it in your own scripts that you don't share with anyone or put on multiple machines.

Look up the syntax of require and use in the perlfunc manpage, and see what the problem is. Use perl -v on the command line to see what version is actually running.

>> The use utf8 directive is absolutely essential; the unicode hex notation is not allowed in the regex without it.

Correct. That's one reason I used \x{7f} instead of \x7f which would mean the same thing: to make sure it was compiled with Unicode support turned on. Then taking out the use like I suggested would produce a meaningful error and tell you what else use utf8 affects.

The /g modifier doesn't make the regex return a list of all matches. The context controls that, and it's being called in scalar context.

Rather, the /g will return one at a time, and each time through the while loop will get the next one. See, the regex stuff is remembering the state in an object that's created when you use the matching syntax. This is important to understand, and you'll see it in a few other places in Perl, too.

The idea of calling context is important in Perl. I see you wrote print "found ". keys(%chars) . " distinct non-ascii chars\n";. What would happen if you used commas instead of the two dots? Why does it change its meaning?

Your variation using my @matches= will indeed get all the matches at once. So why use a while loop? the rest of it doesn't make sence. Use it this way:

my $_= "sdfdsf,65dfdf**3#ooijoi4asdfdsf.";
my @matches= /[a-z]/g;
print "I found: @matches\n";
[download]

When you get back, try a new top-level thread, and be detailed since it will be picked up by new readers.

Parsing each line in the file into fields? What format is it in? If it's comma-delimited text or something like that, there is a module which does that. If it's a unique character that doesn't appear within the field itself, just use the split function.

—John

A quick note and thank you...
All worked out;
require needs "v"
the while loop was unnecessary as you indicated
the data structures allow me to pull out all the info I want:

push @{$conIDs{conid}}, {$descid};
[download]

I'll find another problem that is over my head...