Hi Melly,

Just to touch on one point in your OP:

From the docs for utf8:

"The use utf8 pragma tells the Perl parser to allow UTF-8 in the program text in the current lexical scope (...)
Do not use this pragma for anything else than telling Perl that your script is written in UTF-8."

(Emphasis in original)

So I'd only need utf8 if I was specifying my search expressions in utf-8 rather than using the octal notation? (or, for some reason, wanted to use utf in a variable name).

Basically, I read your advice as "you can take it out - it's not needed for reading from a utf8 file" - is that right?

map{$a=1-$_/10;map{$d=$a;$e=$b=$_/20-2;map{($d,$e)=(2*$d*$e+$a,$e**2
-$d**2+$b);$c=$d**2+$e**2>4?$d=8:_}1..50;print$c}0..59;print$/}0..20
[download]

Tom Melly, pm (at) cursingmaggot (stop) co (stop) uk

That's right. You only need (and should only use) use utf8; if the source code of your script is in UTF-8, e.g. variable names, text within the script file, etc.

For reading in a file that is UTF-8, use an IO layer. See binmode, the docs for which say:

To mark FILEHANDLE as UTF-8, use :utf8 or :encoding(UTF-8) . :utf8 jus
+t marks the data as UTF-8 without further checking, while :encoding(U
+TF-8) checks the data for actually being valid UTF-8. More details ca
+n be found in PerlIO::encoding.
[download]

The way forward always starts with a minimal test.

Can anyone see a good way to load the RegExes from an eternal file?

There is probably a module for this somewhere, but if not, something along the lines of the following may serve. Your search patterns all seem to be plain octal sequences of various lengths and the replacement strings character string literals, so that simplifies things. I assume from your OP that you're confident about reading raw string data from a file and massaging it appropriately, so:

c:\@Work\Perl>perl -wMstrict -le
"use Data::Dump qw(dd);
 ;;
 my @filelines = (
   '\102\105\105 B',
   '\104\125\102\131\101 W',
   '\130 EKS',
   '\132 ZEE',
   );
 ;;
 my %search_terms = map split, @filelines;
 ;;
 my %xlate =
   map { eval(qq{ qq{$_} }) => $search_terms{$_} }
   keys %search_terms
   ;
 dd \%xlate;
 ;;
 my ($search) =
   map qr{ $_ }xms,
   join ' | ',
   reverse sort
   keys %search_terms
   ;
 print $search;
 ;;
 my $s = 'the BEEroDUBYAn foX jumped the laZy dog';
 print qq{'$s'};
 ;;
 $s =~ s{ ($search) }{$xlate{ $1 }}xmsg;
 print qq{'$s'};
"
{ BEE => "B", DUBYA => "W", X => "EKS", Z => "ZEE" }

(?^msx: \132 | \130 | \104\125\102\131\101 | \102\105\105 )
'the BEEroDUBYAn foX jumped the laZy dog'
'the BroWn foEKS jumped the laZEEy dog'
[download]

Some other more or less random thoughts:

•     $line =~ s/\302\240/\s/g;        # nbsp to sp
  If this is supposed to translate something to a space character, it doesn't:  \s has no special meaning in a double-quoted string:

  c:\@Work\Perl>perl -wMstrict -le
  "my $s = '>nbsp< hi nbsp there';
   print qq{'$s'};
   ;;
   $s =~ s{ nbsp }{\s}xmsg;
   print qq{'$s'};
  "
  Unrecognized escape \s passed through at -e line 1.
  '>nbsp< hi nbsp there'
  '>s< hi s there'
  [download]

  Perl could have warned you about this had you had warnings enabled.
•     $line =~ s/\342\211\244/\>\=/g;  # utf >= to ascii >=
  This is not critical, but in general, escaped characters like  \>\= in interpolated strings have no special meaning and should, IMHO, always produce warnings.
•     $line =~ s/\302\256/(a)/g;       # @ to (a)
  Just out of curiosity, why not translate to ASCII '@'?

• if($line =~ /[^[:ascii:]]/){
    $line =~ s/([^[:ascii:]])/ ... lotsa subexpressions ... /ge;
    print "Unhandled sequence: $line\n";
    }
  [download]

  This is only cosmetic, but this could be much simpler and IMHO clearer in consequence. The  s/// operator does not do substitutions if the pattern is not matched, and can be used as | to control a conditional because it returns the number of matches/substitutions in scalar/boolean context. One can also capitalize on obscure sprintf format specifiers:

  c:\@Work\Perl>perl -wMstrict -e
  "my $s = 'aXbYcZd';
   print qq{'$s' \n};
   ;;
   print qq{unhandled sequence: '$s' \n}
     if $s =~ s{ ([XYZ]) }{ sprintf '[%d/%1$#X/%1$#o]', ord $1 }xmsge;
  "
  'aXbYcZd'
  unhandled sequence: 'a[88/0X58/0130]b[89/0X59/0131]c[90/0X5A/0132]d'
  [download]

Update:
    $line =~ s/\302\240/\s/g;        # nbsp to sp
So, I guess this raises the question of just how you get a literal space from a space-delimited file to use as a replacement string, anyway? One way would be to use a non-space delimiter, but this may just shift the problem to another boundary. Another approach would be to specify all such characters in their octal equivalent,  \040 and so on. This sets us upon the the long and winding road of  s///ee and template processing (pursue the links from Re: Evaluating $1 construct in literal replacement expression). In any event, here's one way:

c:\@Work\Perl>perl -wMstrict -le
"use Data::Dump qw(dd);
 ;;
 my @filelines = (
   '\102\105\105 B',
   'nbsp \040',
   '\130 EKS',
   '\132 ZEE',
   );
 ;;
 my %search_terms = map split, @filelines;
 ;;
 my %xlate =
   map { eval(qq{ qq{$_} }) => $search_terms{$_} }
   keys %search_terms
   ;
 dd \%xlate;
 ;;
 my ($search) =
   map qr{ $_ }xms,
   join ' | ',
   reverse sort
   keys %search_terms
   ;
 print $search;
 ;;
 my $s = 'the BEErownbspfoX >nbsp< the laZy dog';
 print qq{'$s'};
 ;;
 $s =~ s{ ($search) }{ qq{ qq{$xlate{$1}} } }xmsgee;
 print qq{'$s'};
"
{ BEE => "B", nbsp => "\\040", X => "EKS", Z => "ZEE" }

(?^msx: nbsp | \132 | \130 | \102\105\105 )
'the BEErownbspfoX >nbsp< the laZy dog'
'the Brow foEKS > < the laZEEy dog'
[download]

Many thanks for that - most useful.

In answer to some specific questions:

Silly me regarding \s - these are nbsp characters embedded within a quoted string, so replacing with " " works fine.

Re @ - I'm an idiot. I did a search for the hex sequence, saw it was a @, thought to myself "oh - I thought @ was ascii, oh well", and did the replacement (why does utf bother to encode a character that exists in ascii?).

UPDATE - utf does have an alternative for @, but I've just realised that that particular sequence is a registration mark - in this case, a doctor is using it as a short-hand for 'right' in a word doc, and then pasting into a db.

And, yes, my escaping habits are terrible and I am suitably ashamed.

map{$a=1-$_/10;map{$d=$a;$e=$b=$_/20-2;map{($d,$e)=(2*$d*$e+$a,$e**2
-$d**2+$b);$c=$d**2+$e**2>4?$d=8:_}1..50;print$c}0..59;print$/}0..20
[download]

Tom Melly, pm (at) cursingmaggot (stop) co (stop) uk

So, I guess this raises the question of just how you get a literal space from a space-delimited file to use as a replacement string, anyway? One way would be to use a non-space delimiter, but this may just shift the problem to another boundary.

That is a well known problem and it has been solved over and over again. Probably the easiest way to do it is by using a "CSV"-style file and Text::CSV knows how to do this perfectly.

CountZero

A program should be light and agile, its subroutines connected like a string of pearls. The spirit and intent of the program should be retained throughout. There should be neither too little or too much, neither needless loops nor useless variables, neither lack of structure nor overwhelming rigidity." - The Tao of Programming, 4.1 - Geoffrey James

My blog: Imperial Deltronics