An approach that has served me well (perfectly so far) is to decode everything to utf8 first and then re-encode it to entities. It works because decoding a plain & is a no-op. So you're essentially normalizing the text and then encoding it. You will lose your original entities but the new ones will probably be better as they will be uniform. I recommend numeric entities; the example shows named ones.

use strict;
use warnings;

use HTML::Entities;
use Encode;

for my $line ( <DATA> )
{
    # This is a no-op on plain &s
    my $utf8 = HTML::Entities::decode($line);
    print Encode::encode_utf8($utf8);
    my $proper = HTML::Entities::encode($utf8); # OR encode_numeric()
    print $proper;
}

__DATA__
Purus Accumsan Felis &#8240; Maecenas Nibh &theta; Eget Phasellus & Mi
+ Amet.  Odio Amet && Purus.  Mi Ullamcorper Lorem Eget Nibh.

http://www.example.com/?name=John&residence=Vatican+City&job=Pope
[download]

I may be missing something, but it looks like you're printing and then discarding the utf8-encoded text, then continuing on with the non-utf8 text. Shouldn't it be something like this?

    my $utf8 = HTML::Entities::decode($line);
    $utf8 = Encode::encode_utf8($utf8);
    my $proper = HTML::Entities::encode($utf8);
    print $proper;
[download]

Update: Ah, nevermind, I misunderstood what you were saying initially.

__________
Systems development is like banging your head against a wall...
It's usually very painful, but if you're persistent, you'll get through it.

Uh... no. Did you run it? The print Encode::encode_utf8($utf8); is just there to see the intermediary step. Encode::encode_utf8 makes the output "safe" for the terminal: no "wide character" warnings.

s/
   &
   (?!
      (?:
         # (?: x[\da-f]+ | \d+ )
      |
         [a-z]+
      )
      ;
   )
/&/xi
[download]

I factored out the "#" and removed extraneous captures and groupings, but the key is (?!)

Update: And if you wanted to only accept known entities,

local our %known = map { $_ => 1 } qw(
   eacute
   Eacute
   ecirc
   Ecirc
   ...
);

s/
   &
   (?!
      (?:
         \# (?: x[\da-f]+ | \d+ )
      |
         ([a-z]+) (?(?{ !$known{$1} }) (?!) )
      )
      ;
   )
/&/xi
[download]

or

use Regexp::List qw( );

my @known = qw(
   eacute
   Eacute
   ecirc
   Ecirc
   ...
);

my $known = Regexp::List->new()->list2re(@known);

s/
   &
   (?!
      (?:
         \# (?: x[\da-f]+ | \d+ )
      |
         $known
      )
      ;
   )
/&/xi
[download]

Update: Escaped "#" as per reply.

Shouldn't that be \# or [#] in an extended (ie., /x) regex, otherwise a naked # begins a comment-to-end-of-line?

To be pedantic, don't use \d as a substitute for [0-9]. \d matches a lot more than just western digits, it matches digits in many other scripts as well, but those aren't legal in numeric HTML entities.

Furthermore, the ';' is optional if a named entity is used, and isn't followed by other letters. So, I'd use:

  /&(?![a-zA-Z]++(?:;|\b)|x[0-9a-fA-F]++;|[0-9]++;))/
[download]

Good point about "\d". As I stated, I assumed the original pattern was correct. I did so because I didn't look up what an valid entity could be.

As for the optional ";", the rules are hidden in the SGML spec. Perhaps it would make sense to add the ";" if it's missing (using s/(&[a-zA-Z]++)(?!;)/$1;/g;).

Going by your description of what is valid, using \b is incorrect. \w matches more than letters, even without unicode semantics. That's easily fixed by simplifying "(?![a-zA-Z]++(?:;|\b))" to "(?![a-zA-Z])".

Also, "#" is missing in your pattern, and you have an extra ")".

Fix:

s/&(?!\#(?>x[0-9a-fA-F]+|[0-9]+);|[a-zA-Z])/&/g;
[download]

By the way, I used (?>) instead of the possessive quantifier since the former dates back to at least 5.6, whereas the latter was introduced in 5.10.

I do recommend that you not only check that you have something that looks like a named entity, but that you check that the name you found is actually an actual entity name.

Here is the list of entity names that I currently use, I don't claim to be sure that it is perfect and complete, but it'll be pretty close.

my @names = qw(
  amp lt gt apos quot nbsp iexcl cent pound curren yen brvbar sect uml
  copy ordf laquo not shy reg macr deg plusmn sup2 sup3 acute micro
  para middot cedil sup1 ordm raquo frac14 frac12 frac34 iquest Agrave
  Aacute Acirc Atilde Auml Aring AElig Ccedil Egrave Eacute Ecirc Euml
  Igrave Iacute Icirc Iuml ETH Ntilde Ograve Oacute Ocirc Otilde Ouml
  times Oslash Ugrave Uacute Ucirc Uuml Yacute THORN szlig agrave
  aacute acirc atilde auml aring aelig ccedil egrave eacute ecirc euml
  igrave iacute icirc iuml eth ntilde ograve oacute ocirc otilde ouml
  divide oslash ugrave uacute ucirc uuml yacute thorn yuml OElig oelig
  Scaron scaron Yuml fnof circ tilde ensp emsp thinsp zwnj zwj lrm rlm
  ndash mdash lsquo rsquo sbquo ldquo rdquo bdquo dagger Dagger permil
  lsaquo rsaquo euro Alpha Beta Gamma Delta Epsilon Zeta Eta Theta
  Iota Kappa Lambda Mu Nu Xi Omicron Pi Rho Sigma Tau Upsilon Phi Chi
  Psi Omega alpha beta gamma delta epsilon zeta eta theta iota kappa
  lambda mu nu xi omicron pi rho sigmaf sigma tau upsilon phi chi psi
  omega thetasym upsih piv bull hellip prime prime oline frasl weierp
  image real trade alefsym larr uarr rarr darr harr crarr larr uarr
  rarr darr harr forall part exist empty nabla isin notin ni prod sum
  minus lowast radic prop infin ang and or cap cup int there4 sim cong
  asymp ne equiv le ge sub sup nsub sube supe oplus otimes perp sdot
  lceil rceil lfloor rfloor lang rang loz
  spades clubs hearts diams
);
[download]

Also note that I included "'", which is a legal XML entity, but actually not a legal HTML entity. So, if you encounter it, you ought to replace it by its numerical entity: "'".

There are various ways to detect if a name is in a set:

• You can use them as hash keys for quick detection

  my %isname;
  $isname{$_} = 1 foreach @names;
  if($isname{$name}) ...
  [download]

• In perl 5.10 or later, you can use the smart match operator to see if the name is in an array

  if($name ~~ @names) ...
  [download]

• You can use grep to detect if a name is in an array

  if(grep $name eq $_, @names) ...
  [download]

• With Regex::PreSuf or a similar module, you can build a fast regular expression:

  use Regex::PreSuf;
  my $re = presuf(@names);
  if($name =~ /^($re)$/o) ...
  [download]

  In modern perls, a simple dumb regexp of alternatives separated by "|" is automatically optimized so it'll be quite fast, too

  my $re = join '|', @names;
  if($name =~ /^($re)$/o) ...
  [download]

  A regexp like this can easily be incorporated into the real s/// statement, instead of the generic name matcher, so it will only match real entity names. (Don't forget a /\b/ anchor at the end, to avoid matching partial names.)
• You can use a large string of names separated with spaces, plus a space at a front and at the end; and use index to search if a name surrounded by spaces, is in the string

  my $names = ' ' . join(' ', @names) . ' ';
  if(index($names, " $name ") >= 0) ...
  [download]

Here is a proposal of some code, using a hash to detect if a name is a proper name. If a proper entity is matched, a missing semicolon at the end will be added.

s/&((?:(#\d+|0x[\da-f]+)|([a-z0-9]+));?)?/
  $2 ? "&$2;"  # numerical
     : $3 && $isname{$3} ? ($3 eq 'apos' ? "'" : "&$3;")  # named
                         : ($1 ? "&$1" : "&")  # not an entity
/gie;
[download]