Another approach:

> I basically want to replace the string &,&#,&#3,&#38 to blank, but it should not replace &

This meets your "requirements" and is IMHO easier to understand and more intuitive than tybald89's solution

  DB<19> p $test
&,&#,&#3,&#38,&#38;&,&#,&#3,&#38,&#38;
  DB<20> p $test =~ s/(&#\d+;)|&#?\d*/$1/gr
,,,,&#38;,,,,&#38;
[download]

The trick is to first match correct entities and leave them unchanged by replacing them with themselves.

Incorrect entities are then found by backtracking and replaced with an empty $1.

Please handle this with care, I'm not sure if your requirements didn't miss edge cases.

NB: yes, it will also replace &38 without #

  DB<22> p ',&38,' =~ s/(&#\d+;)|&#?\d*/$1/gr
,,
[download]

otherwise you can add other or-conditions to exclude this case.

update

replaced s/(&#?\d*;)|... with s/(&#\d+;)|