Hello Chris Daniel, and welcome to the Monastery!

Please note that your requirement as stated is self-contradictory. I assume you don’t want to remove the entity &#38. In which case, you can use a negative lookahead assertion to ensure that the entity to be removed is not immediately followed by another digit:

20:54 >perl -wE "my $x = '<Remarks>1 SW PLT SLAC 6 PCS &#3</Remarks>';
+ $x =~ s/&#\d(?!\d)//g; say $x;"
<Remarks>1 SW PLT SLAC 6 PCS </Remarks>

20:54 >perl -wE "my $x = '<Remarks>1 SW PLT SLAC 6 PCS &#38</Remarks>'
+; $x =~ s/&#\d(?!\d)//g; say $x;"
<Remarks>1 SW PLT SLAC 6 PCS &#38</Remarks>

20:54 >
[download]

See “Lookaround Assertions” in perlre#Extended-Patterns.

Hope that helps,