exists works on single elements of a hash (or array), not an entire hash (or array). (Monks, is is appropriate to say exists works in scalar context?) If you remove the % sigil and first set of curlies (leaving  if (exists $rHoH_ProteinFamilies->{$proteinfamily}->{$PROTEIN}) Your error should go away.

See perldoc -f exists