The converse is equally important: you may escape any non-alphanumeric and be confident that it will literally match.

{ and } are a strange case: as long as you don't have a well formed {min[,[max]]} quantifier, you don't need to escape them. So if you always escape either { or } you are ok. People sometimes get in trouble by saying {,max} which is a literal match, not a quantifier.

Update: _ seems to be a grey area. perlre seems to guarantee that \_ won't ever become a metacharacter, but quotemeta doesn't quote it.