In your case, delete $ENV{_} is the same as delete $ENV{"_"}. But that doesn't mean file handles are illegal to appear inside hash brackets. Any expression may appear there. Perl will just evaluate that expression (in scalar context (list context when slicing)), and, if necessary, cast the result to a string.

print "$ENV{FOO}\n"; # FOO stil in environment!!!

Considering that the bareword FOO can also be a filehandle, you really do not want perl to warn if a (potential) filehandle is used as a hash index. How's perl to know that when you write FOO, you really meant FOO, but when you write _, you might have typoed $_?