$! will not have any useful content. $! is only good for system functions. Also, delete returns the value of the element deleted, and since the string "0.0" is true, you will get your error message. If you know all of your values are numeric then just use '== 0'. E.g.
for ( keys %hash ) {
delete $hash{$_} if $hash{$_} == 0;
}