The correct error variable to check after an eval is $@, not $! - the latter is actually empty in this particular case, but could potentially contain an error caused by a previous operation, leading to misleading results (try perl -Mstrict -we 'chdir("/foobarquz"); eval q(x=3) or die $!'). Also note perl -Mstrict -we 'eval q(x=3); die' has the same behavior as your second example, the reason is documented in die:

If LIST was empty or made an empty string, and $@ already contains an exception value (typically from a previous eval), then that value is reused after appending "\t...propagated".