$! is a magic variable that reflects the error of the latest error that occurred while doing a syscall (errno()).

Most likely, croak() does some syscall that succeeds, while die does not.

So, if you want to keep the value of $! at some point in time, you need to make a copy of it. One way to do that is to use double quotes, like you showed.