There's a long-standing rule in Unix that one should never check the value of errno except immediately following a failed system call -- otherwise there are no guarantees about its value. Since $! is a magic alias for errno, the same rules apply. That's basically what perlvar says about it, too:

If used in a numeric context, yields the current value of errno, with all the usual caveats. (This means that you shouldn't depend on the value of $! to be anything in particular unless you've gotten a specific error return indicating a system error.)