See perlvar, it's pretty clear.

$!  aka $ERRNO      
            If used in a numeric context, yields 
            the current value of errno, with all
            the usual caveats.  
$?  aka $CHILD_ERROR
             The status returned by the last pipe 
             close,  backtick(``) command, or 
             system() operator. 
$@  aka $EVAL_ERROR
             The Perl syntax error message from the 
             last eval() command
[download]

So unless the error is in the context of starting a new process, or an eval(), it's $! you want.

One possible exception: Modules may use some variable or method of their own to expose errors. See the docs for the module.