do you have a %SIG handler activated, like with CGI::Carp ?

Update

Oops, I just noticed you mentioned CGI::Carp explicitly, so my guess was not far fetched. And it's using SIG-handlers to intercept those errors.

Problem is, that SIG handlers need to to take care about being called inside an eval. They need to check $^S and ignore those exceptions.

The $SIG{__DIE__} hook is called even inside an eval(). It was never intended to happen this way, but an implementation glitch made this possible. This used to be deprecated, as it allowed strange action at a distance like rewriting a pending exception in $@. Plans to rectify this have been scrapped, as users found that rewriting a pending exception is actually a useful feature, and not a bug.

Update

Well the solution is explicitely documented in

CGI::Carp#SUPPRESSING-PERL-ERRORS-APPEARING-IN-THE-BROWSER-WINDOW

A problem sometimes encountered when using fatalsToBrowser is when a die() is done inside an eval body or expression. Even though the fatalsToBrower support takes precautions to avoid this, you still may get the error message printed to STDOUT. This may have some undesirable effects when the purpose of doing the eval is to determine which of several algorithms is to be used.

By setting $CGI::Carp::TO_BROWSER to 0 you can suppress printing the die messages but without all of the complexity of using set_die_handler. You can localize this effect to inside eval bodies if this is desirable: For example:

eval {
  local $CGI::Carp::TO_BROWSER = 0;
  die "Fatal error messages not sent browser"
}
# $@ will contain error message
[download]

Please check, if this solves your problem!