One downside I see is that it's not easy to read. You're more likely to read

open $fh, '<', $file or die $!; 
process($fh) if is_valid($fh);
[download]

than

die $! unless open $fh, '<', $file; 
is_valid($fh) and process($fh);
[download]

because the first version puts the important information first. In that idiom, the important part (the assignment) is torn away to both ends.

Also if you have something like this:

my $data;
{
  local $/;
  open my $fh, '<', $file or die $!;
  $data = <$fh>;
}
[download]

You can afford not to know what $/, $! and '<' mean, and still have a pretty good idea that a file is being opened, and the content written to $data (even if you don't understand exactly why it's written like that). In your idiom, you have one line with many harder concepts, and you can't really extract the easy part.

I'm tempted to say the best solution would be to use a function that abstracts all that away, but I honestly I never do it myself :P.

it's bad style to rely on the implicit close

Personally I even consider it good style, because if you actively rely on the implicit close, you are going to focus on having the correct scope for your handle. The alternative is either you close before the end of the scope, or rewriting the variable, in which case the close doesn't do anything more, or you close the handle and keep it, which is only fine if you actually meant, and needed it to do it.