I think the problem is that the -s file test returns undef when the file is empty (i.e., when it has a zero size). One way to fix this is to test for undef and change it to zero using the // (logical defined-or) operator (see perlop#Logical-Defined-Or):

#! perl
use strict;
use warnings;

opendir D1, ...;

print "$_\n" for map  { sprintf qq[%s: %d], $_, -s $_ // 0 }
                 sort { (-s $a // 0) <=> (-s $b // 0) }
                 grep { ! -d }
                 readdir D1;
[download]