You are calling -f on full file name, yet -s on just the base name, hence the discrepancy. I would just cache the full name once and for all (well, until next loop iteration). Something like this:

while(my $file = readdir DIR) {
    next if $file =~ /^./; # still working with name only
    $file = File::Spec->catfile( $dir, $file );
    next unless -f $file; # working with full path from now on
    # ....... proceed with the rest of loop
};
[download]