It just irks me to see that awkward code there..

local $_;
for(readdir DIR) {
    $filesize{$_} = -s if /\.log$/i;
}
[download]

But that buys you nothing.

I can't imagine any reason for the stat to be so slow; that performance sounds more like it's open()ing the files than stat()ing them. (Imagine a 10 minute break here as I'm posting.) Running out of straws to grasp.. have you tried explicitly stat()ing the files and picking up the size value rather? As in:

local $_;
for(readdir DIR) {
    $filesize{$_} = (stat($_))[7] if /\.log$/i;
}
[download]

Chances are it 99.9% likely won't make a difference.. but..