Checking 20000 files in 3 minutes means you do more than 100 stat checks per second (actually you do double that number because you stat twice for each file) many even on a server link. I think that is pretty fast.

What kind of performance were you expecting?

Update:

I ran a little test myself.

use Modern::Perl;
use Time::HiRes qw/time/;

open my $LIST, '<', 'M:/YP/list.txt' or die $!;
my @stats;

my $start = time;
while (<$LIST>) {
    chomp;
    push @stats,  stat "M:/YP/$_";
}
my $end = time;

say 'stat checks took ', $end - $start, ' seconds.';
say 'That is ', 540 / ($end - $start), ' checks per second.';
[download]

I am checking 540 existing files myself over my local network (the "M" drive is on a NAS) and got about 54 stat checks per second. So your performance is actually quite good!