There may be an issue with your file system. I just created a directory with 20,000 subdirectories. Then I used the following code to get the name of all directories. It took no (noticeable) time to run:
use warnings;
use strict;
use autodie;
my $dir = '/tmp';
opendir my ($dh), $dir;
my @dirs = grep { -d "$dir/$_" } readdir $dh;
print scalar(@dirs), "\n";