If the debugger is accurate, you are stepping through the array three times to produce your results. If you have a directory with a large number of files in it, this is not the most runtime-efficient way of doing it.

This version makes a single pass through the directory listing.

#Assume all the open code
foreach( readdir DH ) {
  next if ( $_ eq '.' or $_ eq '..' );
  push @subdirs, "$dir/$_" if ( -d "$dir/$_" );
}
[download]

Of course, you may still not like this. It will be kinda hungry on memory, especially if the ratio of files to directories is very large - the foreach will suck in every file at once. It may be better to try it with a while loop - it will be easier on memory but will take longer due to buffering issues.
mikfire