This is one reason why I prefer glob to opendir:

my @dirs= grep { -d } glob( "$dir/*" );
[download]

works. It forces you to either chdir or prepend the directory name (and often makes much simpler code).

You also forgot to filter out "." and ".." in your version. Of course, my version won't report ".cpan" (which might be a bug or a feature, depending), and I wish glob provided an option to make that trivial to do.