Ouch, you're right. Doing -f wouldn't help much, either. What about

my @all = glob '* */';
@dirs = grep m(/$), @all;
# etc.
[download]