Depending on the underlying filesystem there may not be anything you can do to speed it up. For example on Linux ext2 is going to start to chug on directories over around a thousand entries (at least rule of thumb you'll start to feel the performance hit). If this is the case you'd be better served reorganizing contents if you can so that the directory structure itself does the filtering itself by virtue of how you've organized things.