ls sorts the list of files by default. If you don't care about the order of the files you can shave off a bit of time. ls -f will turn off sorting.
The difference you see will depend on your filesystem and you will have to try it out to see how much faster it is. In my experience there is a small difference with ReiserFS but there is a noticeable difference with Solaris UFS.
Comment on Re: greater efficiency required (ls, glob, or readdir?)
The man page for ls on my linux system states that ls -f is equivalent to enabling -a (among other things). This would change the behaviour regarding plain ls so I would recommend using ls -U instead.