I've found ls -f ( -f disables sorting and highlighting) to be much faster then the default ls options under some conditions.
Some linux distributions put 'alias ls="ls --some --options"' in /etc/profile (or equivalent), which can cause ls to stat() each file to determine the correct highlighting options before displaying anything.