Does it matter if you have 3 million files in the directory? Have you actually tested to see if it'll be a problem?

With three million files, ls will be slow, but on a reasonable filesystem I would expect that file access speed would be immeasurably different from in a directory with only a thousand files.

As an example, I have a directory here which has tappity-tap 110,000 files. It takes a few seconds to ls. But statting a particular file is as near as damnit instantaneous.