And you don't mention the OS or how your files are laid out, but be aware that on many *NIX variants some filesystems have problems with large (FSVO large that varies by filesystem) numbers of files in a single directory. For (say) an ext2 fs 10,000 would count and you'd notice a good bit of latency for any operation on the containing directory.

A common workaround is to break the files up into subdirectories keyed by part of the filename or something derived from the filename (e.g. if the filenames were 5 digit numbers have 10 top level directories 0..9 each of which has its own subdirectories 0..9 and the file 12345.foo would be located in 1/2/12345.dat).