in reply to Re: Quickly reading the first line of a large number of files...
in thread Quickly reading the first line of a large number of files...

If all of the files are in the same directory with many others, then that may be one of your performance problems. Many file operations have to scan the directory listing, and as the listing gets long, this gets inefficient.

If you can scatter your files across many different directories in some way, you will probably get a performance increase. I believe that the Reiser filesystem is also specifically designed to be fast in this case, and others probably are as well, so you may be able to improve performance at the OS level. (Then again, you may not, and that kind of performance tweak is likely to be forgotten when the next admin looks at the system.)

  • Comment on Re (tilly) 2: Quickly reading the first line of a large number of files...