But wouldn't it still need to go through the directory on every request from the web? I still think an RDBMS would be better. You'd only loop through the files when you create the table, and the memory requirements would be minimal, beyond the mySQL daemon running.

After you had your table with filenames, you would then only select the few filenames you needed to create each index page. The list of files is already prepared and stored in the table, there's minimal extra stuff involved to give a user a page.