The best logic is to break the main folder into a series of subfolders based upon some logical criterion. It's usually done with some logical portion of the name. For example, if files have date-like names (2009-02-09.log), you could use the path 2009/02/2009-02-09.log. Without knowing more about your problem, I can't really say more.

Update: I was looking for Efficient processing of large directory, but apparently MidLifeXis's Super Search Foo is greater than mine. Definitely a good thread to read.