Is there a limit to how many files I should have in a such a directory?

There certainly is a limit, but before you ever hit the limit you will hit a slowdown in filename lookup caused by too many levels of indirect blocks in the directory structure which holds the file names. But that's not the main point.

You definitely want to keep the limit much lower, just in case you want to get rid of that directory. See Re^5: greater efficiency required (ls, glob, or readdir?)