For me, that point is when I've grown tired of investing my time in band-aid code. Hopefully this comes early in the design process, if not it might be more fruitful to slowly, gracefully, "steer" the project in the new direction.

In the case of Hatchet, I learned soon enough that the advanced sorting/search/statistics features that users were begging for would be damn near impossible using the existing design. Fortunately, the project was immature enough that it wasn't too harsh an effort to ask folks to move from the old design to the new SQLite-based engine. Yes, it required that they basically un/re-install the whole shmere, but it also gave them a better upgrade path with exciting new features (that I could incorporate without fear of burnout).

In short, there is no easy answer, but you'll probably know when the time is right. Just don't be afraid to jump because you might piss off users. Tomorrow might be a day too late. ;-)