Another way to look at this is to consider the question

What is a timely (i.e., non-premature) optimization?

There are several good answers to this. Some answers center around comparative opportunity costs (e.g., is the value per expended time for implementing this optimization greater than I would get from doing something else?). But all answers that I know of relate in some way to value, and that's were we go awry.

In terms of the criteria that Ovid lays out

1. An optimization can have negative value if it causes us to decrease scope (e.g., by burning up our time budget before we finish required features).
2. An optimization can have negative value if it increases project cost in excess of the value of the optimization. (If the customer isn't willing to pay extra for an optimization, it probably isn't worth doing.)
3. An optimization can have negative value if it decreases product quality (e.g., by making the optimized area harder to maintain).
4. An optimization can have a negative impact on development speed.

All of which speak to the need to be judicious when choosing where and when to optimize.