My thoughts on this mostly echo Kyle's, but there is another point to bring up: significance.

Benchmarks with tiny differences often aren't even significant. That is to say, that 3 millionths of a second you gain in your example might not just be tiny, but not actually reliably exist. I'm searching now for, but cannot find, an old node here that showed that trivial and seemingly unrelated changes to the source bumped the results around by a few percent.

As far as the original point goes, CPU cycles almost never matter. But there do exist cases where swapping one algorithm for another can offer you a big speed difference in a place where it actually matters.

I think my own optimization decision making gets summed up pretty well by the following:

1. Is it slow?
2. Should I care that it is slow?
3. Is there an obvious fix? (I.e. am I accidentally iterating over a whole data set when I could drop out on the first success or failure, etc.)
4. Will new, faster hardware be in place by the time I finish this thing, anyway?
5. Can I buy the problem away by throwing hardware at it?
6. Okay, guess it is time to optimize.