I was actually analyzing the win/loss of the branch.

For large arrays, the overall running time is going to be O(n). No question. But will trying to skip useless swaps overall speed you up or slow you down? Well on large arrays you speed up on log(n)+O(1) times, and slow down all of the other iterations, so it is a loss.

But doesn't change the overall big-O of the algorithm, just the constant.