I had to wait the moment in which this would have been really necessary!

I think that's the key point to take away. The whole idea of refactoring is to make your code simpler. If your code is getting more complicated to understand something is probably going wrong :-)

The time to refactor is when the code let's you know that refactoring is necessary (through duplication, hard to understand code, etc.)