I do have fear when I see a bad code that is ripe for refactoring, but I think the decision should be based on cost-benefit-analysis, not fear.

So when I see a code that needs refactoring, but refactoring would not add any significant benefit to the whole project, I would not do it.

On the other hand, when a code is so complex but refactoring would gain much benefit in terms of maintenance, less bugs, or performance, I would go for it, even when I'm afraid it would break it. It also very much depends on the cost of the breakage.

(But sometime you wouldn't know what benefit you will/will not gain until you finished doing it, that what makes the fear, I guess)