I'd suppose the need to refactoring the context of this dilemma is relative to how well the code satisfies given criteria and criteria do change over time. Knowing what to refactor, when and how much are also essential design criteria and should be constantly be reviewed over a programming project.

Shying away from refactoring of code could lead the project (and program) to become unwieldy and perhaps also restrict evolution in ideas and concepts (useful mistakes, interesting discoveries, etc) along the way -- Generally, it would be wise to strive in finding a balance between flexibility of code and readability, programmer and program effeciency, scalability, etc -- too much refactoring towards optimization might leave the code inflexible to changing criteria over a project's lifetime.

We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. - Knuth

If it ain't broke .. refactor it. - Microsoft Windows