Premature optimization is the root of all evil.

First start with a clean and modularized design. Then identify bottlenecks. Then fix the identified bottlenecks. Up front coding designs that make it hard to later on identify and fix bottlenecks usually hurt performance in the end more than they help it. And they always hurt you on development time, maintainability, and bug counts.