I would just second the comment that you should make sure you have done some profiling to really prove where you are spending most of your time. It's easy to point your finger at the wrong thing when optimizing, and in many cases the more you know, the more likely you are to focus on the wrong thing.