Why bother re-inventing the wheel? Just use
Object::InsideOut. You get the speed of array-based objects, and all of the messy details you've discussed are already handled plus some you didn't discuss such as thread-safety.
Remember: There's always one more bug.