If you really want to save memory, you may want to avoid pinning your inside-out array at the high-water mark. Store a stack of reclaimed indices and push to it in the destructor and shift from it when generating the reference for a new object. Then you can shrink the array if necessary.

(When you ask "Wait, won't it stay at the high-water mark even if there were 1000 elements and are only two now?", I'll sigh and talk about compacting garbage collectors and double-indirection the other flaws in this scheme. Still, it's a fun idea.)