If you want objects which aren't referenced to go away, you could use store a 'weak' reference to each of your objects in your array, and the weak reference will become undef if there are no 'real' references left to the object. That might do what you need. (See Scalar::Util for weaken).

Otherwise, it seems an unusual route to go down and that makes me think it might not be the right path. Could you let us know more about your motivation for this, so we can offer an opinion on this?