But it will (should, if it doesn't, it's a bug) destroy the array and all elements in it and reuse their memory in perl except for the one element that still has a refcount > 0.

So, a standard perl container (hash or array) will have a reference to its elements, but not the other way around. This is also true for regular references; they only point 1 way and only up the refcount for the variable that they point to.