Thinking about it some more made me realize that ref $x eq 'REF' can be used more efficiently then ref $x eq 'SCALAR'. (Although not really on the previous code, since the goal was to keep one reference at the end)

If you have ref $x eq 'REF' then you know that the scalar to which $x refers to is yet another reference, so noone stops you from doing $x = $$$x;, which avoids yet another call to the ref function...