I don't think it's impossible. I just think there are probably a lot of little issues to get it to work in the genreral case so that you don't even have to think about it, it just works. I've done similar things with PadWalker but they wouldn't work in all cases and the subs had functions that had to be explicitly called to restore state.
One point about using a hash for the references, no matter what you use they end up being strings from my understanding as they are automatically quoted.