Looking at Set::Scalar, I don't think it's going to help here, at least not by itself. What it does is give you a way to hold a set of values without associations. In hash terms, it only holds keys, not values.

For the kind of problem you're describing, I think a tied hash could work well. See perltie for how to make one. You'll probably want to start with Tie::Hash or Tie::StdHash as a base class. You'll override FETCH at the very least.