Your solution looks ok in my eyes. What's the problem? Adding enums is just another if statement (Update: when you want to implement them as described and not as a regex like qr{\A(?:red|green)\z}xms). For more exotic types just override the mutator, like the email example in the documentation:

 # Only accept addresses which look valid.
 sub email {
     my($self) = shift;
     my($email) = @_;

     if( @_ ) {  # Setting
          require Email::Valid;
          unless( Email::Valid->address($email) ) {
          carp("$email doesn’t look like a valid address.");
          return;
      }
   }

   return $self->SUPER::email(@_);
}
[download]

I think if you are happy with Class::Accessor, use it. I don't think you would save much lines of code with any other module.