But all of our permissions could be represented as giant integers exactly 128 bits wide. The only operations they need to do are bitwise-AND and bitwise-OR, both of which are simple.

Did you ever consider using vec?