You could turn this:

        if( grep( $_, @{ $sets[ $set ] } ) == @{ $sets[ $set ] } ) {

into a use of all from List::MoreUtils:

        if( all { $_ } @{ $sets[ $set ] } ) {
[download]

(Note that the overhead of a function in this module is larger than the overhead of a plain grep.)

But I think may get a better result by inverting the boolean, and thus setting the boolean for each item before you start, and clearing its value. Testing if an array contains no true value at all, might be faster than testing if enough of them are true. Anyway, I don't expect you can even double the speed this way...

Also, you might reconsider using vec and storing an array of booleans into a single string (of bits). That way testing every item in a set is simply a matter of a single test on a string, instead of testing each item in a whole array.