Using johngg's HashofArrays wouldn't it suffice to have a number for each set representing the number of booleans referenced by it. Just decrement this number for every set that contains the just-switched boolean (i.e. every set in the array corresponding to the boolean). Any set for which the count turns to 0 has to be removed.

Essentially this does the same job as the garbage collection, but since you do the "garbae collection" yourself you know which sets are done.