If you define colour in terms of what the human eye sees, there are only three of them

I think you're misconstruing me! I am limiting colours to be 'things which humans can see' which doesn't correspond to 'the specific frequency ranges which stimulate only one cone type'. (And in fact the frequency ranges overlap so your definition becomes a bit more tricky than you make out)

I can see EM radiation with wavelength 570nm (yellow), so I would define it as a colour! I just happen to see it by stimulating more than one cone type at once.

so if you define colors as wavelengths of electromagnetic radiation then the cardinality of the set of all colours in the rainbow is aleph-sub-one

As tilly pointed out, you don't get mixtures of blue and red light in a rainbow, which we see as pink-purples (the line of purples on the CIE chromacity diagram.)

Update: In fact, you don't get any mixing in a rainbow (approximately, obv mist is not a perfect refractor and probably some other caveats like angle and distance), so it's like going around the edge of the chromacity diagram... so there's quite a few less than all possible colours/hues in a rainbow.. but still probably infinite (I'm not a mathematician either, so no fancy 'alehps' from me... :P). That's my current conjecture anyway!