Colo[u]r being those frequencies of light visible to humans...

If you define colour in terms of what the human eye sees, there are only three of them, four if you count luminosity that's not in any of the three color categories, or throw in tetrochromacy; five if you count both. So yellow, for instance, is not a color if you define it this way. It's a combination of colours (specifically, red and green), or else it's a pigment that absorbs a certain color (blue).

And yeah, the output of the sun is pretty well continuous across the electromagnetic spectrum, so if you define colors as wavelengths of electromagnetic radiation then the cardinality of the set of all colours in the rainbow is aleph-sub-one, the same as the cardinality of the set of all real numbers. I suppose that means the number of colors in the rainbow is actually greater than infinity, if you define infinity in the usual gradeschool way (which comes out in math as aleph-sub-naught, the cardinality of the set of natural counting numbers).