By that token maybe 4, because we also have separate receptors for brightness?

I don't think that you can classify intensity as a fourth "color".

If you increased the intensity of the spectural components in the rainbow whilst leaving their ratios unchanged, the observer would tend to squint, closing the iris, to reduce the overall amount light (number of photons), reaching the retina, with the result that the observer would perceive the same information.

Besides which, the rods only tend to come into play in low-intensity situations, when color information becomes distorted anyway. For example, at night, the green leaves of a rose tend to appear brighter than the red of the petals.