The source of much rancor seems to be the conflation of two meanings of the word marriage: an ancient religious meaning (where "gay marriage" directly contradicts many long-standing doctrines and deeply held beliefs) and the more recent meaning of a civil union recognized by the state.
The problems come when it does affect you. The controversy over a baker's refusal to bake a special cake for a gay wedding (because doing so would conflict with that baker's own religious beliefs — note that the baker was willing to make a cake but not to decorate it in a gay wedding theme) is an example.
The correct answer to the whole mess, of course, is to get the state out of the marriage business: the state can grant civil unions (all existing marriages would be recognized retroactively as civil unions) and "marriage" is rightly between you and your religious beliefs or lack thereof.