While the impacts of a no vote would, in essence, have a far greater impact on a larger group, to say that there would be no impact upon the no voters if they lose would be wrong.
This is evidenced in other countries where SSM is legal. Certain groups have taken it upon themselves to try and force religious organisations and individuals to go against what they believe and perform SSM, no matter the cost to those people.
Is it the same level of distress in numbers? No. Does it happen though? Yes.
One group will impose their viewpoint on the other regardless of any personal distress that might arise from that worldview. There is no way to get around that fact and people need to accept this and try and help all involved, regardless of the worldview that is held.