Yep, this is the big issue. Majorities rule by force. If the minorities were able to change things by fighting back, they would have already done so. But as it currently stands, one side has a much larger capacity for violence, and it is not the minority.