The Rev. Oliver White of Grace Community United Church of Christ in St. Paul, Minnesota, has lost two-thirds of his congregation. The mass exodus from his pews started back in 2005 when he voted in favor of gay marriage at the national United Church of Christ assembly meeting. Following his support for the measure, the majority of his 320-member flock was flabbergasted and left the church.

“I was really in shock. I visited one member after another, who told me they weren’t coming back because of it,” White, who has pastored at the house of worship for 22 years, said in an interview with The Star Tribune. “The membership evaporated right before my eyes. Their fundamental belief was it’s wrong for a man and a man to marry or a woman and a woman to marry. A lot of them said if I change, they’d come back. But I told them, it is what it is. I am who I am. And this is what I fight for.”

In an interview with CNN, he reiterated these comments, claiming that his church members simply couldn’t rectify their personal beliefs with the acceptance of same-sex marriage.