Texas pastor Robert Jeffress made an "unfortunate" comment in a "sidebar conversation" at the Values Voter Summit, Family Research Council President Tony Perkins said Sunday, referring to Jeffress calling Mormonism a "cult."

Jeffress made the remark to reporters Friday after speaking at the summit where he was invited to introduce Texas Gov. Rick Perry.Since then, Jeffress has stood by that comment, only clarifying to Fox News that he was talking about a "theological" cult and not a "sociological" cult.

Perkins, who is head of the organization that hosts the annual summit, now in its sixth year, said he does not agree Mormons are a "cult," but he does believe "there are clearly theological differences between Evangelicals and Mormons."

Jeffress is not Perry's pastor, but Perkins said Perry's staff signed off on the introduction by Jeffress. However, nobody was aware Jeffress was going to incite such a heated discussion.

Perkins said Jeffress is the pastor of the First Baptist Church of Dallas, one of the largest churches in Texas, and is "very well respected."

But, Perkins said he wants it clear that the purpose of the summit is to unite faith-based voters, not discuss theology.

"We're not building a church, we're building a coalition to move the country back to the right road," he said.