A federal appeals court restored Utah's ban on polygamy Monday, handing a defeat to the family from TV's "Sister Wives" and other polygamists who say the ruling could send plural families back into hiding out of fear of prosecution.

Others cheered the decision that they say will help authorities prosecute people for crimes tied to the practice, such as underage marriage and sexual assault.

The 10th U.S. Circuit Court of Appeals dismissed a 2013 ruling that struck down key parts of Utah's law against bigamy, or holding multiple marriage licenses. U.S. District Judge Clark Waddoups had found the state violated polygamists' right to privacy and religious freedom.

The appeals court ruled that Waddoups should not have considered a lawsuit from Kody Brown and his four wives because they were never charged with a crime and there was little chance they would be.

The decision brings back a rule forbidding married people from living with a second purported "spouse," making Utah's law stricter than those in every other state and creating a threat of arrest for plural families. Like most polygamous families in Utah, Brown is legally married to one wife and "spiritually married" to the others.

State authorities have long said they don't go after polygamists who otherwise follow the law, and they reiterated that stance after the ruling. Officials wanted the clause kept on the books to help in criminal cases related to polygamy.

"We want them to come out of any shadows to report crimes," Utah Attorney General Sean Reyes said.

Prosecutors pointed to Warren Jeffs, the imprisoned polygamous sect leader convicted of assaulting girls he considered wives. Authorities have recently cracked down on his sect on the Utah-Arizona border, charging leaders in a multimillion-dollar food stamp fraud scheme. A jury in Phoenix also found the towns denied nonbelievers basic services such as police protection.

The Browns have never belonged to Jeffs' group, and lumping in good, honest polygamist families with him isn't fair, said Brady Williams, who has five wives.

"We're only guilty of trying to love a different way than the norm," he said. "They are marginalizing a minority class in the United States. That's unconstitutional."

The Browns will appeal the ruling, either asking the 10th Circuit to reconsider or taking the case to the U.S. Supreme Court, said their lawyer, Jonathan Turley.

"The underlying rights of religious freedom and free speech are certainly too great to abandon," Turley said in a statement.

The family has said other laws exist to target crimes linked to plural marriages and that banning the practice can sow distrust of authority. They argue their show is evidence that polygamous marriages can be just as healthy and monogamous ones.

Kristyn Decker, a former polygamous wife, disagrees. She says that people don't report problems amid pressure to avoid casting the practice in a bad light.

"We have all been told over and over again, 'We need to protect the principle of plural marriage at any cost,'" said Decker, who argued that the ban lets authorities get a foot in the door.

The state said the appeals court made the right call.

"I don't think the Browns had a legitimate fear of prosecution," said Parker Douglas, who argued the state's case.

There are about 30,000 polygamists in Utah, according to court documents. They believe polygamy brings exaltation in heaven — a legacy of the early Mormon church. The mainstream Church of Jesus Christ of Latter-day Saints abandoned the practice in 1890 and strictly prohibits it today.