Updated

A federal appeals court on Monday dismissed a landmark decision that decriminalized polygamy in Utah, marking a legal defeat for the family from the reality TV show "Sister Wives."

Kody Brown and his four wives cannot sue the state over its ban on plural marriages because the family never faced charges and local prosecutors later said they would not prosecute consenting adults with multiple wives, according to the 10th U.S. Circuit Court of Appeals in Denver.

The panel's decision reverses a 2013 ruling that removed the threat of arrest for polygamous families. U.S. District Judge Clark Waddoups had found that the possibility of prosecution drove the Browns out of the state and that key parts of Utah's bigamy law violated their right to privacy and religious freedom.

State prosecutors have a longstanding policy against charging consenting adult polygamists, but attorneys argued in their appeal that the ban should stay on the books to help authorities go after those who commit crimes related to the practice, such as sexual assault, statutory rape and exploitation of government benefits.

Prosecutors pointed to imprisoned polygamous sect leader Warren Jeffs, who was convicted of assaulting underage girls he considered wives.

The Browns will discuss whether to ask the 10th Circuit to reconsider or appeal to the U.S. Supreme Court, said their lawyer, Jonathan Turley.

"The underlying rights of religious freedom and free speech are certainly too great to abandon after prevailing below in this case," Turley said in a statement.

The family has said other laws exist to target crimes linked to plural marriages and that banning the practice can sow distrust of authority. They say their TV show is evidence that polygamous unions can be as healthy as monogamous marriages.

Bigamy, or holding multiple marriage licenses, remained illegal amid the court fight. Brown has a license for only one of his marriages and says his other unions are spiritual.

The 10th Circuit did not weigh the constitutional questions of the case because it found that the Browns did not have legal standing to sue.

The state said the appeals court made the right call.

"It is gratifying," said Parker Douglas, with the attorney general's office. "In this case, I don't think the Browns had a legitimate fear of prosecution."

There are about 30,000 polygamists in Utah, according to court documents. They believe polygamy brings exaltation in heaven — a legacy of the early Mormon church. The mainstream Church of Jesus Christ of Latter-day Saints abandoned the practice in 1890 and strictly prohibits it today.