Updated

The polygamous family from TV's "Sister Wives" will take their fight to legalize polygamy to the U.S. Supreme Court, their lawyer said on Friday.

Kody Brown and his four wives say that polygamous marriages can be just as healthy as monogamous unions, and they argue a Utah law that makes plural marriages a felony violates their right to freedom of speech and religion.

Attorney Jonathan Turley decried an appeals court decision that threw out the Browns' challenge to the law. The judges decided the Browns can't sue because they weren't charged under the law, so they didn't consider the constitutional issues.

Turley said an investigation by local authorities after the Brown's reality TV show "Sister Wives" premiered nevertheless amounted to government abuse. Though the investigation was closed without filing charges, the Brown say they had to flee the state and the threat of prosecution still hangs over their heads.

A federal judge sided with them and overturned key parts of the state's bigamy law in 2013, but an appeals court overturned that decision in April. The appeals court refused the Browns' request to reconsider the case in an opinion handed down Friday.

Utah Attorney General Sean Reyes has pledged to leave polygamists alone unless they break other laws. But he argues the state still needs the ban to go after polygamists like Warren Jeffs, who serving a life prison sentence for sexually assaulting girls he considered wives.

A spokesman for the Reyes declined to comment on the "Sister Wives" appeal. The family's lawyer didn't immediately say when the appeal might be filed.