Same-sex marriage foes are flagging a mostly-missed moment from last week's Supreme Court arguments over the constitutionality of gay marriage, calling it a warning sign for religious freedom.

They point to an exchange between Justice Samuel Alito and the administration's top lawyer, Solicitor General Donald Verrilli. Alito asked if, in the event the Supreme Court holds that there is a constitutional right to same-sex marriage, would religious colleges risk losing their tax-exempt status if they continued to advocate for traditional marriage only.

Verrilli responded, "I don't deny that ... It is going to be an issue."

Travis Weber, director of the Family Research Council's Center for Religious Liberty, said he believed Verrilli's logic would eventually extend to churches and just about "anyone holding a traditional view of marriage."

Weber warned that if the current trend continues, those who refuse to endorse same-sex marriage could face "fines, potentially imprisonment."

Just days ago, speaking at the Women in the World Summit 2016, Democratic contender Hillary Clinton said, "Deep-seated cultural codes, religious beliefs and structural biases have to be changed."

The statement sparked concern among those who believe the federal government should not be in the business of using its significant leverage to coerce others to adhere to any specific viewpoint.

There is also concern among the religious community that the IRS has reached some sort of agreement with the Freedom From Religion Foundation (FFRF), which was in the process of suing the agency.

In a legal document dated July 29, 2014, the IRS pledged to engage in tougher monitoring and enforcement of political restrictions on churches, in exchange for FFRF's promise to drop its lawsuit.

Attached to that document is a letter explaining that at least 99 churches had been referred for a "high level examination" by the IRS' "Political Activities Referral Committee."

Despite numerous Freedom of Information Act requests for documents related to any deal with FFRF, the IRS has been less than forthcoming.

As a result, Judicial Watch, a conservative but nonpartisan organization, has filed a federal lawsuit asking that the IRS be forced to disclose them.

When asked to explain Verrilli’s comment, the White House referred Fox News to the Justice Department, which also declined comment and suggested contacting the IRS.The IRS did not immediately respond.