Supreme Court justices signaled Monday that they could deal a major blow to public-sector unions, in a case challenging the controversial dues that organized labor forces workers to pay.

The court’s conservative justices, and a key swing justice, made skeptical comments about the legality of the payments during 80 minutes of oral argument in Washington.

At issue is a four-decade-old precedent that allows public unions to collect "fair share" fees from non-members to help cover the costs of collective bargaining.

Those dues have helped preserve the strength of America’s public-sector unions even as their private-sector counterparts have withered over the decades. But a group of California public school teachers claims those mandatory fees violate the First Amendment rights of workers who disagree with the union's positions.

“Really, these unions are not speaking on my behalf. They're speaking on behalf of the union and the union leadership,” school teacher and plaintiff Rebecca Friedrichs said.

Justice Anthony Kennedy, often a swing justice, rejected arguments made by lawyers for the state of California and the California Teachers Association that the current fee system is needed to prevent non-members from becoming "free riders" who get all the benefits of union bargaining and grievance procedures without paying for it.

"The union is basically making the teachers `compelled-riders' on issues with which they strongly disagree," Kennedy said.

Arguing in support of the union, California Solicitor General Edward Dumont said the state needs a reliable bargaining partner that is funded by all the workers it represents. He said the fees for collective bargaining typically apply to non-political issues such as mileage reimbursement, working hours and other mundane issues.

But Chief Justice John Roberts said even routine matters can become politically charged if they involve how the state spends money.

"That's always a public policy issue," Roberts said.

The California teachers want the high court to overturn a 1977 precedent that allows public unions to collect money from non-members to help cover the costs of collective bargaining as long as the money doesn't go to political causes. The court in that case, Abood v. Detroit Board of Education, said the arrangement was justified to prevent non-members from becoming free riders.

But the teachers argue that unions have become more political over time. They say even a push for higher salaries and pension benefits raises political questions about the best use of tax dollars for cash-strapped localities.

A federal district court ruled against the teachers, saying the outcome was clear under Abood. The 9th U.S. Circuit Court of Appeals affirmed.

Arguing for the teachers, Michael Carvin said if the current regime is upheld, every public employee in California is essentially subsidizing "the governor's political war chest."

Justice Elena Kagan warned that the challengers bear a heavy burden to overturn a nearly 40-year-old case on which thousands of contracts and millions of employees rely.

Half the states already have right-to-work laws banning mandatory fees, but most members of public-employee unions are concentrated in states that don't, including California, New York and Illinois.

Labor officials fear that union members will leave in droves if they realize they can get all the benefits of representation without paying for it. Union advocates say the lawsuit is part of a conservative agenda to weaken powerful labor unions, known for reliably supporting Democratic candidates and policies. The teachers are backed by the conservative Center for Individual Rights. 

Carrie Severino, chief counsel at the Judicial Crisis Network who filed a brief supporting the plaintiffs, said in a statement Monday that the court seemed to recognize that the fees force workers to effectively “subsidize” political speech.

“As Justice Kennedy observed, forcing nonmembers to subsidize a union they disagree with doesn't make them free riders, it makes them compelled riders. Today it seems like the court is finally ready to let them get off that train,” she said. 

A decision in the case, Friedrichs v. California Teachers Association, 14-915, is expected by late June.

Fox News’ Shannon Bream and The Associated Press contributed to this report.