The Supreme Court on Tuesday seemed willing to make it tougher for the government to recover ill-gotten gains from people convicted of securities fraud.

Most of the justices — including newly confirmed Justice Neil Gorsuch — suggested during arguments that such recoveries are subject to a strict five-year statute of limitations.

That could prevent the Securities and Exchange Commission from collecting huge amounts of money in many cases where the alleged fraud goes back years or decades before officials bring charges. The agency collected more than $4 billion in those so-called "disgorgement" actions and other penalties in the 2016 fiscal year.

The high court is deciding whether venture capitalist Charles Kokesh must return $35 million from investor funds he used to pay himself and others at his New Mexico-based operation from 1995 to 2006.

The federal appeals court in Denver ruled that Kokesh must pay the full amount to the SEC. Lawyers for Kokesh say the five-year window should reduce his payment to just $5 million because the SEC did not bring charges against him until 2009.

At issue is a law that prevents the government from going back more than five years to impose penalties or seek forfeiture of assets. The SEC argues the time limit doesn't apply because it's not assessing a penalty, only trying to recover illicit profits.

But justices across ideological lines seemed skeptical of the distinction.

"It does seem to me that we kind of have a special obligation to be concerned about how far back the government can go when it's something that Congress did not address," Chief Justice Roberts said.

Justice Elena Kagan said the SEC seemed to be using disgorgement both to compensate victims as well as to punish misconduct.

"It's a little bit artificial to try to tear them apart," she said.

Kokesh's attorney Adam Unikowsky told the justices that if disgorgement is about both recovering assets and imposing a penalty, it has to be considered punitive overall and subject to the five-year limit.

"The purpose of the remedy is to impose unpleasant legal consequences of wrongdoing," he said.

Justice Department lawyer Elaine Goldenberg said it's not a punishment "because it doesn't take away anything that anyone was rightfully entitled to in the first place. It just remedies unjust enrichment."

But Gorsuch noted that in a criminal case, the same kind of remedy would be considered a penalty.

"So why does it make a difference that we just happen to be in the civil context," he said. "I mean, goodness gracious, the difference between civil and criminal has vexed this court for many years."

The case comes to the high court from the 10th U.S. Circuit Court of Appeals, where Gorsuch served for more than a decade before he was confirmed to the Supreme Court. Gorsuch was not part of the three-judge panel that ruled against Kokesh last year. But he might find himself in the position of reversing his former colleagues if the high court sides with Kokesh.

A ruling is expected by June.