Banned Los Angeles Clippers owner Donald Sterling is refusing to pay the $2.5 million fine levied by the NBA over his recent racist comments, Sports Illustrated reported.

Maxwell Blecher, an attorney for Sterling, wrote in a letter to NBA executive vice president and general counsel Rick Buchanan that Sterling has done nothing wrong and that "no punishment is warranted," according to the report.

"We reject your demand for payment," Blecher writes in the letter, which reportedly discusses two possible legal defenses for Sterling as he moves toward litigation.

The NBA's constitution, which Sterling signed as controlling owner of the Clippers, gives its Board of Governors broad latitude in league decisions including who owns the teams.

NBA Commissioner Adam Silver is pushing for a swift vote against Sterling, which requires a minimum of three-fourths of the other 29 controlling owners to agree.

Silver also has imposed a lifetime ban on Sterling in addition to the fine.

In the letter, Blecher argues that Sterling's "due process rights" were violated when he was permanently banned after a mere four-day investigation. Blecher also says Sterling has not violated any article of the NBA constitution, Sports Illustrated reported.

Sterling has repeatedly said he does not want to sell the Clippers. In his recent interview with CNN's Anderson Cooper, he cast doubt on going to court if the NBA governors ultimately do vote to force him out.

"People want me to hire a wall of lawyers and them to have to hire a wall of lawyers and go to war," Sterling said on CNN. "I don't think that's the answer."

Sterling's longtime attorney, Robert Platt, declined to comment when contacted Wednesday.

Shelly Sterling's attorney, Pierce O'Donnell, did not respond to email requests for comment from The Associated Press. But he has previously said she wants to remain a passive owner of the Clippers even if her husband is no longer involved.

For now, the NBA has installed former Time Warner and Citigroup chairman Dick Parsons to oversee the team's business operations. Parsons said this week that a prolonged legal battle "is in no one's interest."

"I would hope we could avoid that," he said.

If he is forced out, Sterling still stands to reap a huge financial windfall in a Clippers sale. He bought the team for $12.5 million in 1981, and Forbes magazine recently placed its 2014 value at $575 million, or No. 13 in the NBA. Of course, there would also be a sizable capital gains tax bill for that.

The Associated Press contributed to this report.

Click here for more from Sports Illustrated.