The White House on Wednesday rejected former President Jimmy Carter's claim that an "overwhelming portion" of criticism directed at the president is based on race. 

"The president does not believe that that criticism comes based on the color of his skin," White House Press Secretary Robert Gibbs. 

Gibbs gave a similar response Sunday after New York Times columnist Maureen Dowd claimed that administration critics like South Carolina Rep. Joe Wilson, who accused President Obama of lying during his health care address to Congress, are motivated by race. 

Carter, though, is the highest-profile person so far to push that claim. Carter said Tuesday that Wilson's outburst was "based on racism" and, more broadly, that "an overwhelming portion of the intensely demonstrated animosity toward President Barack Obama is based on the fact that he is a black man." 

Asked repeatedly about such assertions at his daily press briefing, Gibbs said that much of the criticism directed at the president, to the contrary, is based on his decisions regarding the economy. 

"We understand that people have disagreements with some of the decisions that we've made and some of the extraordinary actions that had to be undertaken by both this administration and the previous administration," Gibbs said. 

He said Obama has always had "great concerns" about race relations but downplayed the controversy over Carter's remarks as well as charges of racism from other Democrats. 

"I'm not sure I see this large national conversation right now," Gibbs said. 

He said Obama has not asked other Democrats to stop talking about race.