Updated

Former President Jimmy Carter drew widespread criticism Wednesday for saying that Rep. Joe Wilson's "You lie!" outburst last week was "based on racism" and that an "overwhelming portion" of similar demonstrations against President Obama are rooted in bigotry.

Obama's supporters have attributed racist motives to some opponents of his health care plan for weeks, but Carter is the highest-profile person so far to push that claim.

While some anti-Obama demonstrators have been seen carrying over-the-top or racially offensive signs, administration critics say Carter is flat wrong to claim that those fringe protesters make up the bulk of Obama's detractors.

"I don't see race as an issue. It's all about the policies that are coming out of the current administration," said Deneen Borelli, a black conservative who spoke at the protest rally held in Washington Saturday. Much of the condemnation of Obama's critics has come as a response to that protest, where tens of thousands demonstrated against big government and over-spending.

"I just see this as the race card being used once again to distract the American people from the core issues," Borelli said.

More On This...

The White House, too, distanced itself from Carter's comments.

"The president does not believe that that criticism comes based on the color of his skin," White House Press Secretary Robert Gibbs said.

But, Carter againWednesday repeated what he said a day before.

"I think people are guilty of that kind of personal attack against Obama, have been influenced to a major degree by a belief that he should not be president because he happened to be African American," Carter said during a town hall meeting in Atlanta.

Adam Brandon, spokesman for protest organizer FreedomWorks, said Carter's comments were "absurd." He noted that last Saturday's protest featured about a dozen black speakers.

"To say this crowd was racist is absolutely absurd when black speakers were probably the most popular speakers," he said.

"I think it's very destructive for America to suggest that we can't criticize a president without it being a racial act," former House Speaker Newt Gingrich told FOX News.

The suggestion that race is behind criticism of Obama has been made by New York Gov. David Paterson and Reps. Charlie Rangel of New York, Diane Watson of California and Hank Johnson of Georgia, among others.

But a poll released Wednesday by Rasmussen Reports showed that just 12 percent of voters believe that most opponents of Obama's health care reform plan are racist. The survey of 1,000 likely voters, taken Monday and Tuesday, found that 67 percent disagree with that contention, while 21 percent are not sure. The survey had a margin of error of 3 percent.

Carter, though, said in an interview with NBC that race is the driving factor.

"I think an overwhelming portion of the intensely demonstrated animosity toward President Barack Obama is based on the fact that he is a black man, that he's African American," Carter said. "I live in the South, and I've seen the South come a long way and I've seen the rest of the country that shared the South's attitude toward minority groups at that time ... and I think it's bubbled up to the surface, because of a belief among many white people, not just in the South but around the country, that African-Americans are not qualified to lead this great country."

At a town hall at his presidential center in Atlanta Tuesday, Carter also said Wilson's outburst -- the South Carolina Republican shouted "You lie!" at Obama during his health care address to Congress -- was racially motivated.

"I think it's based on racism," Carter said in response to an audience question. "There is an inherent feeling among many in this country that an African-American should not be president."

Wilson's eldest son stepped up to his father's defense.

"There is not a racist bone in my dad's body," said Alan Wilson, an Iraq War veteran who is running for state attorney general in South Carolina. "He doesn't even laugh at distasteful jokes. I won't comment on former President Carter, because I don't know President Carter. But I know my dad, and it's just not in him. ... It's unfortunate people make that jump."

South Carolina's former Democratic Party chairman said that he doesn't believe Wilson was motivated by racism, but said the outburst encouraged racist views.

"I think Joe's conduct was asinine, but I think it would be asinine no matter what the color of the president," said Dick Harpootlian, who has known Wilson for decades. "I don't think Joe's outburst was caused by President Obama being African-American. I think it was caused by no filter being between his brain and his mouth."

Daniel Hannan, a conservative British politician and member of the European Parliament whose public criticism of the British health care system has drawn international attention, said Wednesday that there's unavoidably some "element of racism" in the most aggressive criticism of Obama. But that's not the majority.

"The overwhelming majority of critics of the president are not motivated by any personal dislike but have reached the view that he's making a mistake and he's indebting the country, that he's enlarging the federal government at the expense of both of the state and of the citizens," Hannan told FOX News. "I hope (Carter) thinks again about that phrase ... It really isn't a race thing."

Rep. Donna Edwards, D-Md., a member of the Congressional Black Caucus, said she, too, didn't think Wilson's outburst was race-related.

"I think there's a lot of opposition, visceral opposition to his policies, but the reality is that this president won, he won with an overwhelming majority of support across the board from the American people and not not just from African-Americans, and it is time for us to move on and get down to the business of making the kind of change that the president outlined when he won the election last year," she said.

But several Obama supporters say they worry that racism is not only at the core, but could lead to worse incidents than a shout on the House floor.

People will be putting on "white hoods and white uniforms again and riding through the countryside" if emerging racist attitudes, like those subtly supported by Wilson, are not rebuked, Johnson said.

The Associated Press contributed to this report.