President Obama has been taking it on the chin lately from one of his most committed constituencies -- members of the Congressional Black Caucus who now say Obama isn't considering race enough when he makes decisions on jobs, the economy, health care and other issues.

"We can't allow race to be denied, we can't allow race to just be a part of the public discourse at this point in time," Rep. Bobby Rush, who is from the president's home state of Illinois, told Fox News on Wednesday.

"As a candidate, President Obama said in a speech on race during the Democratic primary, 'race is an issue that I believe this nation cannot afford to ignore right now," Democratic Rep. Barbara Lee, chair of the CBC, said in a statement Tuesday night.

"The Congressional Black Caucus recognizes that behind virtually every economic indicator you will find gross racial disparities," she added.

Among those disparities, Lee pointed to the fact that 24 percent of blacks currently live below the poverty line and that African Americans are 55 percent more likely to be unemployed than other Americans.

White House Press Secretary Robert Gibbs told reporters Wednesday that Obama believes the economic plan he has outlined will help those living in communities where the unemployment rate is higher than the national average.

But, he added: "I don't think the president believes that we should address only one part of the unemployment rate."

The criticisms come after the 43-member caucus boycotted a financial overhaul vote last week, and after Democratic Rep. John Conyers, a founding member of the CBC and the longest-serving African American member of the House, told the Hill newspaper that the president's agenda is askew from the larger black community.

"I have been saying I don't agree with him on Afghanistan. I think he screwed up on health care reform, on Guantanamo" and on removing Counsel Greg Craig from his post, Conyers told the newspaper.

Conyers also said that Obama called him recently to ask why Conyers was "demeaning" him. He reportedly told the president it was not personal, "just an honest difference on the issues."

Asked about the phone call on Wednesday, Gibbs said Obama believed that criticisms from Conyers and others were "untrue" so the president "reached out and touched someone."

A spokesman for Conyers told Fox News on Wednesday that the House Judiciary Committee chairman "has great respect for the president, is appreciative of his outreach and is confident that they can continue to work together on issues of importance to the American people."

Fox News' Chad Pergram contributed to this report.