Pressure is mounting on Michael Steele to straighten up and fly right after stumbling out of the gate as the chairman of the Republican Party.

Conservatives are losing patience with him. Democrats are predicting his ouster. And Republicans, the people he serves, say he is spending too much time in front of the cameras delivering verbal misfires instead of developing a behind-the-scenes strategy to put the "Grand" back in the Grand Old Party.

Steele's critics are threatening a revolt if he keeps it up. But they may find that removing the party's first black party boss is easier said than done.

"I think the only political way something like that could be done is if they fill the position with Ken Blackwell," Ronald Walters, a political science professor at the University of Maryland, told FOXNews.com, referring to the former Ohio secretary of state. Blackwell, who is black, threw his support behind Steele after losing to him in the RNC election on Jan. 30.

If Blackwell were to replace Steele, "then the issue of race would not be as salient," Walters said.

But Blackwell, because of his conservative credentials, wouldn't be able to fulfill Steele's mission of expanding the base of the party, Walters said.

Blackwell, who told Steele "to get to work or get out of the way" after his latest controversial statement on abortion, said Friday that Steele has two choices as party chairman.

"One track is relevancy and effectiveness. The other is ineffectiveness and irrelevancy," Blackwell told FOX News."It is as difficult to get rid of a chairman as it is to get rid of a bad teacher in an urban public school system, so the choice is his."

Blackwell added that Steele has chosen the former path after clarifying his stance on abortion. Steele recently told GQ magazine that abortion was "an individual choice," but he clarified that position on Thursday by saying he was speaking legally. He said the Supreme Court's 1973 Roe v Wade decision established that women have the legal right to choose abortion, and his statement was "just a recognition of the law. I went on to say that Roe v. Wade was wrongly decided and should be overturned." 

The clarification satisfied Blackwell, who said  it "means [Steele is] going to be the leader of the party that has a clear platform that stands on the side of life."

Steele has always been a provocative figure, but he's become a lightning rod for controversy since he became the party's chairman in a close election in late January and fired the entire RNC staff.

He has raised eyebrows by sending "some slum love" to Louisiana Gov. Bobby Jindal, dismissing popular conservative radio host Rush Limbaugh's show as "incendiary and ugly," and most recently with his abortion remarks.

Steele has apologized to Limbaugh and clarified his stance on abortion, but one RNC member has already called for Steele to step down and rumors are swirling that he could be headed for a no-confidence vote as early as March 31.

Tony Perkins of the Family Research Council said Steele's wounds are self-inflicted.

"I wouldn't say that anybody is attacking Mr. Steele," he told FOX News. "I would say Mr. Steele has been stepping in traps set by himself."

Perkins said Steele made a miscalculation if he viewed the job as a platform to continue his media exposure.

"The RNC is not about exposure of the chairman, it's about building a party that can win elections," he said, adding that the jury is still out on whether Steele will last.

Pennsylvania Gov. Ed Rendell, a Democrat, predicted Thursday that Steele's perch at the top of the GOP pyramid will be short.

"Do [Republicans] want a chairman who is basically pro-choice? Not on your life. They won't permit it," Rendell said.

"Michael Steele's days are numbered. Fortunately for us, his days are numbered."

Democratic strategist Maria Cardona said Steele is sabotaging his own potential by pandering to the conservative base of his party.

"Yes, he did have a rocky start, but I got to tell you, the comments he made about Rush had the effect of having moderates, independents and Obama Republicans think twice about the Republican Party," she told FOX News.

She said that with Steele as chairman, some on-the-fence Republicans were thinking "'Wow, maybe there's room for me there.' Not so much anymore."

FOX News' Brian Wilson contributed to this report.