Updated

House Speaker John Boehner is facing mounting criticism from the conservative wing of the party for allegedly punishing certain members over their voting record, with some suggesting his speakership could be at risk.

The odds right now of Boehner losing his post are quite low, barring the sudden entry of a challenger who galvanizes the rest of the caucus. But two developments have raised hackles among conservative members and advocacy groups, prompting grumbling about Boehner's position.

They're disappointed, for one, that he offered to put increased revenue on the table as part of fiscal crisis talks. But they leveled pointed criticism at Boehner after he removed several key conservatives from plum committee assignments. Those who were removed, including Reps. Tim Huelskamp, R-Kan., and Justin Amash, R-Mich., claimed the reshuffling was dealt as punishment for voting against the leadership's wishes.

"I think many of my colleagues are outraged, because the vote is sacred," Huelskamp told Fox News on Thursday.

Days earlier, Huelskamp made a not-so-veiled threat, saying lawmakers saw this kind of behavior under former Speaker Newt Gingrich, "and it didn't work out too well." Gingrich, facing a GOP caucus uprising, resigned his post and later his seat after the 1998 midterms.

More On This...

    The group American Majority Action has also launched the #fireboehner hashtag on Twitter, calling the so-called "purge" of conservatives "the nail in the coffin."

    Thanks to the logistics of speakership elections, though, Boehner still has a lot going for him. The full House -- not just the Republican wing of the House -- would vote for the next speaker. Whoever wins the majority of all votes cast would win the post.

    That means Republicans would want to be united enough to ensure that a Democratic candidate -- most likely, House Minority Leader Nancy Pelosi -- does not win. With one vacancy in the House, the breakdown of the new Congress will be 234 Republicans and 200 Democrats. A few dozen Republicans abstaining could deprive Boehner of the votes he needs, but could also hand the gavel to the minority party.

    Asked on Fox News whether there was any movement in the Republican caucus toward challenging Boehner, Amash did not say. But he said Boehner had stirred "unrest" in the House.

    "The speaker may have miscalculated here, thinking there's just going to be a few of us that are concerned about it, but it's clear he made a threat to the rest of the conference and people aren't taking it very well," Amash said.

    "I think this is going to backfire on Speaker Boehner, and people who run a conference like this have to be very careful."

    It was bound to be a rocky few weeks for the speaker. Aside from the backlash over committee assignments, Boehner is in the unenviable position of having to strike a balance in fiscal talks between a White House that is demanding tax rate increases on top earners and a Republican caucus that is sharply divided on the issue. Some Republicans in Congress have indicated a willingness to compromise on the matter, while others have insisted that not only should tax rate hikes be off the table, but so should efforts to raise revenue by limiting deductions -- as Boehner has proposed.

    "When he couples this conservative purging with a negotiated tax increase of $800 billion, we're starting to see more and more signs that he's not dedicated to fiscally conservative beliefs," Andrew Roth, vice president of government affairs for the Club for Growth, told Fox News. Roth also claimed Boehner could be risking his speakership if he's not careful.

    Boehner's top deputies, though, were on board with both the committee changes and the tax increase offer. The House Republican proposal in fiscal talks with the White House was co-signed by Republican Leader Eric Cantor, R-Va.; Republican Whip Kevin McCarthy, R-Calif.; and Budget Committee Chairman Paul Ryan, R-Wis.

    Fox News' Carl Cameron and Chad Pergram contributed to this report.