Rep. Paul Ryan has become the top target of pro-Trump forces lurking in the halls of Congress.

Although the Wisconsin Representative said he cast his ballot for the party’s nominee on Tuesday, the House Freedom Caucus reportedly held a “secret meeting” Wednesday morning to discuss their post-election plans. Allegedly on the agenda: a challenge to Paul Ryan’s position as Speaker of the House.

The Freedom Caucus said the meeting was to “chart the group’s moves” in November and beyond, but most of the members, who came to power during the GOP’s Tea Party phase, from 2009 to 2014, are Trump allies or, at least, supporters, who know Ryan is Trump’s top target.

And while the Caucus is small, it’s mighty. Its last victim was House Speaker John Boehner, whom they ousted to make way for Ryan just last year. And they’re determined to punish Ryan this time around for forsaking the party’s nominee.

The Caucus may or may not be able to make good on their promises. The Republicans look set to lose seats in their House majority, and the Freedom Caucus’s numbers aren’t enough to boot Ryan outright. But they could run a Freedom Caucus candidate against him for the speakership role, or challenge him openly over upcoming legislation—likely, trade deals that are still on the table, or Ryan’s pet “better government” reforms.

Read more from Heat Street.