IRS Commissioner John Koskinen met with two groups of House Republicans Wednesday amid an internal GOP debate about whether to move forward with an impeachment vote. 

Koskinen met with members of the moderate "Tuesday Group" before being invited to join the conservative "Republican Study Committee."

"It got a little contentious," Rep. Bill Flores, R-Texas, said of Koskinen's discussion with the Republican Study Committee.

Conservative Republicans say Koskinen should be expelled from office for allegedly impeding a congressional investigation into the IRS' improper targeting of tea party groups seeking tax exemptions. House Republican leaders oppose impeaching Koskinen, saying such a move risks making GOP lawmakers look overly partisan and sets a dangerous precedent.

"If you were to impeach and dump this right into the Senate, throw it in their hands right before a very contentious election, is that something they want to deal with right now?" asked moderate Rep. Charlie Dent, R-Pa.

Flores said that some of the Republican Study Committee's members have "made up their mind ... but a lot of people have some thinking to do."

Even if Koskinen were impeached, there is little chance that the Senate would vote to remove him from office. The House has voted only 19 times to impeach federal officials: Two presidents, 15 federal judges, a senator and a Cabinet member. Only eight — all judges — have been convicted and removed from office by the Senate.

House GOP lawmakers are due to meet next Thursday to discuss the matter.

Fox News' Chad Pergram and the Associated Press contributed to this report.