A former State Department employee who helped Hillary Clinton set up her private email server refused to testify to a House committee Thursday, asserting his Fifth Amendment right not to answer questions. 

Bryan Pagliano appeared briefly behind closed doors Thursday before the House committee investigating the deadly 2012 Benghazi attacks. 

Afterward, Chairman Trey Gowdy, R-S.C., confirmed to reporters that he asserted his right not to testify. He would not comment extensively on the implications of that. 

"He has a right to not answer questions that he thinks may incriminate him -- and you have a right to glean whatever inference you want from the fact that he has," Gowdy said. 

Pagliano's lawyers said he also will refuse to testify if served with a subpoena by two Senate committees looking into the email controversy. 

Gowdy's Democratic counterpart, Rep. Elijah Cummings, D-Md., charged that Republicans on the committee were wrongly turning the panel's attention away from the 2012 Benghazi attack to focus on Clinton's personal email controversy. He said Pagliano probably has no information on Benghazi and was dragged before the committee over "politics." 

He said the testimony "has nothing to do with the Benghazi attacks and everything to do with Republicans' insatiable desire to derail Secretary Clinton's presidential bid." 

At the same time, Cummings said he wants to hear what Pagliano has to say and urged his colleagues to consider granting him immunity -- something Republican Sens. Chuck Grassley and Ron Johnson have said they're considering. 

But Pagliano's lawyers said in a letter to the senators that any such offer is premature. The senators' proposal, made in a letter earlier this week, "creates the very practical risk that our client will later be said to have waived his constitutional protections," said the letter, written by lawyers Mark MacDougall, Constance O'Connor and Connor Mullin. 

Clinton, the former secretary of state and front-runner for the 2016 Democratic presidential nomination, has been dogged by questions about her use of a private email account for government business. 

In a separate letter, the three lawyers accused the chairman of the House Benghazi committee of trying to bully their client by forcing Pagliano to appear before the panel. 

Forcing him to appear at a public location "can only be intended to intimidate our client, cause him personal embarrassment and foster further political controversy," the lawyers wrote in a letter to Gowdy. 

Gowdy denied that in a letter to MacDougall and cited comments by Clinton's former chief of staff, Cheryl Mills, complimenting Gowdy and other committee members for their "professionalism and respect" during more than nine hours of closed-door testimony before the committee last week. 

The Associated Press contributed to this report.