The buck stops with the president. Or maybe it's the secretary of Defense.

As the fallout grows over the controversial Taliban prisoner swap, White House officials are emphasizing that Defense Secretary Chuck Hagel was the one who "signed off" on the deal -- a move one Republican lawmaker suggested is an effort to shift blame.

"I hope they're not just pushing him out to be a fall guy for this," Rep. Buck McKeon, R-Calif., told Fox News.

According to a congressional source, Deputy National Security Adviser Tony Blinken refused to answer directly when asked point-blank during a briefing with House lawmakers Monday evening who made the call.

The source said Rep. Tom Cotton, R-Ark., asked the question. Blinken would only respond by saying: "Secretary Hagel signed off on it."

That comment is not necessarily a revelation. By law, the secretary of Defense is required to sign documents approving detainee transfers.

McKeon noted the White House is stressing Hagel's role, though, following intense controversy over the trade of five Taliban members held at Guantanamo Bay for Sgt. Bowe Bergdahl. He noted that he "didn't see Secretary Hagel" on TV when President Obama first announced the release alongside Bergdahl's parents in the Rose Garden.

"It sounded like it was a presidential program all the way," McKeon said. "Now that there's been a little pushback ... they said Secretary Hagel [made the call]."

McKeon added: "I think people understand who made this decision. This goes right to the top -- or maybe we don't know who's in charge of the White House."

Defense officials say that Hagel's comments on NBC's "Meet the Press" immediately following the release speak for themselves.

"I signed off on the decision," Hagel said in that interview. "The president made the ultimate decision."

Asked Tuesday to clarify who made the final call, White House spokesman Josh Earnest gave a multipronged response.

He said it is the commander-in-chief's responsibility to make sure no servicemember is left behind. Asked again whether Obama approved the swap, Earnest pointed to the president's role in reviewing security concerns about releasing the five Gitmo detainees.

"The president determined that ... that threat to our national security had been sufficiently mitigated," he said. "That was something that the secretary of Defense also certified. But ... that was also the widespread, unanimous agreement of the president's national security team."

Hagel is set to testify before the House Armed Services Committee on Wednesday, and is likely to face many questions about the Taliban-for-Bergdahl trade.

Fox News' Justin Fishel and Ed Henry contributed to this report.