Democrats say Panetta told Congress last month that senior CIA officials have concealed significant actions and misled lawmakers many times since 2001.
House Intelligence Committee Chairman Silvestre Reyes and six other of the 13 Democrats on the panel sent Panetta a letter on June 26 demanding he retract a May 15 statement that it is not CIA "policy or practice to mislead Congress."
Panetta made the earlier statement after Pelosi accused the spy agency of lying to her and Congress over enhanced interrogation techniques used on terror detainees during the Bush administration.
Reyes, D-Texas, also wrote to Michigan Rep. Peter Hoekstra, the committee's senior Republican, to say that he is considering opening a full investigation into the CIA's communications with Congress.
Pelosi had little to say Thursday about the events that shine the spotlight back on her controversial remarks.
"I've seen the letters from the members and obviously they have concern," Pelosi said at her weekly news conference.
"The Intelligence Committee has the oversight responsibility for intelligence in the House and its equivalent committee in the Senate," Pelosi said "I'm sure they will be pursuing this in their regular committee process and that's the way it will go."
CIA Spokesman George Little told FOX News that Panetta "stands by his May 15 statement."
"It is not the policy or practice of the CIA to mislead Congress. This agency and this director believe it is vital to keep the Congress fully and currently informed. Director Panetta's actions back that up. As the letter from these six representatives notes, it was the CIA itself that took the initiative to notify the oversight committees."
Sources told FOX News that Panetta gave the June 24 testimony because he didn't feel an issue had been "adequately" briefed but they didn't say what the issue was. However, the sources said the issue spanned a number of CIA directors and did not include waterboarding or the enhanced interrogation program.
Reyes, D-Texas, said in the briefing Panetta revealed the CIA outright lied about one case.
However, the letter sent to Panetta said the Democratic lawmakers wanted a retraction because Panetta told the committee that "top CIA officials have concealed significant actions from all members of Congress, and misled members for a number of years from 2001 to this week. This is similar to other deceptions of which we are aware from other recent periods."
Republicans say the allegations are just political maneuvering to protect Pelosi, who has been under constant fire since she made the allegation.
On Thursday, Hoekstra called Reyes' letter "one of the most bizarre episodes in politics that I've seen in my time here in Washington."
"It looks like they're working on the political equation," Hoekstra said on CBS' "The Early Show." "They're not trying to foster a bipartisan consensus on national security."
Hoekstra said Panetta disputed the characterization of what Democrats said he told the committee last month but is wary of being more vocal.
"I think he's caught between a rock and a hard place," Hoekstra told FOX News. "He's a former Democratic member of Congress and they are mischaracterizing what he said to the intel community and I think he's just stepping back."
Ever since Pelosi made her accusation against the CIA, Republicans have demanded she provide proof that the CIA failed to tell her the truth. Pelosi said Thursday she was not present for Panetta's testimony and that she learned about the letter through the press.
She added that Republicans are looking for "any excuse" to stir the debate.
"Our success is driving Republicans to distraction. Any excuse will do," she said. "But the fact is is that there is a briefing of serious concern to members of the committee, and they have their course of action to deal with it."
Hoekstra said the timing of Reyes' allegations is surprising.
"You would have thought he would have started that in May when the speaker, the third most powerful person in America said the CIA misleads and lies to Congress consistently over a period of years. We are now in the middle of July and guess what? Silvestre Reyes and the Intelligence Committee have held zero hearings on the speaker's allegations or on any of these more recent allegations. They're not doing anything," he said.
The letter to Panetta is not written on official congressional letterhead. A House source said this was not an official communication from the House Intelligence Committee and that others on the panel were concerned that the co-signers were trying to force Panetta to apologize for alleged problems created by the Bush administration's CIA.
The cryptic letter and CIA statement came on the eve of a House debate on an intelligence bill. The debate is expected to revive a partisan argument that has raged on and off for months about whether Pelosi knew in the fall of 2002 about the CIA's use of waterboarding weeks earlier.
In an interview with FOX News, Reyes said he was concerned that Republicans might try to corner Pelosi on the interrogation flap during the bill's debate.
"We all know there have been a number of contentious issues that have been much publicized," said Reyes, who indicated he wrote the letter in an attempt to curb GOP efforts to target Pelosi.
"Let's leave for the moment the politics out of it," Reyes said. "Our national security is about the most serious thing we deal with here on Capitol Hill. And it shouldn't be politicized."
As for the intelligence bill, House Republicans oppose at least one provision, and they have an unusual ally: the White House.
Obama's aides have said they will recommend he veto the bill if it includes a Democratic-written provision requiring the president to notify the intelligence committees in their entirety about covert CIA activities.
Under current law, the president is only obligated to notify the top Democratic and Republican leaders of the House and Senate and the senior Democratic and Republican members on each chamber's intelligence committee.
Democrats want to open the briefings to all members of the House and Senate intelligence committees unless committee leaders agreed otherwise. That would be about 40 lawmakers, depending on shifting membership rosters, instead of the eight required by law.
They claim the Bush administration sought to undermine congressional oversight. However, the White House is concerned that briefing more lawmakers might compromise the most sensitive U.S. intelligence operations.
FOX News' Chad Pergram, Jim Angle, Catherine Herridge and The Associated Press contributed to this report.