The chairman of the House oversight committee confronted the Secret Service director on Tuesday over what he claimed was a "botched response" to a bomb threat outside the White House, playing videotape he said shows it took far too long to secure the scene after a woman dropped off what she claimed was the bomb.

The March 4 incident is the same one in which two Secret Service agents were accused of driving through the crime scene after returning from a party. However, oversight committee Chairman Jason Chaffetz, R-Utah, said at a hearing with agency Director Joseph Clancy that the problem goes deeper than the two potentially intoxicated agents.

He claimed it appears the bomb threat was mishandled from the start -- at a time when President Obama was in the residence.

"We expect answers," he said.

Chaffetz showed surveillance footage provided by the Washington, D.C., police department. The time-lapse video showed a woman dropped off a package outside the White House on the night of March 4. She apparently claimed this was a bomb. The video, toward the end, also showed the two agents in question arriving on the scene and bumping into a security barrier.

But Chaffetz, before playing the video, said after agents were initially unable to apprehend the suspect, the package sat "unattended as traffic drove by for a long period of time." He said it took 11 minutes for the agency to call the Metropolitan Police Department bomb squad. And for 17 minutes, he said, traffic continued to drive by and "several pedestrians walked within feet of the potential bomb."

"I don't understand how that happens," Chaffetz said.

Chaffetz also said that an agent initially followed the suspect's car but was "mistakenly called off the pursuit when the Secret Service identified the wrong car as the suspect's." It took 30 minutes after she fled for the Service to issue a "lookout" for the vehicle.

After an hour and 20 minutes, the scene was cleared. The "bomb" turned out to be a book. The suspect was apprehended three days later.

Chaffetz was joined in his criticism by the committee's top Democrat, Maryland Rep. Elijah Cummings, who said he has "great concerns" about the incident.

Both committee leaders continue to press for answers about the two agents who drove through the scene, and near the suspicious package, after a night of partying. But their questions about the handling of the bomb threat itself open a new area of concern -- and just the latest for an agency that has been accused of botching the response to a series of threats in recent months.

Clancy, for his part, stressed that the Department of Homeland Security inspector general is investigating the incident. He said if misconduct is revealed, people will be held "accountable."

He continued to say he was outraged that he didn't learn about the two agents driving through the scene until five days after the fact. He apparently only learned about this after hearing about an anonymous email circulating in the Secret Service. That email, released by the committee, said the two agents were said to be "extremely intoxicated" when they arrived on the scene and "nudged" the barrels -- then were "waving their White House passes around confused as to why the post was evacuated."

That email said officers were going to arrest them but a watch commander "said not to."

Hours into Tuesday's hearing, Chaffetz lashed out at Clancy as he voiced frustration with the director's response. "This is the United States of America," he said, raising his voice. "The threat is real."

The hearing is Clancy's latest appearance on the heels of the March 4 incident.

Clancy was the only witness at Tuesday's hearing, though committee leaders had wanted to hear from multiple Secret Service officials. Chaffetz and Cummings wrote a harsh letter to Clancy over the weekend saying they were "sorely disappointed" to learn he would be the only witness.

Chaffetz and Cummings said Clancy was "unable to answer questions" at a briefing on March 17 about what happened, how the suspect involved with the suspicious package left the crime scene, and other issues.

And they said Clancy had offered "no explanation whatsoever" for why other officials couldn't testify.

In an interview with FoxNews.com Friday, Chaffetz also questioned a claim by the Secret Service that key surveillance footage of the March 4 incident had been taped over as a matter of policy. Chaffetz called the claim "highly suspicious" and said he had "a hard time believing that's true." Sunday's letter from Chaffetz and Cummings said that the lack of available video had "further limited the Committee's understanding of the incident."

Lawmakers have been shown two tapes so far -- one of which Chaffetz called "clearly useless" -- but footage from other fixed positions around the White House grounds may be lost because of the Secret Service's apparent practice of taping over most footage after 72 hours.

Clancy agreed Tuesday to show the committee any video it has, though said he could not provide copies.