Updated

Defense Secretary Leon Panetta has not visited Pakistan since taking over as defense secretary a year ago.

In a visit to India and Afghanistan this week, he skipped Islamabad again and expressed outright frustration with the Pakistani leadership and the safe havens they provide the Taliban in the Northwest Frontier province, especially the so-called Haqqani network.

Admiral Mike Mullen in a parting shot at the Pakistanis did not mince his words when he said Pakistan's intelligence service provides direct support of the Haqqani network, whose fighters are killing U.S. troops in Afghanistan.

In an exclusive interview with Fox News on the tarmac in Kabul, Secretary Panetta was forthright in expressing his mounting frustration with the Pakistanis.

"We cannot continue to tolerate a situation where Haqqanis, terrorists on their side of the border, come across, attack our troops, kill our troops, and then return to a safe haven in Pakistan. That's intolerable," Panetta said before boarding his plane back to the U.S.

"You know we have urged them time and time again that they have to deal with that situation, that we cannot allow that to happen. And we’ve reached the limits of our patience with regards to just standing back and not having the Pakistani’s take the action that they have to take if we're going to control that situation."

He added the U.S. is at war in the FATA, or Pakistan's Federally Administered Tribal Areas, but stopped short of saying the US was at war with Pakistan.

"You know, without ever getting into the details of what we may or may not do I think it suffices to say that the United States will do whatever we have to do to protect our forces," Panetta said.

He added the Haqqani network was responsible for a sophisticated attack this week that left dozens of Afghans dead near a NATO base.

They were the strongest words yet from a U.S. defense secretary about the double game Pakistan is playing with the United States in Afghanistan.

Pakistan closed the borders with Afghanistan last November, forcing the U.S. military and NATO to come up with alternative more expensive supply routes through Central Asia.

Pakistani Ambassador Sherri Rehman called Panetta's remarks, "unhelpful."