The shake-up comes as the American public questions whether the fight in Afghanistan can be won, and the Defense Department is reeling from losing its top war commander — Gen. Stanley McChrystal.
Defense Secretary Robert Gates told reporters that he was impressed with the general's "strategic insight and independent thinking."
Mattis is a blunt-talking, seasoned war veteran best known for leading troops into the bloody battle of Fallujah in Iraq in 2004.
He is a bit of a surprising pick. On the one hand, Mattis has significant ground combat experience and is considered an intellectual who grasps the nuances of fighting a complicated counterinsurgency.
But he is also known to speak bluntly and off-the-cuff — much like Petraeus' predecessor Gen. McChrystal who was fired for speaking ill of his civilian bosses.
In 2005, Mattis was chastised by his superiors for saying in a public speech in San Diego that it was "fun to shoot some people." Mattis was a three-star general at the time, stationed in Quantico, Va., when he told an audience that some Afghans deserved to die.
"Actually, it's a lot of fun to fight," he said. "You know, it's a hell of a hoot. . . . It's fun to shoot some people. I'll be right upfront with you. I like brawling."
He added, "You go into Afghanistan, you got guys who slap women around for five years because they didn't wear a veil. You know, guys like that ain't got no manhood left anyway. So it's a hell of a lot of fun to shoot them."
His comments evoked laughter and applause from the audience.
His boss at the time, Gen. Mike Hagee, said that the comments reflected the "unfortunate and harsh realities of war" but that Mattis had been asked to watch his words in public.
Gates said Thursday that appropriate action was taken at the time. He also said he raised the issue with Mattis when interviewing him for the job and was confident that such statements would not be made in the future.
"I think the subsequent five years have demonstrated that the lesson was learned," he said.
In 2004, Mattis' Marine division led the assault on Fallujah and he played a key role in helping Iraqi security forces negotiate with insurgents inside the city.
His remarks at the time of the battle suggest his thinking is very much in line with the counterinsurgency strategy pursued by Petraeus and McChrystal, which restricts military operations in order to win the support of the local population.
"All along we had intended a softer approach, using civil-military operations... unless someone chooses to fight, and then we would fight," Mattis said in 2004. "Welcome to war with all its complexities and shifting centers of gravity."