Samuel Nuuyoma, the governor of the Namibian region of Erongo, confirmed the family's departure Saturday but declined to specify the date or where they went.
The Hollywood couple thanked the government and its people at a news conference Wednesday for the peace they enjoyed in the southwest African country.
"We are very proud that our daughter was born here, and we leave with fond memories and definitely hungry to return," Pitt, 42, said.
It was their first public appearance since Jolie, 30, gave birth to Shiloh Nouvel Jolie-Pitt on May 27 at a private clinic in the resort town of Walvis Bay. Delivered by Caesarean section, the baby weighed 7 pounds (3.15 kilograms) and was said to be in good health.
Pitt and Jolie had retreated to Namibia for government-assisted privacy in the weeks leading up to the birth. They were accompanied by Jolie's children, 16-month-old Zahara, adopted from Ethiopia, and 4-year-old Maddox, adopted from Cambodia.
Both children had their surnames legally changed to Jolie-Pitt after Pitt announced his intention to adopt them as well. But the couple has denied rumors of an impending wedding, saying they want to concentrate on the children for now.
Pitt and Jolie were first linked romantically after appearing together in the 2005 movie "Mr. & Mrs. Smith." Jolie, who won an Oscar for her supporting role in 1999's "Girl, Interrupted," is divorced from Billy Bob Thornton and Jonny Lee Miller. Pitt and Jennifer Aniston divorced last year.