He died of diabetes complications, according to his former Dominican team, Leones del Escogido. The Giants also confirmed his death and said Alou had been sick for several years with a variety of health issues.
Hall of Famer Orlando Cepeda said he knew his "great friend" had been ailing.
"We roomed together a few times with the Giants," Cepeda said by phone. "Very funny guy, hell of a ballplayer. When Matty was playing with the Giants, he was a dead fastball hitter, he could pull anybody, I don't care how hard they threw."
A two-time All-Star, Alou became the 1966 National League batting champion with the Pirates when he hit .342. He spent his first six years with San Francisco from 1960-65 and also played for St. Louis, Oakland, the New York Yankees and San Diego.
"Although he played for six different teams, Matty remained a part of the Giants family as a longtime employee and will be forever linked with his brothers, Felipe and Jesus, — as the first all-brother major league outfield, the Giants said.
Alou was a career .307 hitter with 31 home runs, 427 RBIs, 1,777 hits and 236 doubles in 15 major league seasons.
"He went to Pittsburgh and switched from a light bat to a heavy bat, and he hit .340 one year," Cepeda recalled.
The Alou brothers made history in 1963 when they appeared in the same outfield for several games. Felipe Alou, who managed the Giants for four seasons from 2003-06, takes pride in how the Alou name has endured in baseball.
"It's a family legacy," Felipe said during his time managing the Giants. "The Alou legacy is a legacy in itself. I see all of us together being a force going through this game, and still going. The respect, I'm proud of that, and length of service."
Leones President Luis Manuel Bonetti added: "Dominican baseball in general and Leones in particular, has lost one of its most emblematic figures and an extraordinary human being."
Felipe Alou still believes he cost the Giants a championship in 1962 when he failed to get down a bunt in the ninth inning of Game 7 of the 1962 World Series. It would have moved Matty from first to second. The Giants lost the game 1-0 and the Series to the New York Yankees.
In 2010, San Francisco finally captured the city's first title since moving west in 1958.
Associated Press writer Dionisio Soldevila in Santo Domingo, Dominican Republic, contributed to this story.