He was 85 years old. The IOC and Tunisia's president praised Mzali's contribution to sports and culture over the decades.
Mzali was a minister in Tunisia's Cabinet for several years before serving as prime minister from 1980 to 1986. He was elected a member of the IOC in 1965 and held several top positions on the committee.
His family said his body was being flown back to Tunisia on Thursday for burial.