Allen died of renal failure Wednesday at a hospital in Takoma Park, Md., The Washington Post reported Friday.
Allen, who was black, started at the White House in 1952, when racial segregation prohibited him from using public restrooms in his native state of Virginia. When he left the White House in 1986 after 34 years, he had witnessed not only defining moments in the country's history, but also in America's civil rights movement.
And on Jan. 20, 2009, he watched Barack Obama being sworn in as the nation's first black president.
"I never would have believed it," Allen told the Post from his seat at the inauguration. "In the 1940s and 1950s, there were so many things in America you just couldn't do. You wouldn't even dream that you could dream of a moment like this."
Allen began washing dishes and stocking cabinets at 1600 Pennsylvania Ave. before rising to maitre d' during Reagan's presidency.
He crossed paths with entertainers including Sammy Davis Jr., Duke Ellington and Elvis Presley. He met Martin Luther King Jr., traveled to Romania with President Richard Nixon, and had a seat at the table as a guest at one of Reagan's state dinners.
Although first lady Jacqueline Kennedy invited him to President John F. Kennedy's funeral, Allen volunteered to stay at the White House to help with the meal after the service. She gave him one of the president's ties, which Allen framed.
Born July 14, 1919, in Scottsville, Va., Allen shared the same birthday as President Gerald Ford and joined in Ford's birthday parties at the White House.
Allen lived in Washington. His wife of 65 years, Helene, died in 2008. He is survived by his son Charles, five grandchildren and four great-grandchildren.