The 71-year-old made the revelation during an appearance Thursday on Ellen DeGeneres' talk show. Letterman quit network TV in 2015 after 33 years as a late-night host on CBS and NBC, and is beginning his second season on his more leisurely paced Netflix show.
"Here's the mistake I made. I stayed on television way too long," Letterman told DeGeneres, to which she replied, "That's not true."
"Yes, it is true," Letterman responded. "It turns out nobody had the guts to fire me.
"You want to make sure you have some energy to direct toward other things," he added.
Letterman, still in his bushy post-retirement beard, said that you want to make sure you have enough energy for other things in life and, while his talk show was on, he didn't.
"I had to realize, 'Oh, I've been looking through the wrong end of the telescope,'" he said. "There's more to life than, 'Tell us about your pet beaver.'"
DeGeneres quipped: "Who was that guest?"
"Martha Stewart," Letterman joked.
The conversation then pivoted to talk show hosts' favorite topic of conversation: President Trump.
DeGeneres asked Letterman what he would ask Trump in an interview.
"I'm disappointed in his administration," he stated. "I would say to him, 'Don, why are you such a putz?'"
While Letterman was happy to discuss politics, his fellow former veteran late-night host, Jay Leno, recently complained about late-night hosts being overly political on the air.
Leno said on the "Today Show" last week that there's a perceived lack of “civility” from late-night hosts such as Stephen Colbert and Jimmy Kimmel amid the genre's heavily political climate.
“You know, everything now… everyone has to know your politics,” Leno said. “I tried to use Johnny’s model, and I would get hate mail from both sides equally and thought ‘well that’s fabulous, that’s exactly what I want’.”
“But when people see you as one-sided, it makes it tough. And, you know, I did it when Clinton was horny and Bush was dumb, and it was just a little easier."
Fox News' Liam Quinn and The Associated Press contributed to this report.