Letterman's contract reportedly expires early next year.
"I used to think I'd make some big deal about walking away and all that," Letterman said. "Now I'll just wait until they fire me."
On having a baby:
"I wasn't prepared for the fact he'd be so amusing," Letterman said of his new son, Harry. "Before he was born, I kind of tiptoed to the edge a little bit. But when he showed up, everything changed."
So why have a baby now? Stern asked.
"I just felt like we [he and girlfriend Regina Lasko] were pretty much in the bottom of the ninth and if we don't do it now, when are we going to do it?" Letterman said. "I'm 56 - I have no business having a kid."
Stern then asked Letterman why he hasn't yet married Lasko.
"I had one [marriage] years ago, and I suppose one day it'll happen again," Letterman said.
Stern, who turned 50 yesterday, and Letterman also spent some time commiserating about their respective ages.
"I'm freaking out that I'm 50," Stern said.
"When you round the corner at 55 and you're staring at 60, that gets your attention," Letterman said. "I always thought that when I got to be 40, I couldn't be on television.
"But here I am still cashing the checks," he said. "It's ridiculous."