After David Letterman broke up with his sexy young assistant and she found a new boyfriend, the late-night TV talk king couldn't stand being around the guy, sources revealed yesterday.

And that grudge may have driven dissed beau Robert "Joe" Halderman's alleged plot to extort $2 million from Letterman by threatening to expose his sexcapades with "Late Show" underlings.

SLIDESHOW: Celebrity Affairs Exposed.

Halderman started dating Letterman's personal assistant, Stephanie Birkitt, 34, after she split with her boss — and Dave made it clear he didn't want her new 51-year-old producer boyfriend at any "Late Show" office parties and other staff events, the sources said.

He wanted Stephanie there, not Halderman — he didn't like him," one source said.

FOX411: Halderman a 'Nightmare,' Letterman 'Big Flirt.'

Years later, Halderman, an Emmy-winning producer for CBS's "48 Hours Mystery," allegedly engineered the extortion scheme.

Halderman and Letterman had Birkitt in common — but little else, sources said.

Halderman liked to live well, golf and party hard, one source noted.

RELATED: Halderman Lawyer Calls Letterman 'Manipulator.'

Letterman, on the other hand, has a well-honed reputation as private, reserved and geeky.

Even the cultures of the two men's workplaces were worlds apart, the CBS insider noted.

At "Late Show," the boss — who in March married longtime girlfriend Regina Lasko, his son's mother — might have been playing around, but the female staffers were happy with the work environment, the insider insisted.

"I would be shocked if someone came forward" with a sexual-harassment complaint, this insider said.

"I've worked in a lot of places, and ['Late Show'] is one of the better places for a woman. Dave's not a groper."