Short-lived White House communications director Anthony Scaramucci said Monday that if it were up to him, top adviser Steve Bannon would be gone from President Trump's administration.

But, he notes, "it's not up to me."

"The Mooch," a few weeks removed from his spectacular flameout following an expletive-laden conversation with a reporter, appeared Monday on CBS' "Late Show" with Stephen Colbert. Colbert has seen his ratings soar since Trump's inauguration with his relentless comedic attacks.

Scaramucci suggested in an earlier interview that Bannon is the source of at least some of the West Wing leaks and that his connection to Breitbart.com and its association with the far right is hurting the presidency.

Scaramucci Reuters

White House Communications Director Anthony Scaramucci accompanies U.S. President Donald Trump for an event about his proposed U.S. government effort against the street gang Mara Salvatrucha, or MS-13, with a gathering of federal, state and local law enforcement officials in Brentwood, New York, U.S. July 28, 2017. REUTERS/Jonathan Ernst - RTX3DC5F (REUTERS)

Trump gave Scaramucci the White House job in late July, in part for how he deftly forced CNN to take down an incorrect online story connecting him to the Russia investigations.

However, he was fired 11 days later, after a profanity-laced phone call with The New Yorker reporter Ryan Lizza and amid the arrival of retired Marine Gen. John Kelly as the new White House chief of staff.

“For the record, I thought that (call) was off the record,” Scaramucci, a former Wall Street financier and member of the Trump campaign’s finance committee, said Sunday. “That was a very deceitful thing that he did. … But we don't need to debate that anymore. That's past news. I made a mistake. I'm accountable for the mistake. I paid the consequences of that.”

Colbert showed a picture of Scaramucci and former White House Chief of Staff Reince Priebus glaring at each other. Scaramucci said there was "no love lost" between the two.

He said he and Priebus got along well when he was writing checks to the Republican National Committee, which Priebus once led.

Scaramucci said he knows Trump "as a compassionate person," while reiterating that he thought the president should have spoken more harshly than he did initially of the white supremacists involved in the violent protest in Charlottesville, Virginia.

Of Trump's frequent off-the-cuff remarks, Scaramucci said, "That's him wearing his heart on his sleeve."

The Associated Press contributed to this report