The 63-year-old said called DeGeneres "one of the coolest and kindest people I've met in this business" during an appearance on PEOPLE (the TV Show!) Thursday.
He detailed how long he's known DeGeneres, noting that she's "the person that I've known for a number of years, that I saw back in the comedy club days, when we used to split time with the same management."
In July, several current and former staff members claimed the show fostered a toxic workplace culture. In interviews with Buzzfeed they detailed examples of sexual harassment and racially insensitive comments.
DeGeneres also faced accusations of rude behavior herself. The show has since experienced a massive shakeup after an internal investigation was launched. Executive producers Ed Glavin and Kevin Leman and co-executive producer Jonathan Norman "parted ways with" the show.
Harvey reasoned that "people want to take something and make something out of everything," he said. "[Ellen] has no control over what every staff member does, what every producer does, you can't control that. You don't even know some of the stuff that's happening."
But he agreed that because "it's her name that's on the show" she had to "take the hit." Harvey said he would do the same on his own talk show.
The "Family Feud" host added how he doesn't believe DeGeneres, 62, has a "racist tendency."
"I'm a 63-year-old Black man. I've been Black the entire time. I ain't took no days off. I ain't experimented with nothing else. I ain't got nothing else going. I've been a Black dude my whole life. I know racism," he described.
"If I have a conversation with you, I know if you got racist tendency somewhere in you, because it radiates. I can feel it. I've been around a long time, man. That's not Ellen DeGeneres. That's just not her, period," he reiterated.
Harvey said when the news first broke, he texted the Emmy winner to "keep your head up" and thinks that she'll make a strong comeback in the fall when the show's new season premieres.
"I think Ellen's going to be absolutely fine myself. I really do," he said.
Fox News' Nate Day contributed to this report.