Roger Ailes saw me coming through the door to his office and with his needling wit he said: “We can’t have you working here.”

My jaw dropped. Roger knew how to get your attention with a cutting comment in good times and bad. I was going through a very bad time.

This was late October of 2010 and the night before I’d been fired by NPR. Later, I was labeled a bigot and in need of psychiatric care by the head of a major, national media operation, the then-president of NPR (National Public Radio).

Ailes, who died Thursday at 77, started laughing as he watched my reaction. Then the executives in the room went from stone faced to laughing along with the boss. Ailes told me to sit down and said he was offering me a new contract and increased role with the network.

In Ailes fashion, he cut to the emotional power and pain of the moment for me. He asked me what my wife and kids had to say about the public beating I was getting. Then he asked how much I was paid by NPR. He said he would make up the money so I would not have to go home and explain why we had lost money because of NPR’s action.

Also true to Roger’s style, he said he was watching to see how the left-wing media handled the story of a “serious journalist” being silenced for speaking his mind.

I met Ailes while I was a White House correspondent for The Washington Post and he did some work for President Reagan’s political director, Ed Rollins. And later we talked when he was in George H. W. Bush’s 1988 campaign. He was a good source for a black kid with a big Afro who some conservatives found hard to take.

He was a tough political fighter with a disdain for self-righteous liberals and he was willing to push emotional buttons among voters -- from fear to rage to needling humor that often came so close to the edge it made his friends nervous and scared his opponents.

In 1984, he mocked one of Mitch McConnell’s opponents in a Kentucky Senate race with an ad featuring a hound dog searching for the politician with the spotty record of showing up for votes. He signed off on a 1988 ad that led to national laughter at the Democrat’s presidential nominee, Michael Dukakis, for riding around in a tank with a helmet on as if he was a little boy pretending to be General MacArthur.

That sharp edge and his immense curiosity about people – he knew the family story, the gossip, the education, the weaknesses – and his willingness to pull those levers made him a power player. He elected presidents, senators and created major media personalities as well as the dominant cable news channel of his time.

It also had a damning consequence. He lacked restraint. It led to his downfall and worst of all it hurt many people. As a black person and a civil rights historian, sometimes the buttons Roger allowed to be pushed on race saddened and alarmed me.

His reckless lack of restraint also brought scandal to the institution he so carefully built and that remains his legacy.

Ailes personally recruited me to come to Fox News as he was creating the network in 1996. I signed my first contract in 1997, after he convinced my wife that it was OK for me to leave what was then the top brand in cable news: CNN.

I was once sitting in his office, filled with television monitors with the sound off, listening to his comments about what was on the screen. He snickered about poor production that made for boring television. He wanted intense debate, good looking people and surprise twists and turns in conversation and facial expression that drew attention to the screen.

One of his assistants once told me that Ailes’ secret was he saw the world through the eyes of a three-year-old boy. Nothing was too complicated. It was either good or bad, interesting or boring, an idea or person he wanted or he didn’t.

This cut across political lines. Roger’s top aide at Fox and friend for many years was Chet Collier, a Democrat.

They differed on politics but they agreed on putting on a good show that drew eyeballs to the television. Collier saw a kindred spirit in Ailes as a 22-year-old when he hired him as a production assistant on the Mike Douglas Show.

Collier told the story of Ailes advising candidate Richard Nixon, as a guest on the Douglas show, that television had to be a top tier concern for any politician. Nixon later hired Ailes. And he would tell that story but also note – with his distinctive belly laugh - that Nixon was constantly backing away from the snake handlers and famous stripper also scheduled for the show that day.

Loyalty mattered greatly to Ailes because he knew low moments. He once told me of being out of money and fellow Republicans turning their back on him. He decided that sad character was not him. He preferred going out with a fight and got back in the fight and became the major media executive of his time.

Ailes invited my two sons to have lunch with him one day in his private dining room at Fox. They wanted career advice. He talked about hard work, loyalty to friends and even more, loyalty to the truth and anger at hypocrisy – and being smart enough to see what is plain and obvious in the world and willing to fight for it. Never let the hypocrites win, he said. He was not focused on the fact that the boys are black and their father a Democrat.

“Juan has been a staunch defender of liberal viewpoints since his tenure began at Fox in 1997,” Ailes said in a statement after NPR fired me. “He is an honest man whose freedom of speech is protected by Fox News on a daily basis.”

It was protected by Roger Ailes.

Juan Williams currently serves as a co-host of FOX News Channel's (FNC) "The Five" (weekdays, 9-10PM/ET) and also appears as a political analyst on "FOX News Sunday with Chris Wallace" and "Special Report with Bret Baier."