The consequences of allowing protesters and rioters to call the shots in big cities like Chicago and Portland, Ore., might not always be predictable, Dr. Ben Carson said Friday during an appearance on "Hannity."

"The people should recognize that when you invite anarchists into your neighborhood and you join with them, there's going to be long-lasting destruction that you're going to have to deal with your children and everybody else is going to have to deal with," Carson, who serves as President Trump's secretary of Housing and Urban Development (HUD) said.

During the interview with guest host Jason Chaffetz, Carson addressed multiple topics, including violence in American cities. He also defended the Trump administration's decision to revoke an Obama-era rule purportedly aimed at diversifying the suburbs.

CHICAGO POLICE TRYING TO COMBAT VIOLENCE LIMITED BY POLITICS-DRIVEN POLICY: FORMER CHIEF

Carson noted one thing that demands and critiques from rioters and protesters have in common.

"They blame Donald Trump for everything," he said.

"They blame Donald Trump for everything."

— Dr. Ben Carson

Instead, local leaders should be held responsible, the secretary said. He urged voters to always carefully evaluate who they voting for -- in order to prevent future problems.

"There is a contract, an unwritten contract, between the local government and the people. Local government is supposed to protect the people, they're supposed protect the businesses and their property. And when they don't do that, they are shirking their duties," Carson said. "And there should be consequences for that. And some people have started talking about that already.

"I appeal to voters," he added. "Look at the people that you are putting in office. Evaluate what they are getting done. Are they accomplishing things or are they not?" Carson said. "We have to be much more intelligent than that or we get what we deserve."

CLICK HERE TO GET THE FOX NEWS APP

Carson also said the revoking of the Obama-era housing regulation affecting the suburbs was a good move.

"I've already started to hear people saying, well, the Trump administration is racist and they're backing away from civil rights and fair housing. What a bunch of garbage," Carson said, defending the administration's record. "So we don't need another bunch of layers of bureaucracy to tell people how they're supposed to live, where they're supposed to live."

The Associated Press contributed to this story.