Former Acting Attorney General Matthew Whitaker defended federal agents on Saturday by pushing back on the idea that their presence inflamed the unrest in Portland, Ore.

He was speaking with Fox News host Neil Cavuto, who said federal agents' presence was "only a more recent appearance."

"I mean, as I said, we've had 64 nights of protests in Portland. The arrival of federal forces, agents, whatever you want to call them, was only in the last week or two, so is that a fair criticism?" Cavuto asked.

"I don't think it was a fair criticism," Whitaker responded. "I think you had some violent extremists that were agitating the law enforcement that were trying to protect this courthouse. Let’s remember, this is a courthouse that we all paid for as taxpayers and it should not be destroyed. And it should not be defaced or anything. So, it seems very appropriate to establish a perimeter."

PORTLAND SEES LARGELY PEACEFUL PROTESTS WITH MORE THAN 1,000 DEMONSTRATORS AS FORCES WITHDRAW FROM CITY

He added: "I don’t think that the federal law enforcement, many who live and work in that community in Portland, would do anything to agitate these protesters whose ultimately became rioters ... I think cooler heads have prevailed. I hope that by the local politicians engaging in this situation, allowing law enforcement from the state and the city to be involved in the protection of the courthouse, I hope that does de-escalate the situation because no one wants to see that and certainly no one in Portland wants to see that in downtown and their own community."

Whitaker's comments came just days after Oregon's governor announced the Department of Homeland Security (DHS) would pull out of the city while state police took their place guarding the courthouse. As of midnight on Saturday, no federal agents had emerged from the courthouse, which has been the center of protests for weeks, and there was no noticeable law enforcement presence surrounding the area.

Friday’s overnight protest mimicked that of Thursday, which was the first time in weeks that demonstrations ended without any major confrontations, violence or arrests, according to the Associated Press.

Both Portland Mayor Ted Wheeler and Oregon Gov. Kate Brown tweeted media stories arguing that peace had returned because of federal officers' departure.

PORTLAND MAN CHARGED WITH ARSON AFTER ALLEGEDLY SETTING FIRE TO DOWNTOWN COURTHOUSE

DHS said on Friday that Oregon was finally "stepping up" and "cooperating" with federal authorities. Acting DHS Secretary Chad Wolf also warned that his agents wouldn't leave the area until the department felt assured the Courthouse and other federal properties wouldn't come under attack.

CLICK HERE TO GET THE FOX NEWS APP

"For 2 months as rioters attacked federal officers & buildings in Portland, we called on local leaders to do their job & work with us to quell violence. Finally OR is stepping up. DHS surge forces will stay until we know our officers & facilities are safe," Wolf tweeted.

The Associated Press contributed to this report.