Hundreds of people crowded outside the South Carolina Statehouse Saturday and taunted each other during separate rallies staged by groups from outside the state.

Black Educators for Justice, a Florida-based group, held its rally on the north side of the Statehouse, where the Confederate flag was removed earlier this month. Later, the North Carolina-based Loyal White Knights of the Ku Klux Klan held a rally on the opposite side of the building to protest the flag’s removal.

William Bader, the imperial wizard of the Kentucky-based Trinity White Knights of the Ku Klux Klan, said its members plan to take back America, according to The Post and Courier.

“They’re taking our heritage from us,” Bader said. He also said he wants to see the Confederate flag back on Statehouse grounds.

The South Carolina Department of Public Safety said about 2,000 people showed up for the rallies. Spokeswoman Sherri Iacobelli said five people were arrested and 23 people needed medical attention. A statement from Richland County Emergency Service said many of the 23 were treated for heat, but no specific number was given.

Several people carried the Confederate flag along the margin of the crowd at the black educators rally. At least 40 members of the KKK marched up the Capitol steps and waved flags. Many in the crowd jeered.

The Post and Courier reports Klansmen and sympathizers clashed with counter-protesters outside the barricades. Protesters screamed obscenities at the Klansmen leading to the slew of arrests. The tensions forced police to keep the demonstration to about an hour, the newspaper reported.

Two men who were involved in a scuffle were led away by police. During the rallies, Leroy Smith, director of the Department of Public safety, was seen helping a man wearing a T-shirt bearing a swastika get out of the heat.

Gov. Nikki Haley earlier this week urged residents to avoid the KK rally and in doing so would honor the nine people shot and killed at a predominantly black church in Charleston last month.

Click for more from The Post and Courier.

The Associated Press contributed to this report.