Updated

A United Nations envoy is urging restraint amid a tense standoff on the front lines of Colombia's conflict between security forces and farmers who expelled police allegedly under pressure from leftist rebels.

Hundreds of riot police were deployed Saturday to the rural hamlet of El Mango in southwestern Colombia four days after a mob burned down a makeshift barracks and demanded that a small police contingent leave.

Authorities blamed the expulsion on civilians coerced by the Revolutionary Armed Forces of Colombia. Both sides in Colombia's long-running conflict have been stepping up attacks in recent weeks, jeopardizing the future of peace talks. Human rights groups warn that more confrontations putting civilian lives at risk are around the corner after the rebel group known as the FARC lifted a unilateral cease-fire last month.

Todd Howland, representative in Colombia for the U.N. high commissioner for human rights, said residents are being unfairly stigmatized as FARC supporters.

"People feel they receive very little benefit because police in these conflict zones don't actually do police work," Howland told The Associated Press. "They're sandbagged in. It's like having a military base in town."

People in the town said the police, bending to the popular outrage, relocated to a bluff overlooking El Mango as thousands of people from the town and neighboring villages marched peacefully to demand that they not be made human shields if fighting breaks out.

But Gen. Rodolfo Palomino, the head of Colombia's police, insisted Sunday that security forces have returned for good, saying in a statement it was the government's constitutional obligation "to assure and maintain the peace, order and territorial integrity."

With residents threatening to abandon the town if the police do not withdraw, it is not clear how the impasse will be resolved. Negotiators from Colombia's ombudsman's office and the Organization of American States are in El Mango trying to defuse the tension.

Authorities initially blamed the expulsion on FARC rebels dressed as civilians and the ombudsman, whose job it is to speak up for vulnerable communities, harshly criticized residents for kicking out the police.

Residents of El Mango said they lived peacefully for years, but have suffered numerous attacks from the FARC since security forces became entrenched there in 2007 as part of a U.S.-backed offensive to beat back the rebels. Dozens of families have left out fear of getting caught in the crossfire and their homes have been occupied by police.

The violence has fallen sharply since the government and the rebels started peace talks two years ago in Havana.

"We don't want the war to return," said Dagoberto Munoz, president of the local community action board, told the AP. "We don't want more victims in the town."