Former President Carter got in a shouting match Wednesday with Sudanese security services who blocked him from a town in Darfur where he was trying to meet with refugees from the ongoing conflict.

The 83-year-old Carter walked into this highly volatile pro-Sudanese government town to meet refugees too frightened to attend a scheduled meeting at a nearby compound. He was able to make it to a school where he met with one tribal representative and was preparing to go further into the town when Sudanese security officers stopped him.

"You can't go. It's not on the program!" the local security chief, who only gave his first name as Omar, yelled at Carter, who is in Darfur as part of a delegation of respected international figures known as "The Elders."

"We're going to anyway!" an angry Carter retorted as a crowd began to gather. "You don't have the power to stop me."

U.N. officials told Carter's entourage the Sudanese state police could bar his way. Carter's traveling companions, billionaire businessman Richard Branson and Graca Machel, the wife of former South African President Nelson Mandela, tried to ease his frustration and his Secret Service detail urged him to get into a car and leave.

"I'll tell President Bashir about this," Carter said, referring to Sudanese President Omar al-Bashir.

Carter later agreed to a compromise by which tribal representatives would be brought to him at another location later Wednesday. But the refugee delegates never showed up.

The Darfur conflict began when ethnic African rebels took up arms against the Arab-dominated Sudanese government, accusing it of decades of neglect. Sudan's government is accused of retaliating by unleashing a militia of Arab nomads known as the janjaweed — a charge it denies. More than 200,000 people have been killed and 2.5 million displaced in four years of violence.

Most refugees appeared too frightened to speak in Kabkabiya, a North Darfur town that has long been a stronghold of the pro-government janjaweed militia.

Branson said some refugees had slipped notes in his pockets. "We (are) still suffering from the war as our girls are being raped on a daily basis," read one of the notes, translated from Arabic, that Branson handed to The Associated Press.

The note said that on Sept. 26, a group of girls had been raped, and a refugee had also been shot two days ago. Branson said it had been handed over by an ethnic African man.

The visit by "The Elders," which is headed by Nobel Peace laureates Carter and Desmond Tutu, is largely a symbolic move by a host of respected figures to push all sides to make peace. Tutu visited a refugee camp in south Darfur, but the U.N. Mission in Sudan deemed it too dangerous for Carter to make a similar visit.

Carter instead flew to a World Food Program compound in Kabkabiya, where he was supposed to meet with refugees, many of whom were chased from their homes by pro-government janjaweed and Sudanese government forces.

But as the meeting was set to get under way, none of the nongovernment refugee representatives arrived, and Carter decided to walk out into the town to try to talk with them.

"We are in the security field. We're not that flexible," said the security chief, Omar, after the confrontation ended. He said Carter already breached security once by walking to the school and would not be allowed to breach security again.

"This illustrates the challenges that communities and humanitarian workers face in Darfur," said Orla Clinton, spokeswoman for the U.N. Mission in Sudan who witnessed the incident.

Carter later returned to the North Darfur capital of El Fasher and where he was planning to meet with community representatives later Wednesday.

"The Elders" delegation is trying to use their influence at a crucial time — with peace talks due to start in Libya and the deployment of a 26,000-strong hybrid African Union-U.N. peacekeeping force to begin later this month.

Tensions are running high after rebels overran an AU peacekeeping base in northern Darfur over the weekend, killing 10 in the deadliest attack on the beleaguered force since it arrived in the region three years ago.

Carter said he felt the trip was proving effective. He said al-Bashir told him this week that Sudan has committed $100 million to a fund for Darfur's reconstruction and another $200 million has been pledged by Chinese diplomatic allies.

Carter said the main goal of the three-day visit to Sudan was to seek guarantees for free and fair elections throughout the country in 2009. Observes fear the elections could be postponed and warn this would imperil the fragile peace in southern Sudan and worsen the conflict in Darfur.

The 2009 vote would be the first democratic election in Sudan since al-Bashir came to power in a military and Islamist coup in 1989. Carter said al-Bashir vowed to allow the election to take place during a private meeting between the two in Khartoum.

"If the CPA fails to fulfill its commitment to free and fair elections and democracy in this country, all other efforts will be futile," Carter said, referring to the 2005 Comprehensive Peace Agreement that ended 21 years of civil war between the government and Christian and animist rebels in the south.