When the despots of the world roll into New York on Wednesday for the annual meeting of the U.N. General Assembly, American taxpayers will pay a hefty price to ensure their safety.

The cost of keeping the dictators safe? At least $20 million.

That's how much New York City forked out during the first two weeks of last year's General Assembly, according to a city official who spoke to FOX News on condition of anonymity.

And $20 million may be just the tip of the iceberg. The Bureau of Diplomatic Security, the Secret Service, the U.S. Marshal’s Service and the New York State Police will also be called in to provide protective details for several of the 138 heads of state expected to attend.

It adds up to a pricetag that doesn't sit well with some critics of the world forum.

"It's an outrage that the U.S. has to admit a man like Ahmadinejad, who will use the U.N. microphone to promote genocide," said Anne Bayefsky, senior fellow at the Hudson Institute, referring to Iranian President Mahmoud Ahmadinejad.

"The U.S. taxpayer pays 22 percent of the U.N.'s regular budget, which includes 20 percent of what is costs to allow [Ahmadinejad] to use the U.N. to spew hatred, anti-semitism and repeated calls for the annihilation of the Jewish state," Bayefsky said.

But other experts say the cost of security is necessary.

"That is an inevitable cost of having the U.N. here. It's an obligation that we freely assume," John Bolton, former U.S. ambassador to the U.N. under President George W. Bush, told FOXNews.com.

"The U.S. has to provide security," said Hillel Neuer, executive director of U.N. Watch, a human rights watchdog group based in Geneva. "Either you host the U.N. and you pay for safety, or you don’t host the U.N. It's one or the other.

"It is also a logical consequence that if you host the U.N., it's the U.N. that decides who gets to come — not the host country," he said.

Ahmadinejad is not the only world leader who will be under heavy protection while in New York. Venezuelan President Hugo Chavez, Libyan leader Muammar al-Qaddafi and Palestinian President Mahmoud Abbas will also be in attendance.

The cost of protecting them includes radiation and explosives sweeps, patrols of the East River alongside U.N. headquarters, and heavy security at the high-end hotels where the leaders are staying.

Ahmadinejad is staying at the Intercontinental Barclay, a hotel the Iranians settled on after being turned down by several others. Qaddafi, who failed to find a hotel that would house him and was refused permission to pitch a tent in Central Park, will reportedly stay at the Libyan Mission a few hundred yards from the U.N.

The money the NYPD pays for security is eventually reimbursed by the State Department, said Deputy Commissioner of Public Information Paul Browne. He added that State is currently behind in payments.

The 64th annual session of the U.N. General Assembly was formally opened last week by its new president, former Libyan foreign minister Ali Triki, but foreign heads of state are not expected to arrive until Wednesday, when general debate begins.

President Obama, who will be chairing a session on arms control and nonproliferation issues, will address the assembly on Wednesday morning — directly followed by Qaddafi. Ahmadinejad is scheduled to leave Iran on Tuesday and speak at the U.N. Wednesday afternoon.

The Iranian president's attendance will mark his fifth visit to the General Assembly; it will be Qaddafi's first trip to the U.S. since taking power 40 years ago.

In 2008, the Bureau of Diplomatic Security provided security for 176 visiting delegations — including 37 protective details for foreign officials attending the assembly. The security measures were implemented based on a threat analysis for each delegation, according to a State Department source.

The U.S. spends $500 million each year for general operations at the U.N. and $15 million annually on U.N.-related security. Most — if not all — of that $15 million is spent during the General Assembly, NYPD spokeswoman Barbara Chen told FOX News.

Some — like Bolton — say the economic benefit of hosting the U.N. far outweighs the cost of keeping world leaders safe.

"On an economic basis, I think that more than makes up for the security costs," he said, adding that if the U.N. was headquartered in another country, it wouldn't get the same level of scrutiny that it gets in New York.

Neuer said the U.S. has a responsibility to provide security, but it must make clear its opposition to the despots' tyrannical regimes. He is expected to protest the leaders' arrivals in New York, along with Ahmad Batebi — an Iranian torture victim who escaped to asylum in Washington.

"The American government must make it abundantly clear that America stands with the victims and not with the murderers," Neuer said.

FOX News' Jonathan Hunt contributed to this report.