The imam in the middle of the Ground Zero mosque controversy said Sunday he would never have picked that location if he knew it would create the conflict it did, but he has no plan to move the Islamic center from the proposed site two blocks from where the World Trade Center fell.

"I would never have done it. I'm a man of peace. I mean, the whole -- the whole objective of peace work is not to do something that would provoke controversy," said Imam Feisel Abdul Rauf.

But the final decision about where the Park51 community center and mosque will be constructed "will be predicated on what is best for everybody," Rauf said.

Rauf, who was interviewed for ABC's "This Week," also accused "certain politicians" of using the project for their "political ambitions." He pointed to Sarah Palin's Tweet asking for him to reconsider the location as "disingenuous, to a certain extent."

"The fact of the matter is, A, this has been used for political purposes. And there's growing Islamophobia in this country. How else would you describe the fact that mosques around the country are now being attacked?" he asked.

Rauf said the debate over the location of the mosque "has been, to a certain extent, hijacked by the radicals. The radicals on both sides." He repeated his claim that to move the proposed mosque now would gin up Muslim radicals and leave the Muslim world with the impression that "Islam is under attack in America."

"This will put our people -- our soldiers, our troops, our embassies, our citizens -- under attack in the Muslim world, and we have expanded and given and fueled terrorism," Rauf said.

That argument didn't win much credence with former New York City Mayor Rudy Giuliani, who was in office on Sept. 11, 2001, and was even trapped for a short time in a building a block from the proposed mosque site.

"I think that tactic is not the kind of tactic I would have expected from an imam who is featured as a man of conciliation," Giuliani said on NBC's "Meet the Press." 

The imam said "if he doesn't get his way, there could be significant and very dangerous violence. Those are very, very strong words and to enter a sort of suggestion of a threat into this, I worry about this as the kind of tactics he pursues," Giuliani said.

Rauf said he never made a threat and never would, but the movements and the discourse in the United States "are being watched very, very closely. And if we make the wrong move, it will only expand and strengthen the voice of the radicals and extremists."

In a recent poll taken by the Washington Post, 49 percent of respondents said they had an unfavorable view of Islam.

Rauf said he thinks that Muslims in the U.S. today now have a heightened concern about a spike of Islamophobia, "which is reaching and perhaps even possibly exceeding what happened right after 9/11."

But Giuliani said Islamophobia has nothing to do with moving the mosque, which he argued is on "sacred ground."

"The people he's hurting here most are the families that have lost loved ones down there. They don't all feel that way but 80 percent or 90 percent feel extremely hurt by this and it's making them relive the pain. They should be the ones to get the most consideration. Not the imam, not me, not the president, not the mayor. They're the ones that are the most affected by this," he said.