The plot to blow up two synagogues and shoot down a military plane in a holy war against Jews and the United States began with a conversation in a modest mosque in Newburgh, N.Y., federal prosecutors say.

The Hudson River city of 40,000 lies 65 miles north of New York City and is home to 67 houses of worship. One of them is a mosque — Masjid al-Ikhlas — which has been serving Newburgh's growing Muslim community since 1992.

It was there that at least two of the four alleged plotters were known to have attended prayers and services, according to local reports and the federal complaint that outlined the charges against them.

And it was there that James Cromitie, the 44-year-old ex-convict at the heart of their alleged yearlong plan to strike the U.S., first met the FBI informant who would bring about his arrest.

In June 2008, Cromitie met the FBI's confidential witness at Masjid al-Ikhlas, where the suspect explained that "he was upset about the war" in Afghanistan and "unhappy that many Muslim people were being killed in Afghanistan and Pakistan by the United States Military forces," according to the federal complaint filed Wednesday. Cromitie allegedly said he wanted to do "something to America" to strike back.

Click here for photos.

But civic and religious leaders in Newburgh say that embrace of violence goes against the teachings of the mosque and its imam, Salahuddin Muhammad, who preaches peace from his pulpit and constantly reaches out to other faiths.

"He is a man of peace," said the Rev. David Mason, who is head of the Greater Newburgh Ministerial Association and leads interfaith efforts in which Muhammad plays a large role.

Muhammad, who could not be reached for comment, has spoken out against terrorism and attacked the radicals who promote it during an interfaith ceremony last Thanksgiving, Mason told FOXNews.com.

"He said, 'There's a radical element in Islam that is contrary to what we preach and what we believe here in Newburgh.'"

The city was stunned Wednesday night when federal agents swooped in and arrested four Newburgh men for plotting to detonate powerful explosives outside two synagogues in the Bronx, N.Y., and conspiring to bring down military planes flying over Stewart Airport in Newburgh.

The four suspects — Cromitie, David Williams, Onta Williams, and Laguerre Payen — are all believed to have converted to Islam while in prison, according to federal officials. All are American citizens.

Their alleged crimes shouldn't be held against the members of the Muslim community in Newburgh, said Mayor Nicholas Valentine.

"The good people of a congregation should not be penalized for someone that's bad," Valentine told FOXNews.com. He said the arrests came as a shock to him.

The mosque has expanded in recent years from a community of just 50 families to a membership of about 500 people, according to local reports. But it's still a quiet place that remains empty most of the day, police officials said.

Valentine said the city was recovering and is moving on with its plans to hold a memorial ceremony on Friday to honor a Newburgh-born Muslim lieutenant in the U.S. Army who joined the military after the Sept. 11 terrorist attacks and was killed in action in Afghanistan last September.