Updated

A New York City panel voted unanimously Tuesday to reject landmark status for a building near the World Trade Center site, paving the way for construction of a mosque and an Islamic community center.

Opponents of the project, including 9/11 first-responders and family members of victims of the Sept. 11, 2001, terror attacks, have said the location would be insensitive.

The mosque is slated to be part of an Islamic community center to be operated by a group called the Cordoba Initiative, which says the center will be a space for moderate Muslim voices.

Several members of roughly 50 people who attended the hearing applauded the ruling, while others shouted "shame" as commission chairman Robert Tierney called for the vote. The city Landmarks Preservation Commission then proceeded to vote 9-0 against granting landmark status to the site's 152-year-old building, which can now be torn down to make way for the Islamic center.

One opponent, Linda Rivera, of Manhattan, held a sign reading, "Don't glorify murder of 3,000. No 9/11 victory mosque."

Supporters of the landmark status, including GOP gubernatorial candidate Rick Lazio and some Sept. 11 family members, had argued that the building warranted landmark status because it was struck by airplane debris during the attacks.

But commissioner Christopher Moore noted that the debris struck a number of buildings in the area.

"One cannot designate hundreds of building on that criteria alone," Moore said. "We do not landmark the sky."

The commission was asked to determine whether the building is architecturally important enough to preserve, not to consider the merits of the proposed mosque. Demolition and construction of the mosque can now proceed.

The move was applauded by the New York Civil Liberties Union and the American Civil Liberties Union, citing principles of religious freedom.

"We congratulate the Landmarks Preservation Commission for promoting our nation's core values and not letting bias get in the way of the rule of law," the groups said in a joint statement. "The free exercise of religion is one of America's most fundamental freedoms. For hundreds of years,our pluralism and tolerance have sustained and strengthened our nation. On 9/11, religious extremists opposed to that very pluralism killed 3,000 Americans. Those fanatics would want nothing more than for our nation to turn its back on the very ideals that make this country so great."

Oz Sultan, the program coordinator for the proposed Islamic center, said last week that the building has been changed too much over the years to qualify as a landmark.

"I think a lot of the negativity we're getting is coming from people who are politically grandstanding," Sultan said. "We're completely open and transparent."

Daisy Khan, executive director of the American Society for Muslim Advancement, told The Wall Street Journal in Tuesday's editions that the center's board will include members of other religions and explore including an interfaith chapel at the center.

"We want to repair the breach and be at the front and center to start the healing," said Khan, a partner in the building and the wife of the cleric leading the effort.

But Abraham Foxman, national director of the Anti-Defamation League, said Khan's proposals fail to address the crux of opponents' criticism that constructing the mosque near ground zero is insensitive to 9/11 victims' families.

Last week, the leading Jewish organization came out against the mosque. The ADL said "some legitimate questions have been raised" about the Cordoba Initiative's funding and possible ties with "groups whose ideologies stand in contradiction to our shared values."

Rick Bell, executive director of the New York chapter of the American Institute of Architects, said the building does not deserve landmark status.

"The nature of the current building isn't worth preserving," Bell said.

Mayor Michael Bloomberg supported the mosque's construction, but the project has drawn opposition from former U.S. House Speaker Newt Gingrich and former Alaska Gov. Sarah Palin, among others.

FOX News' Lauren Green, Jonathan Wachtel, Christopher Laible and The Associated Press contributed to this report.

Click here for more on this story from MyFOXNY.com.