The mayor and a top state lawmaker on Friday called for a meeting to resolve a power struggle at ground zero, with growing momentum to build fewer skyscrapers than planned in an uncertain economy.

Mayor Michael Bloomberg invited the governors of New York and New Jersey, Assembly Speaker Sheldon Silver, the agency that owns ground zero and trade center developer Larry Silverstein to a meeting at his mansion next week.

The developer and the Port Authority of New York and New Jersey are in stalled, bitter talks to renegotiate Silverstein's lease at the site for the second time since the Sept. 11, 2001, terrorist attacks.

Silverstein has rights to build three of five planned office towers and controls development on the eastern half of the site, where the Port Authority is building a $3.2 billion transit hub. The Port Authority is building a 1,776-foot tower and has rights to build a fifth tower.

Silverstein has asked the agency to back financing for all three of his towers. The agency has agreed to back at least one.

The standoff threatens to delay already pushed-back deadlines for all of Silverstein's towers, the hub and the Sept. 11 memorial.

Silver on Friday called the delays "an embarrassment to our city, our state and to the nation."

"For whatever reason, the dynamic has broken down again. We can not afford to halt negotiations, to continuously alter and re-alter our plans," he added.

Silver said he backed construction of at least two of Silverstein's towers, in addition to the 1,776-foot One World Trade Center tower being built by the Port Authority.

Gov. David Paterson and the Port Authority issued separate statements appearing to back more modest rebuilding plans.

"We should not repeat the mistakes of the past where unrealistic expectations get in the way of a practical path," Paterson said.

A spokeswoman for New Jersey Gov. Jon S. Corzine said Friday saying he encourages any move to speed up construction.

"The governor welcomes the efforts to expedite the completion of the memorial and the transit hub, and to ensure the accountability of the port authority with regard to the site," spokeswoman Deborah Howlett said.

Port Authority spokesman Stephen Sigmund said, "The Port Authority must be realistic about putting its limited public resources toward the most public benefit, which means keeping the memorial and the public transportation projects moving forward, and building the office and retail development to meet the market."

Janno Lieber, who oversees trade center development for Silverstein, said the entire complex should be rebuilt as quickly as possible.

The five towers, part of a master plan designed in 2003, would replace the 10 million square feet of office space destroyed on Sept. 11, 2001. Officials have touted the rebuilding as a sign of the city's defiance of terrorism and an investment in downtown's financial district.

But the economic downturn and the collapse of major financial institutions has made the office space less necessary, business leaders have said. Developers like Silverstein also have found it impossible to obtain financing for new projects in a tight credit market, forcing him to ask the government agency to guarantee backing.

Silverstein and the Port Authority have split billions in Sept. 11 insurance money to pay for rebuilding, but Silverstein has about $1 billion left, not enough to build three major skyscrapers.

The agency hopes to open most of the memorial by the 10th anniversary of the attacks, its tallest tower in 2013 and the transit hub in 2014. Dates for the rest of the site's projects are uncertain.

Bloomberg invited the developer, the agency, Silver and the governors who control the Port Authority for the meeting next week. The city does not own ground zero but has taken an active role in past negotiations and has agreed to lease half the space in a Silverstein tower.

The mayor said the failing economy shouldn't become an excuse for stalled rebuilding.

"This is precisely the time we need to reaffirm our commitments to the future of our city," he said, "No development is more important than the World Trade Center site."