The second anniversary of Superstorm Sandy arrived Wednesday in a region where recovery in New Jersey and New York is happening unevenly, with many houses, boardwalks and businesses rebuilt but many people still unable to return to their homes.

New Jersey Gov. Chris Christie spoke to a crowd in Belmar, N.J., a town damaged by the storm, defending his administration’s handling of the rebuilding. Christie repeatedly was interrupted by a heckler -- who happens to be a former Asbury Park city councilman and local activist -- which prompted him to tell the man, "Sit down and shut up."

The heckler, Jim Keady, harassed Christie about the pace of storm aid. Keady was holding a sign that read, “Finish the Job,” The Star-Ledger reported. He also yelled at the governor, “Do your job!”

Christie continued to challenge Keady.

“I'll be more than happy to have a debate with you anytime you like, guy, because somebody like you doesn’t know a damn thing about what you're talking about except to stand up and show off when the cameras are here,” Christie said. “I've been here when the cameras aren't here, buddy, and done the work. ... Turn around, get your fifteen minutes of fame, and then, maybe, take your jacket off, roll up your sleeves, and do something for the people of this state."

Keady runs an advocacy group called Finish the Job, which has been critical of the pace of rebuilding in the area. Once Keady tried to explain how he was here a month after Sandy, the governor interrupted him.

" ... and there's been 23 months since then, when all you've been doing is flapping your mouth and not doing anything. So listen, you want to have the conversation later? I'm happy to have it, buddy. But until that time, sit down and shut up."

After a few minutes more of dialogue, the argument between the two was over.

"It seems like a lot longer than two years," Christie said after the incident was over. "This has been a long, long two years and a long struggle. Time doesn't move as quickly as we might like it to."

Officials in both states visited houses and businesses badly damaged by the storm to meet with victims still rebuilding and promised to keep working until the recovery is complete.

U.S. Housing and Urban Development Secretary Julian Castro and federal officials toured a flood-ravaged neighborhood near Raritan Bay in Union Beach where many residents are struggling to rebuild.

Andrea Kassimatis held her 6-month-old daughter as she described living with four other relatives in a 37-foot trailer next to a partially built home.

"It's been a rough and grueling process," she said. "You feel like your government has forgotten you."

Kassimatis received a $150,000 rebuilding grant from New Jersey but only got a third of what her flood insurance policy was supposed to pay — a common refrain up and down the coast.

"Don't believe what you have from a flood insurance policy," she warned. "Because what you're sold is not what you're going to get."

She voiced her complaints to Castro, to New Jersey's two U.S. senators, Robert Menendez and Cory Booker, and to Rep. Frank Pallone Jr., all of whom said more "accountability and transparency" is needed in Sandy aid distribution.

The Oct. 29, 2012, storm, which was spawned when Hurricane Sandy merged with two other weather systems, devastated the coast and caused catastrophic flooding in New York and cities in New Jersey. It was blamed for at least 182 deaths and $65 billion in damage in the U.S. It's New Jersey's worst natural disaster.

New York Mayor Bill de Blasio joined City Council Speaker Melissa Mark-Viverito and other elected officials to work with Habitat for Humanity at a storm-damaged home in Brooklyn. The group has helped rebuild 100 homes in New York. He also toured a Staten Island neighborhood named Ocean Breeze and recounted the destruction.

"This borough, this island, in many ways bore the brunt of the storm: 44 lives lost due to Sandy, 23 from Staten Island," he said. "A lot of pain, and a lot of memory. We are now safer than we were two years ago, that is a matter of fact. But we have a lot more to do. I can say with assurance, when we gather a year from now, we will be safer than we were in 2014."

New York Gov. Andrew Cuomo said his state has supported the repair and rebuilding of nearly 10,000 households, provided $20.8 million in grants to small businesses and facilitated the proposal of approximately 600 projects through the NY Rising Community Reconstruction Program.

The Associated Press contributed to this report.

For more on this story visit NJ.com.