From individual investors to small businesses to doctor's cooperatives — and even to the town itself — Fairfield, Conn., gave its money to Bernard Madoff. And now it's left with a pile of old statements telling investors how much money they never really had.

Madoff, the former chairman of Nasdaq, was arrested by federal officials last week on charges he orchestrated an elaborate Ponzi scheme, defrauding investors of up to $50 billion and leaving a trail of financial devastation in his wake.

The Orthopaedic Specialty Group, a Fairfield surgeons' group with 130 doctors and employees, lost its retirement savings when Madoff's $50 billion Ponzi scheme came tumbling down. The group had invested its entire retirement fund with Madoff, according to a report Wednesday in the Connecticut Post.

Click here for photos.

"For some of us, it's an entire career's worth of work," Dr. Robert Dawe told the Post. Some of the group's doctors had been paying into the retirement fund for 40 years, he said, and were now left worrying about the future.

"I'm not an expert on finance," Dawe told the newspaper. "I'm an expert in orthopedic surgery."

Dawe declined to say how much money the group had invested with Madoff, but he told the Post that Madoff had been recommended by a trusted source and many experts had called him the "best of the best."

The group met with Madoff twice a year and would be shown documents saying the group's money was being invested in "blue-chip" securities, he said.

Public employees also have lost a small fortune. The Town of Fairfield lost millions from its pension fund, first selectman Kenneth Flatto told FOXNews.com.

The town had invested $22 million in Madoff's fund over three years in the 1990s and had received statements and annual audits saying it had grown to $42 million. "The town controller was getting monthly statements and the annual audit reports showing the breakdown and everything. That's why it's just hard to believe," Flatto said.

He said Fairfield's pension board, which met Monday, is exploring strategies to recoup the money. He said the town itself should weather the storm.

"It's hard to say much more than that at this point," Flatto said. "We have been very diversified and this has been a hit for us. Thankfully, we have a lot of the money we need for our retirees."

He said $233 million remains in the fund, and only 14 percent of the pension money had been invested with Madoff. "We were overfunded and were doing well compared to a lot of other governments and the nation," Flatto said.

While the town has suffered a loss, some of its residents may not be able to recover as gracefully.

"I personally have talked to or know of eight or 10 situations where families or businesses unfortunately put most of their eggs in one basket, and put most of their funds or savings into this investment scheme. They're really hurting," Flatto said.

"It's hard enough for us being a pension fund, but to see people losing everything is even harder."

He said the whole town was taken in by investment advisers and businesses that told them Madoff's securities were safe, conservative investments.

"Fairfield County residents and investment advisers knew these people personally and they believed in them," he said. "They pushed these funds very heavily because I hope they believed they were good."

Fairfield Fire Chief Richard Felner said the entire town is reeling.

"Every day something different is in the paper," he said. "But now we have to see where the chips fall and find out more about it. They're saying we can recoup some of the money. It's crazy. It’s a circus. But are we concerned. Of course we are."

Felner said he had read about the Orthopaedic Specialty Group in the newspaper on Wednesday morning. "It's just a shame. I just can't believe someone would do something like that and live with themselves," he said. "This was their whole life savings."