Investors struggled Friday to understand how a former Nasdaq stock market chairman could hide losses of at least $50 billion in what he allegedly called "a giant Ponzi scheme."

Bernard L. Madoff, 70, was arrested on a securities fraud charge Thursday and released on $10 million bail after federal authorities said he ran a fraudulent investment business separate from his Bernard L. Madoff Investment Securities LLC.

Outside those offices Friday, one man who said he had invested with Madoff said everyone who had put money into the firm "has got problems."

"I think at this point it’s all up in smoke," he told a gaggle of reporters. "That’s my personal feelings, but we’ll see."

No one who entered or exited the granite-columned office tower where the firm is housed during lunchtime said they worked for Madoff, and the lone investor who did speak on the condition of anonymity said he was told their offices were closed Friday.

FOXNews.com's calls to the firm were not answered.

Prosecutors allege Madoff ran a secret $17.1 billion investment-advising business that operated separately from his Manhattan securities firm, servicing between 11 and 25 clients.

Click here to read the complaint (FindLaw PDF).

Stephen A. Weiss, a lawyer for several dozen investors, said there are many more investors. He said he "personally spoke with over 30 investors" who are believed to have invested more than $1 billion with Madoff, who had been a well-respected member of the financial community, serving as the treasurer of the board of directors at Yeshiva University in Manhattan for several years.

"We are shocked at this revelation," university spokeswoman Hedy Shulman said in an e-mail to FOXNews.com. "Bernard Madoff has tendered his resignation from all positions and involvement with the university. Our lawyers and accounts are investigating all aspects of his relationship to Yeshiva University. We reserve our comments until we complete our investigation."

In a federal criminal complaint, Madoff is accused of "deceiving investors by operating a securities business in which he traded and lost investor money, and then paid certain investors purported returns on investment with the principal received from other, different investors."

Andrew M. Calamari, associate director of enforcement in the Securities and Exchange Commission's New York office, said the SEC had filed a civil securities fraud charge as well and was alleging "a stunning fraud that appears to be of epic proportions."

The SEC said it was seeking emergency relief for investors, including an asset freeze and the appointment of a receiver for the firm. A hearing was scheduled for Friday.

FBI Special Agent Theodore Cacioppi said in the complaint that a senior employee of the Madoff firm told him that Madoff had told him that "he was 'finished,' that he had 'absolutely nothing,' that 'it's all just one big lie,' and that it was 'basically, a giant Ponzi scheme.'"

Cacioppi said one of the employees told him that Madoff was "cryptic" about the firm's investment advisory business and kept its financial statements locked up. The FBI agent said another employee told him that Madoff last week said clients had asked for about $7 billion in redemptions and he was struggling to meet those obligations but thought he could do so.

Cacioppi said two senior Madoff employees told him that Madoff said during the Wednesday meeting that he planned to surrender to authorities in a week but first wanted to distribute $200 million to $300 million he had left to certain selected employees, family and friends.

Cacioppi said he and another FBI agent arrived Thursday at Madoff's apartment, where Madoff invited them in and acknowledged knowing why they were there.

"Madoff stated, in substance, that he had personally traded and lost money for institutional clients, and that it was all his fault," Cacioppi said.

The agent wrote that Madoff said he had "paid investors with money that wasn't there" and that he was broke and insolvent and had decided that "it could not go on" and that he expected to go to jail.

If the allegations contained in a criminal complaint are true, it may be the largest fraud ever blamed on a single individual and would be a steep fall for Madoff, a former Nasdaq stock market chairman who founded his business in 1960 with $5,000 he earned in part working as a lifeguard on Long Island beaches.

His firm was a market maker, handling trades in some of the largest securities on various stock exchanges, matching buyers and sellers.

Bernard L. Madoff Investment Securities LLC ranks among the top 1 percent of U.S. securities firms, according to the company's Web site.

In 2001, Barron's reported that Madoff's firm was one of the three top market makers in Nasdaq stocks and the third-largest firm matching buyers and sellers of securities on the New York Stock Exchange.

Shortly after leaving law school, Madoff founded his firm in 1960. It was one of five broker-dealers most closely involved in developing the Nasdaq Stock Market, where he served as a member of the board of governors in the 1980s and as chairman of the board of directors.

In the 1990s, Madoff was viewed as a maverick. He angered leaders of the New York and American stock exchanges by taking away some of their business by paying brokerage firms a penny a share to route orders through his system.

Defense lawyer Dan Horwitz called Madoff "a person of integrity" and said he intends to fight the charge.

If convicted, Madoff could face up to 20 years in prison and a maximum fine of $5 million.

FOXNews.com's Sara Bonisteel and the Associated Press contributed to this report.