Convicted Ponzi schemer Bernie Madoff, in his first interview for publication since his arrest in 2008, is pointing fingers at the banks and financial institutions he did business with, saying they were somehow “complicit” in his crimes.

“They had to know,” Madoff told The New York Times. “But the attitude was sort of, ‘If you’re doing something wrong, we don’t want to know.' ”

This bold deflection has some of Madoff’s victims fuming.

Allan Goldstein, a retiree from Spencertown who lost his  pension in the scandal, said he wasn't surprised to hear that the Securities and Exchange Commission did little to stop Madoff even though Madoff said that financial institutions and hedge funds "had to know" of his fraud.

"When I put my money in there... I met with Mr. Madoff and he was asking me a lot of questions. And I said, 'Wait a minute, this is my money, shouldn't I be asking you the questions?' And he said, 'You can check me out on the SEC.  I have a clean bill of health.' And we did check it out with SEC and he did have a clean bill of health."

Jack Cutter, who lives in Colorado, was another person who lost his retirement fund and life savings of more than $1 million in Madoff’s pyramid Ponzi scheme. He was forced to take a retail job, downsize his home and car and live a more frugal lifestyle.

“We are still trying to make a recovery," Cutter told Fox News. “We haven't had any luck getting our money back, but we’re still holding out hope that those who are helping us will be able to help us recover more.”

Cutter says Madoff, “doesn’t seem to have a nickel of remorse.”

“He seems to be pointing the finger at some of the funds and financial institutions saying that they knew or should have known, suggesting that it was their fault, not his. He seems to be pretty aloof about the whole thing.

“As far as those of us who were downstream from this and lost money, he could care less,” Cutter says. "His attitude is you should have known what was going on and if you didn't to hell with you.”

Madoff told The Times he’s cooperating with the court-appointed trustee who’s trying to recover more than $20 billion in assets for fraud victims, claiming he met directly with that trustee, Irving Picard, in the North Carolina prison where he’s serving a 150-year sentence.

But perhaps not surprisingly, the trustee says the convicted liar and cheat is still making up stories.

In a statement obtained by Fox News, Picard’s chief counsel David Sheehan writes, “The New York Times reports today that Bernie Madoff met directly with Irving H. Picard, the trustee for the liquidation of Bernard L. Madoff Investment Securities LLC. This is inaccurate. At no time did any meeting between the two take place and there has been no direct communication between them at any time.”

The Times also reports Madoff says he’s not cooperating with federal prosecutors working on criminal cases of Madoff Associates, and that while he’s thinner and frail, he remains defiant.

Cutter says Madoff still has it too good.

“I was really rankled that he lived in a fairly elite place, the best kind of prison facility," he said. "You know, you want him to be suffering just like the rest of us.”