There they stood, all four living U.S. presidents at the White House, smiling and posing with the soon-to-be newest member of the club, Barack Obama. But at that historic gathering earlier this month, one member of the group, Jimmy Carter, appeared to be cut off from the rest, as if he had crashed the party but could stay if he didn't cause any trouble.
"It was fascinating," said Stephen Hess, a senior fellow at the Brookings Institution, who said the photo opportunity showed the others as clubby while Carter was a step apart. And there's probably a good reason for that too, Hess said.
"He's a person who has stuck his thumb in the eye of every president who has followed him," he said.
At times, ever since he left the White House in defeat 28 years ago, Carter's freewheeling, freelance diplomacy has put him squarely at odds with his successors. While other ex-presidents have rode off into the sunset to enjoy, for the most part, quiet retirements, Carter has stayed busy writing best-selling books, conducting his own foreign policy and winning the Nobel Peace Prize in 2002 for his work.
His supporters hail him as a hero and his critics deride him as self-righteous and egotistical. But in recent years, Carter has lost support even among his defenders for comparing Israeli treatment of Palestinians to South African apartheid in a book he wrote -- and for meeting with Hamas leaders to discuss a peace deal.
"I think in my view, he's jumped the shark," said David Greenberg, a presidential historian with Rutgers University. "He passed the point of no return when he wrote the book."
Greenberg said he believes Carter has morphed from an angry and resentful ex-president after losing to Ronald Reagan to a self-righteous man who has damaged his reputation with his acts.
"Now a lot of people don't know or follow it closely," he said. "They see a man who does conflict resolution and Habitat for Humanity and admire that part. And that is part of the record.
"But when you look at the whole record, with the Middle East and the meddling with other presidents' foreign policy, I think he has, one, behaved badly and, two, discredited himself in a deep way."
Yet Carter's foreign diplomacy has been effective. Last year, his talks with Hamas leaders helped engineer a six-month cease-fire until rocket attacks resumed last month.
That's why some presidential historians don't believe Carter is the odd man out in the president's club.
"I don't see him as a pariah," said James Pfiffner, a professor of public policy at George Mason University,.
"He seems to stand out as somebody who did not go out to get rich and put personal time in things like Habitat for Humanity," he said. "So I think the contrast is most former presidents fly around the world and are hosted in lavish ways. And Carter is on the other end of the spectrum and wants to appear as doing good works."
Pfiffner says conservatives may not view Carter favorably. "But liberals and Democrats will see him in a kinder light," he said.
But Carter has locked horns with Democrats, too. Clinton was said to resent Carter's unsanctioned diplomacy during his presidency. And Carter didn't hold back when he criticized Clinton during the Monica Lewinsky scandal.
Greenberg believes Carter was jealous of Clinton's popularity.
"Jimmy wanted to be the success story of the new South," Greenberg said. "Bill Clinton, for all his problems, turned out to be that. Clinton was everything Carter wanted to be and failed as president."
So when Carter undermined Clinton, Greenberg added, "it wasn't just a partisan thing. He couldn't credit any of his successors. They all had to be inferior in his eyes."
But Greenberg believes Carter may be more cooperative with Obama.
"Obama to his credit, embodies hopes of redemption for a lot of people," he said, explaining that Carter sees Obama's election as progress on race relations, an issue he failed to move the ball on as president. "And for Jimmy Carter, Obama embodies hope of redemption."
It's unclear if Obama will reach out to Carter on foreign affairs. They met this month before Obama's inauguration to talk about the Middle East. Neither disclosed what was said. As president-elect, Obama deferred to Bush on foreign policy, insisting there can only be one president at a time.
"Jimmy Carter apparently doesn't believe in that mantra," Hess said.