Pressure is mounting on International Monetary Fund chief Dominique Strauss-Kahn to step down after his weekend arrest on charges of sexual assault against a hotel maid.

Strauss-Kahn, who is fighting sexual assault charges behind bars, has shown no signs of caving to calls for him to resign, but getting rid of the powerful global agency's managing director may be out of his control.

According to the agency's Articles of Agreement, "the managing director shall cease to hold office when the executive board so decides."

The 24-member board could call a a vote to oust him.

Indeed, the IMF's daily operations won't be affected by Strauss-Kahn's arrest. Deputy John Lipsky has taken over as interim director and the executive board can still approve loan packages for countries needing assistance. And it's expected to authorize rescue loans to Portugal as part of a larger package that European finance ministers negotiated Monday.

But with a debt crisis rocking Europe right now, one former IMF official thinks it's only a matter of time before the executive board drops the ax on Strauss-Kahn.

"I think it's a matter of weeks than months," said Morris Goldstein, a senior fellow at the Peterson Institute for International Economics and former deputy director of the IMF research department.

Goldstein told FoxNews.com that if there isn't forthcoming information soon exonerating him, "my guess is he will come under pressure" to step down. But he added there aren't many examples to offer guidance.

"We're sort of in uncharted territory," he said.

The IMF, which tries to provide global financial stability by providing emergency loans to countries unable to pay their debts, has not said much about Strauss-Kahn's future at the agency since he was arrested. On Monday, the agency said it would not comment on the case and that the IMF "remains fully functioning and operational."

It released another statement later after an IMF board meeting saying "the IMF and its executive board will continue to monitor developments."

The agency did not respond to a request for further comment.

No removal is likely to happen without the United States' consent. It is the largest contributor to the IMF among its 187 members and by far has the most weighted vote in the Washington-based body.

On Tuesday, White House chief spokesman Jay Carney said the administration is confident that the IMF can continue to function and fulfill its role. But Carney refused to comment on whether Strauss-Kahn should resign.

Strauss-Kahn has been in hot water with the IMF board before. In 2008, he was cleared of abuse of power charges for his affair with a female staffer but was warned by the board that such behavior wouldn't be tolerated in the future.

And it's possible that the board has already been considering replacements since it was widely expected before his arrest that Strauss-Kahn would step down within months and launch a campaign for president of France.

Austria's finance minister was the first one to suggest Tuesday that Strauss-Kahn consider resigning to avoid damaging the IMF.

"Considering the situation, that bail was denied, he has to figure out for himself that he is hurting the institution," Maria Fekter said at her arrival at a meeting of European finance ministers in Brussels.

The Spanish finance minister, Elana Salgago, voiced support for the alleged rape victim.

She said Strauss-Kahn is facing "very serious accusations," but added that resignation "is only a decision that Mr. Strauss-Kahn can take."

Other European officials have expressed more support for French Socialist Party leader.

"I'm very sad and upset. And he's a good friend of mine," Luxembourg Prime Minister Jean-Claude Juncker said Monday night.

"I didn't like the pictures I've seen on television," Juncker added, referring to Strauss-Kahn's perpwalk outside a New York precinct house.

The Associated Press contributed to this report.