The U.N. refugee chief who resigned over allegations he sexually harassed a female coworker and several other women stepped down from his post on Thursday, rather than waiting until a replacement was found.

Ruud Lubbers (search) announced by letter Thursday to all U.N. High Commissioner for Refugees (search) staff that he was leaving his job immediately, even though he's maintained that the allegations against him are "made up" and "slander."

Originally, Lubbers and Secretary-General Kofi Annan (search), who are friends, said Lubbers would stay on until a replacement was found. Lubbers said Thursday his decision was in response to "an unprecedented open letter of concern and dismay at the lack of due process" sent to Annan "by a large number of UNHCR staff all over the world."

"We must now turn the page," Lubbers wrote in the letter Thursday. "A new High Commissioner has to be appointed as soon as possible. Therefore, in close consultation with the Secretary-General, I have decided to leave the Office today ..."

Speaking earlier this week about replacing Lubbers, U.N. spokesman Fred Eckhard said Annan told him Tuesday that "he expects that this process will be transparent and rapid."

"I can't predict a specific date, but he'll be moving quickly," Eckhard said.

Traditionally, the top refugee post goes to a leading figure from a country that is a major contributor to the agency.

Allegations surfaced last year that Lubbers had made unwanted sexual advances toward a female employee, identified in the media as an American.

But it was only last Friday that the British newspaper The Independent published the first detailed description of the woman's allegations and statements from four other women who didn't file official complaints but claimed Lubbers sexually harassed them.

With the United Nations (search) struggling to improve its image in the face of scandals over the U.N. Oil-for-Food program in Iraq and sexual abuse by U.N. peacekeepers in Congo, diplomats said Annan decided Lubbers had to go.

U.N. diplomats saw Lubbers' departure as part of a major effort to revamp the top U.N. leadership led by the secretary-general's new chief of staff, Mark Malloch Brown, whose primary job is to improve U.N. performance and overhaul its management.

Annan said Lubbers' resignation was in the best interests of UNHCR, its staff and the refugees it serves.

But UNHCR department heads wrote Annan expressing regret at Lubbers' resignation, saying that he should be remembered for "tireless determination" to help refugees and for creating an environment in which staff could express their views, "however divergent."

"We hope that all this, rather than the leaked investigation report of a complaint which you found could not be sustained, will be properly appreciated as the legacy of his time as high commissioner," they wrote in the letter, a copy of which was obtained by The Associated Press in Geneva.

Lubbers has volunteered his time and paid his own travel and expenses at the agency for the past four years.

FOX News' Catherine Donaldson-Evans, Jonathan Hunt and The Associated Press contributed to this report.