Justice Antonin Scalia (search) says he won't step down from judging a case facing the Supreme Court that involves Vice President Dick Cheney (search), a close friend.

Scalia wrote in a 21-page memo that "since I do not believe my impartiality can reasonably be questioned, I do not think it would be proper for me to recuse."

The Sierra Club (search), an environmental group that is suing Cheney over his withholding information about private meetings of an energy task force, argued that Scalia faces a conflict of interest in judging the pending suit since he and Cheney are hunting buddies.

Cheney and Scalia recently went duck hunting together. Scalia has maintained there was nothing improper about the trip he took with Cheney three weeks after the court agreed to consider the case.

"Even one unnecessary recusal impairs the functioning of the court," Scalia wrote in a 21-page memo.

Pressure on Scalia to stay out of the case had mounted in recent weeks, with calls from dozens of newspapers for the conservative Reagan administration appointee to recuse to protect the court's image of impartiality.

The Sierra Club responded to the decision with a statement that said: "We wish that Vice President Cheney would be as forthcoming with the details of the secret energy task force as Justice Scalia has now been with their vacation together."

Supreme Court (search) justices, unlike judges on other courts, decide for themselves if they have conflicts, and their decisions are final.

Judges must disqualify themselves from cases in which their impartiality "might reasonably be questioned." The Sierra Club asked for Scalia's recusal in February, pointing to the "American public's great concern about the continuing damage this affair is doing to the prestige and credibility of this court."

In his response, Scalia said he would recuse himself when "on the basis of established principles and practices, I have said or done something which requires this course."

He said the hunting trip to Louisiana was planned before the energy case reached the court.

Those "established principles and practices" do not require or even permit him to step aside in the Cheney case, Scalia wrote.

Fox News' Anna Stolley and The Associated Press contributed to this report.