Louisiana Democratic Rep. William Jefferson said Wednesday that he won’t submit to calls for his resignation from a House panel while he is under investigation in a federal bribery case.

House Minority Leader Nancy Pelosi, D-Calif., sent Jefferson a letter asking him to step down from his seat on the tax-writing House Ways and Means Committee.

"In the interest of upholding the high ethical standard of the House Democratic Caucus, I am writing to request your immediate resignation from the Ways and Means Committee," Pelosi wrote.

Jefferson responded in a written statement, saying he will not resign. Jefferson has not been indicted and has denied wrongdoing.

"None of the matters reported to be under scrutiny involve issues under jurisdiction of the Ways and Means committee. Therefore, such a request would be even more perplexing and unreasonable. If I agreed, it would unfairly punish the people of the 2nd District and I will not stand for that," Jefferson said.

"Further, such a request would be discriminatory, in as much as no other member currently under federal investigation has been asked to step down from a relevant committee assignment. Therefore, I will not give up a committee assignment that is so vital to New Orleans at this crucial time for any uncertain, long-term political strategy," he added.

Officials say Pelosi is working behind the scenes to get Jefferson off the committee. The letter is the latest attempt to urge Jefferson to step down until the legal situation is addressed.

Jefferson did attend a House Democratic Caucus meeting Wednesday morning, but the issue surrounding the latest controversy did not come up, said caucus chairman Jim Clyburn of South Carolina.

Meanwhile, lawmakers have engaged in a constitutional battle with federal officials, blasting a raid of Jefferson's congressional office last weekend.

FBI agents searched Jefferson's office in pursuit of evidence related to the bribery investigation. The search warrant, signed by U.S. District Court Judge Thomas Hogan, was based on an affidavit that said agents found $90,000 in cash wrapped and stashed in the freezer of Jefferson's home. The money is said to have the same serial numbers as the $100 bills given to Jefferson through an FBI informant last year.

House Speaker Dennis Hastert and other Republicans have come to Jefferson's defense, saying the FBI should surrender documents and other items that were seized during the raid.

"I think those materials ought to be returned," said House Speaker Dennis Hastert, adding that the FBI agents involved "ought to be frozen out of that [case] for the sake of the Constitution."

Hastert met with Pelosi on Wednesday and the two released a joint statement following their meeting.

"No person is above the law, neither the one being investigated nor those conducting the investigation," the statement reads.

"The Justice Department must immediately return the papers its unconstitutionally seized. Once that is done, Congressman Jefferson can and should fully cooperate with the Justice Department's efforts, consistent with his constitutional rights."

The Department of Justice responded to Hastert on Wednesday, saying federal officials must fully pursue investigations and take necessary steps to obtain evidence.

"As the attorney general said yesterday there is tremendous respect for Congress' important, independent role but the department has an obligation to the American people to fully pursue corruption cases wherever the trail of evidence goes," said Tasia Scolinos, a spokeswoman for the Justice Department.

"We are optimistic that continuing talks with the Congress can produce a result that meets law enforcement's needs and also allays any institutional concerns that Congress may have."

But Hastert and Pelosi say the seizure violates the Constitution.

"The Justice Department was wrong to seize records from Congressman Jefferson's office in violation of the constitutional principle of separation of powers, the Speech or Debate Clause of the Constitution, and the practice of the last 219 years," says the statement.

FOX News' Molly Hooper contributed to this report.