West Virginia Sen. Robert Byrd isn't going gently into any good night, despite reports that 15 Senate Democrats have been talking privately about deposing the Senate Appropriations Committee chairman.

Byrd, 90, has been in and out of the hospital in recent weeks and missed a series of votes before the Spring recess.

That has led, according to Roll Call newspaper, to several senators holding "the first of what's likely to be several private discussions" about whether Byrd can handle his duties as the head of the most powerful committee on Capitol Hill — especially at a time when annual appropriations get into full gear.

But a spokesman for Byrd, who missed Tuesday morning's Senate Armed Services Committee hearing with Gen. David Petraeus and Amb. Ryan Crocker on progress in Iraq, said the appropriations process is in no way suffering from the lawmaker's absences.

"Once again, it appears that Washington insiders are practicing what they do best — petty rumor mongering," Byrd Communications Director Jesse Jacobs told FOX News. "The fact is the 12 individual Senate Appropriations subcommittees are moving forward reviewing the budgets under their jurisdictions. That is the job they are tasked to do. And when it is time, the Senate Appropriations Committee will move forward with the supplemental appropriations bill."

The newspaper reported Tuesday that some suggested that Byrd might be asked to relinquish the gavel and continue in an emeritus role.

"They are struggling to try and figure out how to deal with the situation," said one senior-level Democrat who did not want to be named. He added that senators had raised "serious concerns about his ability to handle his appropriations work, especially with the upcoming hearings on the supplemental."

Asked his opinion, Senate Majority Leader Harry Reid said Tuesday, "I do support keeping Senator Byrd on as chairman."

Byrd is currently serving his ninth six-year term in the U.S. Senate. While he did not attend the morning hearing, he was on Capitol Hill and monitored it and the afternoon Senate Foreign Relations Committee hearing, Jacobs said.

"He has been watching both hearings ... in his Capitol office. And he voted today and met with staff," Jacobs said.

Jacobs said Byrd is still convalescing as a result of a recent fall but "is continuing his steady recovery through physical therapy" and "remains focused on the job at hand. Any suggestions that he is not fulfilling his duties as a United States senator are baseless and untrue."

Click here to read the Roll Call report (subscription required)

FOX News' Trish Turner contributed to this report.