Embattled Rep. Charlie Rangel, chairman of the Ways and Means Committee, said Wednesday he's not taking a primary challenge by a former aide for granted.

Former Rangel aide Vincent Morgan announced earlier this year he would run against his old boss in the New York primary. It's the first time the 20-term congressman has faced bona fide primary opposition in years.

"Any challenge is serious," the New York Democrat said of Morgan's candidacy. But Rangel conceded he "didn't know the extent of the challenge." He also said he hadn't "had time to think about" Morgan's run.

Last week, the House Ethics Committee voted unanimously to expand its investigation of Rangel for a third time since July, 2008. Rangel referred himself to the Ethics Committee last year, telling FOX News he was as "clean as the driven snow." But Rangel has faced a host of alleged ethical breaches which have caused some to question whether he's up to the job as chairman of the House's powerful tax-writing committee.

The Ethics Committee is currently probing Rangel's failure to pay taxes on a villa in the Dominican Republic; questions surrounding his ownership and use of rent-controlled apartments in Harlem; and the use of congressional stationary to try to raise money for a school of public affairs at City College of New York. The school of public affairs bears Rangel's name. But the congressman did not name the school himself.

For his part, Rangel has also paid back taxes that he owed.

House Democrats last week shelved a Republican-engineered effort to strip Rangel of his chairmanship of the Ways and Means Committee until the ethics panel finishes its inquiry.

As chairman, Rangel is a key player in the massive health care reform effort now moving through Congress. And some lawmakers have begun whispering that the stress of the investigation, coupled with the legislative lift on health care, may have worn down the 79-year-old Rangel.

Rangel has visibly lost weight over the past few months. And a few lawmakers have pointed that fact out to reporters as an example of the toll could be exacting on the congressman.

When asked why fellow lawmakers were commenting on his weight loss, Rangel smiled and responded with a single-word quip.

"Envy!" he chortled.