Treasury Secretary Timothy Geithner is on "shaky grounds" these days with Congress and many in the country, according to the Senate Banking Committee's top Republican.

Sen. Richard Shelby said Sunday he doesn't think Geithner will last long unless he starts doing a better job.

Shelby, who voted to confirm Geithner, said he has less confidence in the treasury secretary each day -- over the AIG bonus mess, solving the nation's financial troubles and helping turn around the economy.

"I said competence brings confidence to anything. I don't see a lot of things positively thus far that Timothy Geithner's been involved in," Shelby said Sunday on "Fox News Sunday."

"He's going to have to do a 180-degree turnaround, I believe, to be a successful treasury secretary," he said.

But President Barack Obama has faith in Geithner. The president tells CBS' "60 Minutes" that if Geithner offered to resign, the answer would be, "Sorry buddy, you've still got the job."

While Shelby did not specifically call for Geithner to resign, at least two Republicans -- Reps. Connie Mack of Florida and Darrell Issa of California -- did.

White House economic adviser Christina Romer said those calls are "silly."

"I think all of this discussion is -- is really silly. Tim Geithner is an excellent secretary of the treasury. He has been dealt an unbelievably difficult hand to deal with. But he's actually doing a fantastic job," Romer said on "FOX News Sunday."