But in an election year that's been anything but predictable, Obey unexpectedly announced his retirement in May, saying he was "bone-tired," and Duffy, 38, now finds himself the would-be frontrunner in a race against Julie Lassa, a Democratic state senator, in Wisconsin's 7th Congressional District.
Duffy said even skeptics are starting to believe.
"I could've listened to those who said, 'Why you? You're a county D.A. and you can't make a difference.' And I go... You know what? I'm a father of six. I do lumberjack sports. I'm a D.A. Can I make a difference? I thought I could."
Thirteen years ago, Duffy was on MTV's "The Real World." He met his wife, Rachel Campos-Duffy, while they were taping a spin-off series (Campos-Duffy was on another season of "The Real World.")
But Duffy's interested in more than just the MTV audience's burning questions about boxers or briefs. For the past eight years, he has served as top prosecutor in Ashland, a small city in northern Wisconsin. And political watchers say he's a serious contender for the congressional seat.
"In a year like this, which is bad for Democrats, he has a legitimate shot at winning. In fact, I think he's probably going to win," said Kenneth Mayer, a political science professor at the University of Wisconsin-Madison.
A We Ask America poll released Aug. 4 shows Duffy leading Lassa 42 percent to 33 percent in a hypothetical general election match-up.
Duffy first has to get through a Sept. 14 primary. With some comparing him to Massachusetts Sen. Scott Brown, primary opponent Dan Mielke says Duffy's reality show past proves he's not a true conservative.
If Duffy gets the nomination and goes up against Lassa, it's not going to be an easy fight.
"I'm connecting with voters, hearing the concerns they have and the hopes and dreams they have for themselves," Lassa said.
Lassa has already had a visit from House Majority Leader Steny Hoyer, D-Md., who said Democrats in Washington plan to invest in this race.
Indeed, last week the Democratic Congressional Campaign Committee launched an attack ad against Duffy.
"Politicians like Sean Duffy, just don't get it. When we should be fighting to protect Social Security, Sean Duffy backed a plan to privatize it," the ad says.
Duffy says that's not true.
"We have to make sure that those who have bargained for their Social Security in their retirement that they get exactly what they bargained for," he said.
In this campaign, he's not taking anything for granted.
"You know what? I'm from a small town in northern Wisconsin," he said. "I live (for) my wife and my kids and this campaign. And Time Magazine doesn't vote for me. The New York Times doesn't. The Wall Street Journal doesn't. People here in Wisconsin do. And that's why I'm out every day."
Fox News' Steve Brown contributed to this report.