Duffy said his unborn daughter has a heart condition requiring his full attention, adding that he can't focus on her and his congressional responsibilities at the same time.
"I've had a great eight-and-a-half-year run... I love my job," he said. "But though I love politics, I love my family."
Duffy originally announced his decision Monday on Facebook and said it was the "honor" of his life serving the state of Wisconsin.
"After eight-and-a-half years, the time has come for me to focus more on the reason we fight these battles — family," he wrote.
"Not long ago we found out that our little baby has a couple of holes in the heart," Duffy told "Fox & Friends."
"So after she's born, whether it's two to six months... she's going to need open-heart surgery. They got to crack her open and fix the heart. They walked us through this yesterday. So I announced I was leaving Congress as I was walking in to meet with the heart doctor."
Campos-Duffy, a mother of nine, said the key to their family life is a balance between serving the people of America and properly tending to personal needs.
"Every congressional family... we all talk about work-life balance for our families. It's a really tough thing to do," she said. "It's tough if you have two kids. It's really rough with nine... I feel really proud of what Sean's done in eight-and-a-half years in Congress. I think the Republicans have saved American capitalism."
Duffy said he is going to step back from politics and regroup before he decides which path to take next.
"I don't know what I'm going to do," he said. "It's just that this time I needed to take a moment and make sure Rachel and I and this baby are healthy and doing well."
Democratic Gov. Tony Evers will call a special election to fill the seat and is currently exploring options for when it will be held, said the governor's spokeswoman, Melissa Baldauff.
Fox News' Alex Pappas contributed to this report.