It only takes one state senator to stand up and do his job to balance the budget and avoid a $165 million loss in savings to the state, Wisconsin Gov. Scott Walker said Sunday, challenging any of the 14 Democratic senators who fled the state more than a week ago to return to vote on a budget.

Democrats "need to come back and do what they were elected to do. They don't have to vote for it, they don't have to support it," Walker said on NBC's "Meet the Press." 

Walker is trying to close a $3.6 billion budget gap in his state for the coming two years in part by cutting $1 billion to local governments. That means cutting health benefits and pensions to the state's unionized workers. 

But while state worker unions have agreed to increase their contributions to their pensions and health benefits to 5 percent and 12.8 percent, respectively, Walker said the one-time fix won't do much good if big labor can just resort to collective bargaining in the next budget and drive up state expenses again.

"We want to be unique in Wisconsin to give those local government the tools beyond the 5 and 12 percent," he said. 

Saying defecting lawmakers must make hard choices, even if it won't turn out their way, Walker said without a budget, Wisconsin's debt will increase and state workers will lose their jobs.

"I would go to almost any ends to avoid that and my hope is at least one of those 14 senators would feel the same way," he said. 

But AFL-CIO President Richard Trumka said Walker keeps changing the markers in his argument.  

"This isn't about the budget crisis. Let's look at how his arguments migrated," Trumka said on the same show. "The members out there said, 'We'll accept their cuts.' He said, 'No, we won't accept accepting our cuts.' The most outrageous thing he did is (say) ... you have to accept the loss of your rights or we're going to lay you off."

Trumka argued that public employees are taking responsibility for their contributions to the debt, but to say collective bargaining is the chief driver of budget deficits is wrong since the five states in the country that ban collective bargaining have a "collective debt right now of $222 billion. So this is not about economics."

State budgets are a chief concern facing governors, who were in Washington, D.C., for the weekend at the National Governors Association winter meeting. Mississippi Gov. Haley Barbour said Walker is doing what every governor must do to get a hold of his deficit. 

Sitting down and saying, 'Let's make a very narrow agreement about wages for one year,' isn't going to solve the problem," Barbour said. 

"It's about time somebody stood up and told the truth about the state," said Walker, who did no attend the NGA conference. "I make no apology for the fact that this is an historic moment in time."