Paul Ryan is one of those rare politicians who really does want to spend more time with his family. But there's more to his decision to relinquish power than that.

His frustration in trying to work with President Trump, despite their clashing agendas, was in my view a significant factor in yesterday's announcement that he isn't running for reelection, essentially abdicating as House speaker.

Ryan never wanted the job, as he made clear to reporters. He was drafted by his colleagues. And with little prospect of being able to achieve more of his agenda, Ryan decided to declare victory after helping to pass the tax cuts and stop being a "weekend dad" back in Janesville, Wis.

The pundits raced to read a whole lot more into his decision.

Ryan was "signaling the peril that the Republican majority faces in the midterm elections," The New York Times says.

Ryan is "adding further uncertainty about whether embattled Republicans can maintain control of the House," says The Washington Post.

Now I'm sure the GOP's challenging prospects in the midterms crossed the congressman's mind, despite his demurral yesterday. But he’s not the typical power-hungry pol, so I don't think that's his main motivation. Nor do I believe Ryan's absence on the ballot will hurt other Republican candidates, except perhaps for the loss of his fundraising prowess.

CNBC's John Harwood goes a step further. "Ryan’s retirement from the House increases the chance that President Donald Trump will be impeached," he says.

Because "there's little doubt that a Democratic-controlled House would seek to impeach the president."

Wait, so Nancy Pelosi and company are lying when they say they’re not pursuing impeachment?

Harwood sees it as a runaway train. Still, a pretty sweeping conclusion based on one lawmaker giving up his seat.

It became awkward watching Ryan at his news conferences, deflecting reporters' questions about the latest Trump scandal or tweet and struggling to talk about policy.

What strikes me is how some Trump loyalists are trashing Ryan online, even calling him a RINO. That shows how much the party's center of gravity has shifted.

Ryan was long viewed as a dogged crusader for smaller government. When Mitt Romney picked him as his running mate, the right cheered at such a conservative choice.

But Ryan's devotion to entitlement reform was utterly at odds with Trump's vow to protect Medicare and Social Security. And the $1.3-trillion bill that Trump signed (after briefly threatening to veto it) was the epitome of the bloated, out-of-control federal spending that Ryan had vowed to stop.

Some commentators, stepping away from the horse race, are branding Ryan a failure:

"Who can blame him for not wanting to be the man standing between a volatile caucus, and a president whom he doesn't like, who doesn't respect him, and who doesn't care a whit for the kind of conservatism that has motivated Ryan throughout his long career in government?" The Atlantic asks.

"Faced with an out-of-control president of his own party, the speaker has decided he'd rather quit than deal with it"

And Politico points out that "Paul Ryan Deficit Hawk has never behaved like a deficit hawk. In his two decades in Washington, Ryan has consistently supported tax cuts and spending hikes that have boosted deficits, while consistently trashing Democrats for failing to cut deficits."

But National Review’s Jim Geraghty gives Ryan a respectful sendoff: "He was civil, well-informed, polite, and firm, the opposite of a table-pounding, demagogic extremist, and that probably just aggravated his critics on the left even more."

The larger question is whether anyone can run the fractious House GOP. Three years ago, John Boehner quit in frustration. Now Paul Ryan is quitting in frustration. The party's divide is so deep that speaker—or if they lose, minority leader—may be the ultimate thankless job.