Sen. Judd Gregg's withdrawal Thursday as commerce secretary nominee cast a spotlight on President Obama's controversial plan to share oversight of next year's census and stirred speculation among political observers and congressional aides about the impact it will have on the potential change.

A spokesman for Rep. Darrell Issa, R-Calif., one of several House Republicans calling for Obama to scuttle the census plan, said one of two things will happen.

Obama "realizes the role this played in the withdrawal of Sen. Gregg and abandons his plan to politicize the census," Kurt Bardella told FOXNews.com.

"Or two, I think it's entirely possible, depending on who's the next nominee, if the Congressional Black Caucus doesn't have objections, everything will stay as it was."

Under Obama's census plan, the director of the U.S. Census Bureau, who has yet to be named, would report to White House senior management in addition to the Commerce Department, which oversees the bureau.

Click here for our video coverage about President Obama's Census plan.

The outcome of the census has deep political implications, since congressional districts are drawn based on population. Many federal funds are distributed on the basis of population, as well.

The first whiff of greater White House interest in shaping the census came when members of the Congressional Black Caucus complained to the Obama administration about Gregg's nomination.

Gregg once voted for a broader budget measure that would have abolished the agency, and he opposed increased funding for the 2000 census. Gregg's record raised concerns about his commitment to an accurate census count, a priority for minority groups that historically have been undercounted.

Click here for our video coverage about Judd Gregg.

An unnamed White House official was reported to have told members of the Congressional Black Caucus, including the group's chairwoman, Barbara Lee, that Gregg would not be left in charge of the census.

The White House issued a statement late last week reassuring that the census director would "work closely with White House senior management."

That in turn sparked an uproar from Republicans, who accused the White House of injecting partisan politics into the census and seeking to cut out agency professionals in favor of political operatives.

The White House issued a statement Wednesday, emphasizing Obama's commitment to a "complete and accurate count through a process that is free from politicization" even while seeking to explain that no real change was being made to the census director's chain of command.

But on Thursday, Gregg cited differences with the Obama administration over the economic stimulus bill and the census for ending his bid as commerce secretary, raising questions about which factor played a bigger role in his decision.

Gregg said the census plan was only a "slight catalyzing issue. It was not a major issue."

But some say that doesn't add up.

"I think the census had a larger role," Bardella said, noting that Gregg knew about the stimulus bill before he accepted the position.

"The only thing that changed between his nomination and his withdrawal was the assertion that [Obama officials] were going to bypass the secretary of Commerce" on the census, he said, adding that he doesn't know of "any nominee who would accept a political position that circumvented their authority."

The census plan also underscored a broader issue of how much authority Gregg would have as commerce secretary every time he had a disagreement with the Obama administration.

Bardella suggested that Gregg asked himself, "Is this setting a precedent?"

Sen. John McCain, R-Ariz., Obama's presidential rival, told FOX News on Friday he wasn't sure why Gregg made his decision but said he believed Obama's plan would politicize the Census Bureau.

"It is a huge impact on the American political scene and apparently they moved it to the White House from the Commerce Department where it's been forever," he said. "That's really bad."

Some think Obama may find it more difficult to move forward with his plan to move the census into the purview of the Oval Office.

David Webb, a GOP consultant and former decennial manager at the U.S. Census Bureau, said he believes Obama's selection of Gregg was a smokescreen to allow him to push his census plan ahead.

"I looked at it as a political play from the minute it was started," he told FOXNews.com, explaining that groups that lobbied against his selection gave Obama a reason to share oversight of the census. "With him out, it makes it harder to the make the move," he said.