Consumer confidence soured in May, as Americans fretted about jobs and the overall economy, a private research group said Tuesday.

The Conference Board said its consumer confidence index fell to 103.2, down from the revised 109.8 in April. Still, May's reading was better than the 100.9 expected by analysts.

Click here to visit FOXBusiness.com's Economy page.

The decline stalled a rebound seen since November in the aftermath of last year's Gulf of Mexico hurricanes, except for a sharp dip in February when short-lived pessimism over the job market hurt consumer sentiment.

"Consumer confidence, which reached a four-year high in April, lost ground in May," said Lynn Franco, director of the New York-based Conference Board Consumer Research Center. "Apprehension about the short-term outlook for the economy, the labor market and consumers' earning potential has driven the Expectations Index down to levels not seen since the aftermath of the hurricanes last summer."

The Expectations Index, which measures consumers' outlook over the next six months, fell to 83.7 in May, from 92.3 in April. The Present Situation Index, which measures how shoppers feel now about the economic conditions, slipped to 132.5 from 136.2.

Tuesday's report is disappointing for retailers, which have seen sales slow in May amid cooler temperatures. Economists closely monitor consumer confidence because consumer spending accounts for two thirds of all U.S. economic activity.

While shoppers have remained resilient in the face of higher gasoline prices, which have been hovering around $3 per gallon, Tuesday's report may confirm analysts' fears that consumers have reached a tipping point.

According to a poll of 7,000 consumers conducted for the National Retail Federation by research firm BIGresearch LLC in early May, 76 percent believe fluctuating gas prices have affected their spending habits. That's up from 67.2 percent a year ago.

Job growth has been solid, but shoppers are struggling with big economic challenges, from higher inflation to rising interest rates and a cooling housing market.

In a worrisome report issued last week by the Federal Reserve, core inflation, which excludes food and energy, rose 2.1 percent in April, the biggest gain in 13 months

That's making economists nervous that high increases in oil prices are now expanding into other areas of the economy. And the Fed, which has been on an interest-hike campaign over the past two years, is being confronted with the challenge of keeping inflation in check without slowing the economy further.

Conflicting economic data is also making shoppers feel more uncertain about their future, said Karl Bjornson, retail strategist at Kurt Salmon Associates.

"We are in this conflicting, choppy water place, and consumers are beginning to be concerned," said Bjornson. He cited fluctuating gasoline prices and mixed messages about the housing market. Recent reports show the housing market has exhibited a split personality, with some hot markets in Florida, California and Arizona slowing down while some laggards are picking up momentum.

Bjornson added he is already seeing consumers becoming more selective about how they spend their money, cutting back on eating out at restaurants and choosing to go to stores that offer better value. He believes that this summer, the heavy driving season, will be the telling factor of where consumer spending is headed for the fall and holiday seasons.

The Conference Board index was derived from responses received through May 23 to a survey mailed to 5,000 households in a consumer research panel. The figures released Tuesday include responses from at least 2,500 households.

Consumers' overall assessment of current conditions eased but remains upbeat. Consumers saying conditions are "good" slipped to 28.0 percent from 29.7 percent. Those claiming conditions are "bad" edged up to 15.4 percent from 15.1 percent. Labor market conditions were also less favorable. Those saying jobs are "plentiful" declined to 28.6 percent from 29.4 percent, while those claiming jobs are "hard to get" increased to 20.5 percent from 19.7 percent.

Consumers' outlook for the next six months, which improved moderately in April, soured in May. Those expecting business conditions to worsen increased to 13.2 percent from 9.3 percent. Those expecting business conditions to improve decreased to 16.5 percent from 17.3 percent. The outlook for the labor market was also less upbeat. Those anticipating more jobs to become available in the coming months declined to 14.6 percent from 15.4 percent in April. Those anticipating fewer jobs rose to 18.2 percent from 16.3 percent.

Click here to visit FOXBusiness.com's Economy page.