Stocks ended mixed Tuesday after oil prices hit a new high, spooking investors already jittery about the effects of high energy prices on an economic recovery.

The Dow Jones industrial average ended lower 38.86 points, or 0.38 percent, to end the session at 10,177.68. The broader Standard & Poor's 500 index was down 0.69, or 0.06 percent, to end at 1,134.48. The technology-laced Nasdaq Composite Index gained 3.10 points, or 0.16 percent, to end at 1,955.50.

U.S. light crude prices, which have risen more than 55 percent this year, climbed to $51.09 a barrel on the New York Mercantile Exchange (search), surpassing last week's all-time record of $50.47. The rise is the result of a prolonged production outage in the Gulf of Mexico at a time when major exporters are already pumping nearly full tilt.

Rising oil prices have weighed on stocks as investors fear energy costs will pinch corporate profits and dampen consumer spending.

Adding to the sell-off was a disappointing report from the Institute for Supply Management (search), which said its service sector index fell to 56.7 in September from 58.2 in August. The reading was lower than the 59 expected by Wall Street. A figure over 50 represents expansion in the service sector, but the latest number represented a slowdown in growth.

On the employment front, a report from Challenger, Gray & Christmas Inc. said U.S. planned job cuts soared to an 8-month high in September while new hiring rose only slightly.

"Oil prices at fifty-one dollars a barrel are definitely weighing on the market except for energy stocks," said Robert S. Robbins, president of Robbins Capital.

Energy stocks, including Exxon Mobil (XOM), ConocoPhillips (COP) and ChevronTexaco (CVX), were among the gainers in the S&P 500. Exxon Mobil rose 40 cents, or almost 1 percent, to $49.32, while ConocoPhillips closed at a 52-week high, with its shares up $2.03, or 2.4 percent, to $86.70. ChevronTexaco hit a 52-week high during the session, but eased some to end up 59 cents, or 1.1 percent, at $54.45.

American International Group Inc. (AIG), the world's largest insurer by market value, dragged on the blue-chip Dow and the S&P 500 indexes. AIG is facing a possible lawsuit by U.S. securities regulators over helping a major banking client hide bad loans. AIG shares fell 3 percent to $66.50.

Chiron Corp. (CHIR) slid $7.44, or 16 percent, to $37.98 as the company slashed its 2004 earnings forecast after announcing it will not record any sales of its flu vaccine for the 2004-2005 flu season. Chiron said British regulators had suspended the company's license to manufacture the vaccine.

Medimmune (MEDI), which makes a new flu vaccine called FluMist (search) nasal spray, rose 5.8 percent to $25.78, helping the Nasdaq. FluMist is not approved for the old, the chronically ill, pregnant women and babies aged 6 months to 23 months.

Companies such as heavy equipment maker Caterpillar Inc. (CAT), aerospace company United Technologies Corp. (UTX) at $89.68 and diversified manufacturer 3M Co. (MMM), which are sensitive to swings in the economy, weighed on the Dow. Caterpillar fell 88 cents to $80.37, United Technologies was down 79 cents at $93.76, and 3M slipped 18 cents to $78.76.

Hewlett-Packard Co. (HPQ) pulled on the Standard & Poor's 500 index. Investment bank JP Morgan & Co. cut its rating on computer and printer maker Hewlett-Packard to "neutral" from "overweight," citing the company's exposure to consumer PCs and printers. Hewlett-Packard declined 8 cents to $18.98.

Pulte Homes Inc. (PHM) fell 8.5 percent after the builder issued an earnings warning on Monday, citing softness in the Las Vegas, Nevada market. The stock fell $3.88 to $52.45 and was one of the largest percentage decliners on the New York Stock Exchange.

Eastman Kodak Co. (EK) rose 31 cents to $33.80 after it announced it will cut nearly 900 jobs at three manufacturing facilities in Europe, part of the company's transition from film to digital photography.

Camden Property Trust (CPT) tumbled $1.85 to $45.05 after it said it would pay $1.9 billion in a takeover of rival Summit Properties Inc. (SMT), which would create a major national network of apartment buildings. Summit was up $2.80 at $30.64 on the news.

Technology stocks fell as Advanced Micro Devices Inc. (AMD), Intel Corp.'s (INTC) rival in computer semiconductors, warned that its revenues would be less than forecast for the third quarter. AMD slid 2 cents to $13.68, while Intel gained 19 cents to $21.32.

Genelabs Technologies Inc. (GNLB) dropped 65.64 percent, or $1.70, to 89 cents on the Nasdaq after the company said that a late-stage trial of its drug to limit bone loss in women with lupus disease failed.

Declining issues barely outnumbered advancers on the New York Stock Exchange, where volume came to 1.42 billion shares, compared with 1.53 billion on Monday.

The Russell 2000 index of smaller companies was down 1.75, or 0.3 percent, at 587.34.

Overseas, Japan's Nikkei stock average was flat. In Europe, Britain's FTSE 100 closed up 0.5 percent, France's CAC-40 edged 0.1 percent higher for the session, and Germany's DAX index gained 0.4 percent.

Reuters and The Associated Press contributed to this report.