Specter becomes the third incumbent member of Congress in two weeks to lose his job, after Republican Sen. Bob Bennett lost the GOP nomination at the party's Utah convention and Democratic Rep. Alan Mollohan lost a primary in West Virginia.
In November, Sestak will face Pat Toomey, who won the Republican nomination.
Specter, after calling Sestak to congratulate him, delivered a concession speech in which he thanked his campaign and his supporters.
"It's been a great privilege to serve the people of Pennsylvania," Specter said to a round of applause. "And it's been a great privilege to be in the United States Senate, and I'll be working very hard for the people of the commonwealth in the coming months."
Specter switched parties last year in what he thought would be an easier path to another nomination. But Sestak proved to be a hurdle too high for the five-term senator.
Sestak took 54 percent of the vote to Specter's 46 percent with nearly all precincts reporting. Sestak told Fox News on Wednesday that voters wanted a change.
"I think they're very tired, and rightly so, with deal-making where quite frankly career politicians might do anything to keep their jobs rather than focusing on them," he said.
The vote also was a defeat for President Obama, who supported Specter when the senator abandoned the Republican Party last year. The moderate Specter cast his switch as a decision of principle after inflaming the GOP by voting for Obama's economic stimulus bill. But many Democratic voters questioned his motives.
Still, the reaction from Democratic leaders to Specter's loss suggested they believed they got their money's worth from Specter's party switch.
"Arlen Specter has been a valuable member of our caucus over this past year," Senate Majority Leader Harry Reid said in a written statement.
"Without Arlen's support, we could not have passed health insurance reform, the Recovery Act or other key parts of our agenda," he said. "I am extremely grateful to Sen. Specter for his career of service to the commonwealth of Pennsylvania and to our nation."
On the campaign trail, Specter and Sestak let their punches fly, with Specter attacking Sestak's military record and Sestak linking Specter to former President George W. Bush, who has called him a "firm ally."
The Pennsylvania primary was one of several high-stakes Senate contests that political analysts were closely observing for clues on how the electorate may react in November's midterm elections, as Republicans seek to regain control of both chambers in Congress.
Democratic Sen. Blanche Lincoln has been forced into a June runoff against her top primary opponent, Lt. Gov. Bill Halter. And in the Kentucky GOP primary, Tea Party favorite Rand Paul defeated the party establishment's candidate, Secretary of State Trey Grayson.