The move was announced before the AL East champions faced the Texas Rangers in Game 2 of their first-round series.
Baldelli went 0 for 3 with two strikeouts against Cliff Lee in Game 1. His career has been derailed by a string of injuries and a condition that causes muscle fatigue.
"Not being able to play baseball has been one of the toughest things I've had to deal with for probably the last five years of my life," said Baldelli, who missed all of 2005, as well as most of the 2008 and 2009 seasons because of a variety of injuries that kept him off the field. "It's what I love to do, and it's a difficult thing dealing with it."
Manager Joe Maddon said "cramping ... the same things that's bothered him to this point" surfaced again during Wednesday's 5-1 loss to the Rangers. The Rays announced the 29-year-old was removed because of "left leg fatigue," which is a symptom of mitichondrial disorder.
"He spoke to me after the game. It just wasn't good. ... He was very distraught," Maddon said, adding that Baldelli showed up at Tropicana Field and tried to hit several hours before Game 2 but "realized it's not going to work."
The manager said there didn't appear to be any problems with Baldelli when the decision was made to include him on the roster.
"But with this condition, anything could pop up at any time, and it did," Maddon said. "It's unfortunate."
Baldelli began the season as a special instructor while rehabbing a shoulder injury from 2009. He played briefly in the minor leagues before being recalled Sept. 1. In 10 games, he batted .208 with one homer and five RBIs.
"I can honestly say that I'm not the best guy to do the job. If I could run in the least at this point or even hit at this point, trust me, I would be out there," Baldelli said. "But I have to look at myself and be honest."
Aybar, who was in Thursday's lineup as the designated hitter, batted .220 with six homers and 43 RBIs in 100 games this season.