Albert Pujols tied the game in the bottom of 3 game series.
Jason Motte (4-2) tossed a 1-2-3 top of the 10th inning to earn the win and help St. Louis get within 6 1/2 games of Atlanta for the NL wild card. The Cardinals also sit seven games back of the Brewers for first place in the NL Central.
Michael Bourn went 4-for-5 with an RBI and a run scored in the Braves' sixth loss in their last nine games.
Randall Delgado allowed one run on three hits and three walks through five innings in the start for Atlanta, and Edwin Jackson, pitching on his 28th birthday, surrendered two runs on seven hits and two walks while striking out five over six frames for St. Louis.
Kimbrel surrendered a leadoff single to Skip Schumaker in the ninth, but he was erased on pinch-hitter Punto's fielder's choice to second. After Gerald Laird struck out, Kimbrel gave back-to-back walks to Rafael Furcal and pinch- hitter Ryan Theriot to load the bases. Pujols then slapped a ball down the first-base line, scoring Punto and Furcal. Theriot was thrown out at home on the play to send the game into extras.
The runs by Kimbrel snapped his 37 2/3 inning scoreless streak and his stretch of 25 straight converted saves.
"When you put guys on for free it always seems it comes back to get you," Kimbrel said.
Linebrink gave up consecutive singles to Matt Holliday and Lance Berkman to start the 10th. After Daniel Descalso laid down a pinch-hit, sacrifice bunt to move the runners up, Schumaker was intentionally walked to load the bases.
Punto came up next and took two balls before taking a strike. He then swung and missed at a ball right over the middle of the plate before fouling off two straight pitches. After taking a ball and fouling off an inside fastball, he lifted a 92 mph fastball into left-center to bring across the game-deciding run.
"Punto's at-bat was a classic in my opinion," said Cardinals manager Tony La Russa. "You're not going to win a lot of games coming from behind against them."
The Braves jumped out to an early lead after Bourn led off the game with an infield single and crossed the plate on Brian McCann's double. Two batters later, Freddie Freeman brought McCann home with a double to left, giving Atlanta a two-run advantage.
Furcal worked a walk to start the home first, advanced to third on a Pujols single and crossed the plate on Holliday's groundout to third.
Eric Hinske hit a two-out, pinch-hit single in the ninth inning against Octavio Dotel. After pinch-runner Antoan Richardson stole second, Dotel was replaced by Arthur Rhodes on the bump. Bourn greeted Rhodes by pulling a triple into the right-center field gap to increase the Braves' lead. Chipper Jones grounded out to short to end the inning.
Jackson has not allowed more than three runs in seven straight starts...The Cardinals were 0-for-6 with runners in scoring position before Pujols' single in the ninth. They ended the game 1-for-7 with runners in scoring position and left 11 men on base...The Braves went 2-for-9 with runners in scoring position and stranded eight.