Another solid start by Jaime Garcia slipped away on a disputed call and two wild pitches.
The defending NL Central champs got a much-needed win against Garcia (9-4), who was 5-0 in his career against them.
Cardinals shortstop Ryan Theriot was ejected while arguing a call in the sixth that set up the tying run. Garcia then threw a pair of wild pitches that let Cincinnati pull even.
"There was a lot of passion in this clubhouse and in this game," Theriot said.
The Reds lead the NL Central rivalry 7-5, with one series left in St. Louis from Sept. 2-4. On the verge of slipping out of the race, they took two of three in the weekend series, leaving them in the four-team bunch atop the division.
"It's no different than we thought," Berkman said. "It's going to be one of those mad scrambles."
Garcia was tough in his first start since getting a four-year, $27 million contract extension. The left-hander allowed only three hits through the first five innings, holding fast to the 1-0 lead. Then, it slipped away.
Cozart singled in the sixth and Joey Votto hit a grounder to second baseman Skip Schumaker. His throw to get the forceout pulled Theriot across second base, and umpire Mike Muchlinski ruled Cozart safe.
Theriot ran to Muchlinski and pointed in his face while screaming. The umpire quickly ejected Theriot, and first base umpire Chris Guccione stepped between them.
"I know the job they do is very difficult," Theriot said of the umpires. "It was just such a crucial point in the game. I made it a point to keep my foot on the base, and I felt he was out. It's a big call. It's a call that meant a lot at that point. He ended up scoring."
The Cardinals considered it the turning point.
"In my opinion, that's what swung the game," Berkman said.
Cozart came around to score on a pair of wild pitches by Garcia, the second one a breaking ball in the dirt to Jonny Gomes.
"It was a curveball that I just spiked," Garcia said. "I tried to do too much with it. I was just a little excited with the situation and paid the consequences."
An inning later, Miguel Cairo hit his first triple since 2009 and scored on Hanigan's single up the middle for a 2-1 lead. The Reds finished 1 for 8 with runners in scoring position, but it was just enough. Cozart homered in the eighth off Lance Lynn.
Bailey got his first win since May 16. Berkman connected in the second inning for his 23rd homer at Great American Ball Park, the most by any visiting player.
Gold Glove second baseman Brandon Phillips made a diving catch of Yadier Molina's soft liner with two outs and two aboard in the eighth to preserve the lead.
Francisco Cordero, who blew his last three save chances before the All-Star break, improved to 18 for 23 with a perfect ninth against the heart of the Cardinals' order.
Notes: The first two games of the series sold out, but only 24,841 tickets were sold for Sunday. ... St. Louis expects to put INF Nick Punto on the DL for the third time this season with a sore throwing elbow. He pinch hit in the eighth inning and singled. He's scheduled to go back to St. Louis for an exam on Monday. ... Cozart has hit safely in his first six games in the majors, the longest such streak by a new Reds player since Adam Dunn hit in his first seven games in 2001.