Cam Newton's 16-yard touchdown pass to tight 10.
Newton completed 18-of-34 passes for 158 yards for the Panthers (1-2), who lost to Arizona and Green Bay to start the season.
The Auburn product has thrown for 1,012 yards in his first three NFL games, becoming the first rookie in league history to accomplish that feat.
"It wasn't a pretty win," Newton said, "but the last part counts, right? We got a win and some confidence going into Chicago next week."
Jonathan Stewart carried the ball 10 times for 59 yards. Olsen hauled in seven passes for 57 yards.
Blaine Gabbert, the 10th overall pick in this year's draft, connected on 12- of-21 passes for 139 yards, one touchdown and an interception in his first career start.
The Jaguars (1-2) decided to switch quarterbacks this week after Luke McCown had a 1.8 passer rating in last Sunday's 32-3 loss to the Jets.
Maurice Jones-Drew carried the ball 24 times for 122 yards. He also caught three passes for 45 yards.
Trailing by two, 10-8, the Panthers took over at their own 45 with 6:44 to play. Stewart and Steve Smith caught passes of 18 and 13 yards, respectively, to put the ball at the 24. A DeAngelo Williams one-yard run was followed by a seven-yard Newton scramble.
Faced with a 3rd-and-2, Newton dropped back and found Olsen in the middle of the field. The tight end sprinted into the end zone for the lead and also added the two-point conversion for a 16-10 margin with 4:20 to go.
After each team was forced to punt, Jacksonville took over at its own 20 with 1:02 remaining. With no timeouts left, Gabbert got the Jaguars to the Carolina 36, but the clock hit zero as Gabbert's deep pass fell to the ground.
"It's disappointing we didn't come out with a victory," Jaguars head coach Jack Del Rio said. "We came out with a lot of resolve and played gritty and tough. We just didn't make enough plays. We made too many mistakes when it mattered."
Gabbert's first possession did not go according to plan. Faced with a 3rd- and-13, Gabbert was sacked by Greg Hardy in the end zone for a safety.
The Panthers bridged the first and second frames with a seven-play drive that resulted in a 35-yard Olindo Mare field goal.
Jacksonville responded with a field goal of its own. A neutral zone infraction penalty against Carolina on fourth down gave the Jaguars a fresh set of downs, and the visitors eventually took advantage with a 53-yard field goal from Josh Scobee.
Jacksonville took over at its own 23 with 53 seconds left until half. Jones- Drew broke off a 39-yard run on the second play, then Gabbert dropped back and found Mike Thomas in the middle of the field for a 36-yard score and 10-5 edge.
With 6 1/2 minutes to play in the third quarter, Jacksonville punter Matt Turk fumbled the football. The Panthers took over at the 38 of the Jaguars, but settled for a 32-yard Mare field goal.
The Jaguars lead the all-time series by a 3-2 count...Newton carried the ball seven times for 27 yards...A heavy rain storm occurred right before the half...The Panthers had 265 yards of total offense, compared to 257 for the Jaguars.