David Kampf and Michael Bunting also scored for Toronto, which was outshot 46-29 but won its second straight.
"A good win against a really good team that has been playing really well lately," Rielly said. "The special teams stick out when we take that many penalties, so I thought the PK did a great job."
A fast start and shutting down the Penguins on five power-play opportunities were essential elements of the win.
"Loved those two parts, and Jack Campbell was just outstanding," Toronto head coach Sheldon Keefe said, adding Campbell had one of his best games of the year.
"This was as calm and poised as he’s looked all season. He made the tough saves look easy."
Evgeni Malkin scored for Pittsburgh. Tristan Jarry had 25 saves.
Matthews ran face-first into the crossbar of the Maple Leafs net as he pursued Penguins captain Sidney Crosby with 13:19 remaining in the third period.
Doubled over and holding his face, Matthews left the game but returned with five minutes left. Moments before his mishap, he set up Bunting for Toronto’s fourth goal, less than three minutes after Malkin scored on a rebound to break up Campbell’s shutout bid.
"He has some wounds," Keefe said. "But as far as I know, he’s going to be fine."
Campbell put forth his best outing in weeks, keeping a red-hot Crosby and the Penguins at bay. Even with the defeat, Pittsburgh has gone an impressive 14-4-3 since the start of the new year.
On Tuesday, Crosby scored his 500th career goal in a win at home against the Philadelphia Flyers.
Matthews needed 21 seconds to extend his point streak to nine games. He took a pass from defenseman T.J. Brodie and beat Jarry on the short side.
The breakaway goal was Matthews’ 33rd. He has collected eight goals and 17 points during his nine-game offensive spree.
Rielly increased the Maple Leafs to 2-0 with a breakaway, power-play goal. Instead of dropping a pass to either Mitch Marner or Matthews in the neutral zone, Rielly kept the puck. He sped past Penguins forward Brian Boyle and then split the defense for the goal.
Pittsburgh entered the game with the second-best penalty-killing unit in the league, while the Maple Leafs have the most efficient power play.
The Maple Leafs showed off their stingy penalty kill with a short-handed goal from Kampf late in the second period. The puck caromed to Rielly in the neutral zone after Campbell made a handful of stops with the Penguins on the power play. Rielly sped into the offensive zone to set up Kampf on a two-on-one rush.
Pittsburgh: Host Carolina Hurricanes in opener of a three-game homestand.
Toronto: Host St. Louis Blues on Saturday.