James narrowly missed a triple double with 28 points, 11 assists and nine rebounds, while Mo Williams added 22 points and 10 assists as the Eastern Conference's top team won for the fourth time in five games.
Cleveland (32-11) held a 95-94 edge late in the fourth quarter before breaking away thanks to Williams' three-pointer and capped by a James dunk.
James, the league's reigning MVP, made four successive free throws in the final minute to clinch the win.
"They gave us a little trouble in the first half but we were able to slow that down (in the second half). We were a bit more physical on the defensive end," James told reporters after finishing with three steals and three blocks.
Toronto opened the season with a win over Cleveland that dropped the Cavs to 0-2. Since then, Cleveland have hit their stride and taken over the East, while the Raptors are battling for a playoff position.
Cleveland's Shaquille O'Neal had 16 points and surpassed the 28,000-points mark for his career, the fifth player to do so.
"The only fallback for me is that I missed like 5,000 free throws," O'Neal said. "If I would have at least hit half of those, then I would be at 30,000 right now."
(Editing by Peter Rutherford and Alastair Himmer)