Tim Cahill scored Australia's first ever World Cup goal in the 84th minute after Japan goalkeeper Yoshikatsu Kawaguchi failed to punch clear a throw-in. He added another in the 89th, and John Aloisi scored a third in injury time to seal Australia's first World Cup win in the team's second appearance.
Shunsuke Nakamura had given Japan the lead in the 26th minute with a cross that beat Australia goalkeeper Mark Schwarzer.
His cross from 25 yards (meters) sailed in over Schwarzer, who failed to grab the ball when he crashed into a group of players in front of the net.
Japan forward Naohiro Takahara made contact with Schwarzer as the two leapt in the air but there was no foul. Schwarzer walked off at halftime shaking his head.
Australia almost equalized less than a minute later when Harry Kewell's shot sailed over the crossbar.
Cahill, who came on in the 53rd minute, equalized for Australia when he picked up a loose ball and beat Kawaguchi with right foot shot.
The Everton midfielder added another five minutes later when his 20-meter (yard) shot hit one post and then went into the net.
In injury time, Aloisi threaded his way through the Japanese defense and scored from 15 meters (yards).
The loss was a huge setback for Japan, which dominated most of the second half with Kawaguchi making several key saves off free kicks.
Kawaguchi dove to his left in the 75th minute, stopping Mark Viduka's free kick.
Australia's only other trip to the World Cup was in 1974, when it failed to score a goal in losing to host West Germany and East Germany and drawing 0-0 with Chile.
Japan is in its third straight World Cup. Co-hosts in 2002, the Japanese lost 1-0 in the second round to eventual semifinalist Turkey.
Brazil and Croatia are also in the group.