Totti, who came off the bench in the 75th minute, took the penalty after Lucas Neill brought down Fabio Grosso with 12 seconds of injury time left to play.
The AS Roma playmaker, still not fully fit after 2 1/2 months out following a broken left leg, then scored to send the three-time World Cup champions into the last eight. Italy will face either Switzerland or Ukraine on Friday in Hamburg.
Italy was reduced to 10 men in the 51st minute when Marco Materazzi was sent off for bringing down Marco Bresciano just outside the area.
Australia dominated possession after Materazzi's red card, but rarely created good goal-scoring opportunities.
Italy goalkeeper Gianluigi Buffon twice saved shots from Scott Chipperfield and, with the Australians in control, Italy coach Marcello Lippi gambled by sending on Totti.
In the 89th, Australia midfielder Tim Cahill had another chance from Marco Bresciano's corner, but he headed over the crossbar.
In the first half, Italy striker Luca Toni nearly scored four times before the team was reduced to 10 men.
"I was never worried because there was still extra time to play and penalties," Lippi said. "This was a game that really had everything.
"In the first half, we didn't allow even one shot on goal -- not even from long range. In the second, we struggled but we gave away very little and then there was this great play by Grosso."
The game appeared to be headed for extra time when Grosso cut in from the left, beat one defender and then went down under a challenge from Neill.
Spanish referee Luis Medina pointed to the spot.
"I saw only one replay and I couldn't really tell if it was a penalty or not," Australia goalkeeper Mark Schwarzer said. "Essentially, we just couldn't score. We are very disappointed."
Australia winger Harry Kewell, the Man of the Match in the 2-2 draw with Croatia, missed the game. As he sat on the bench, the Liverpool player supported himself with a pair of crutches after a recurrence of his long term groin problem.
Although the Australians put pressure on the Italians from the start, the European team created the better chances with Toni going close four times in the first half.
He headed narrowly wide from eight meters (yards) in the third minute from Alessandro Del Piero's left wing cross, forced Schwarzer to save with his legs, had another shot blocked by defender Craig Moore and sent another header past the post.
Alberto Gilardino also had a shot pushed over the bar by Schwarzer, and Chipperfield blocked a goalbound drive by Simone Perrotta.
At the other end, Mark Viduka's powerful header went straight Buffon.
Lippi sent on Vincenzo Iaquinta for Gilardino in the second half and Toni shot too high after another promising buildup.
But Lippi watched in dismay when Materazzi, who was having a standout game at the heart of his defense, was sent off for hauling down Bresciano.
Lippi was forced to reshuffle his lineup and take Toni off, replacing the Fiorentina striker with inexperienced Palermo defender Andrea Barzagli.
Now the Socceroos scented an upset and Chipperfield forced Buffon into a blocking save with a clever turn and shot inside the area.
Totti went on for Del Piero and immediately set up a shooting chance for Perrotta, whose powerful 20-yard drive was blocked by a defender.