More than 40 were still missing in the accident, which occurred about 40 miles off the port of Sibolga on western Sumatra, naval spokesman Lt. Col. Jaka Sentosa told el-Shinta radio station.
He said the vessel sprang a leak in bad weather.
At least three foreigners were believed aboard the ship, which was heading to the island of Nias, a popular spot for surfers, Sentosa said. It was unclear whether they were among those rescued.
Two navy vessels dispatched to the scene and a passing freighter rescued the 73 survivors, Sentosa said.
Survivor Hutagalung, who asked to be identified only by her family name, said most passengers had time to put on lifejackets and a passing ship picked up scores of survivors within an hour.
"When the ship began tilting, children and elderly people became hysterical and started to scream," she said. "Passengers were thrown overboard as the wave shook the ship left and right."
The first group of survivors to arrive at Sibolga were taken to a hospital. Hundreds of worried people waiting for news of relatives flocked to waterfront and hugged their loved ones, a witness said.
Sibolga is 860 miles northwest of Jakarta.
Sea accidents are common in Indonesia, a vast archipelago where boat travel is the only way to reach many islands. Safety measures are poorly enforced, and many crafts lack sufficient life jackets and other safety equipment.