Some 26 people remained missing after the fire.
Santa Barbara County Sheriff Bill Brown said crews spotted the additional bodies Monday close to the sunken vessel, but the boat remained unstable and it's uncertain when those remains would be recovered.
The four unidentified bodies recovered Monday were two men and two women.
Coast Guard Capt. Monica Rochester said the search for the other missing victims would continue into Tuesday morning but people should be prepared for "the worst outcome."
Police have interviewed the five crew members who survived the fire, but Brown didn't disclose what they said. No cause for the fire has been identified and Brown said at this stage there was no reason to believe it involved a criminal act.
The five crew members who survived had jumped overboard, making it to a nearby boat. It's unclear what efforts they made to help the passengers.
"I can't breathe," a man was heard saying during a call for emergency help.
CLICK HERE TO GET THE FOX NEWS APP
The Coast Guard said 39 people were aboard the Conception, which was on a three-day scuba diving excursion to the Channel Islands. Six crew members and 33 passengers were on board when the fire broke out around 3 a.m. Monday and quickly engulfed the boat.
The Associated Press contributed to this report.