Fumio Kyuma had come under intense criticism from survivors of the bombings, opposition lawmakers and fellow members of the Cabinet following the comments over the weekend.
"I told Prime Minister (Shinzo) Abe, 'I'm sorry but I must take responsibility and resign.' The prime minister said, 'That's very unfortunate ... but I accept your decision,"' Kyuma told reporters.
Kyuma ignited a political furor less than a month before parliamentary elections when he said on Saturday that the atomic bombings of Hiroshima and his native Nagasaki were an inevitable way of ending World War II.
The statement contradicted the Japanese stance, fiercely guarded by survivors and their supporters, that the use of nuclear weapons is never justified. A ban on possession of such weapons is a pillar Japan's postwar pacifist regime.
Earlier Tuesday, Nagasaki's mayor made an official protest in Tokyo.
"That comment tramples on the feelings of the A-bomb victims, and as a target of the bomb, Nagasaki certainly cannot let this go by," Nagasaki Mayor Tomihisa Taue wrote in a letter handed over to Kyuma on Tuesday.
"I truly apologize for having troubled and caused worry to the people of Nagasaki," Kyuma said.
The bomb comment from the gaffe-prone Kyuma has hit Abe's increasingly unpopular government at a sensitive time, coming just a few weeks before July 29 elections for the upper house of parliament.
Kyuma's repeated apologies and Abe's reprimand of his defense chief have failed to quell the furor, which on Tuesday sparked further public criticism among Abe's own ministers, several of whom called the comment inexcusable.
The opposition had been preparing to submit a formal request for Kyuma's resignation later on Tuesday, and opposition leaders claimed that Abe shared the blame for the gaffe.
At a speech in Chiba outside of Tokyo on Saturday, Kyuma triggered the scandal by suggesting the bombs were an inevitable way of ending World War II.
"I understand that the bombings ended the war, and I think that it couldn't be helped," he said.
Kyuma — who represents Nagasaki in the lower house — said the U.S. atomic bombings caused great suffering in the city, but otherwise Japan would have kept fighting and ended up losing a greater part of its northern territory to the Soviet Union, which invaded Manchuria on the day Nagasaki was bombed.
Abe has struggled to control the political damage. He reprimanded Kyuma on Monday and asked him to refrain from making similar remarks in the future, but did not publicly call for Kyuma to resign.
On Aug. 6, 1945, the U.S. dropped a bomb nicknamed "Little Boy" on Hiroshima, killing at least 140,000 people in the world's first atomic bomb attack. Three days later it dropped another atomic bomb, "Fat Man," on Nagasaki where about 74,000 are estimated to have been killed.
Japan, which attacked the United States at Pearl Harbor in 1941, surrendered on Aug. 15, 1945.
In January, Kyuma raised eyebrows in Washington by calling the U.S. decision to invade Iraq a "mistake" because it was based on the false premise that Saddam Hussein possessed weapons of mass destruction.
Japan and the U.S. are close military allies, and Japan hosts some 50,000 American troops under a security treaty.