China has refused to meet with Koizumi in recent years, in part to protest his annual visits to Yasukuni Shrine, which honors Japan's war dead, including several executed war criminals.
"I think China will someday regret that it did something as incomprehensible as not holding a summit because of one problem," Koizumi told reporters, without elaborating.
Koizumi has long defended his shrine visits as aimed at praying for the country's war dead and for peace, and last worshipped there in October.
He has gone every year since taking office in 2001, and speculation is rising about whether he plans to go again this year, before the end of his term in September.
Koizumi has continued the visits despite a rapid deterioration in ties between Tokyo and Beijing. The two countries are battling over wartime history, maritime resources and territorial claims.
He has refused to reveal whether he will go to the shrine again this year, but reiterated his argument that China's objection to the visits is an interference in Japan's internal affairs.
"The people who criticize my visits tell me that in order to maintain friendly ties with China, I must not visit the shrine," he said. "In extreme terms, it means it is wrong if I do not listen to China."
While Koizumi says he goes to Yasukuni to pray for peace, the shrine was a center of propaganda backing imperialist expansion during the war, and honors fallen soldiers as deities.
The shrine also hosts a history museum that depicts Japan's conquests in Asia and the Pacific in the 1930s and '40s as a crusade of liberation from Western colonialism.
The visits have triggered domestic lawsuits claiming they violate the constitutional division between religion and the state. The shrine is part of Shinto, the emperor-led creed that was Japan's state religion before the war.
Koizumi also took aim at domestic critics.
"I can't understand why an outside country would tell me not to visit Yasukuni Shrine when I am only expressing these regrets for those who died at war," he said. "More so, I cannot understand why Japanese people would empathize with this opposition from China."