The Islamic militant Hamas, which heads the Palestinian government, strongly opposes a referendum.
"President Abbas will set a date for the referendum after the meeting Tuesday of the PLO Executive Committee and parliamentary caucuses," his office said in a statement.
Abbas had set a midnight deadline for agreement. But a participant in late-night talks with Palestinian factions said the president determined they had failed an hour before that.
The Hamas-led government is facing international isolation over its refusal to renounce violence and recognize Israel. The United States, European Union and Israel have cut off cash transfers to the Palestinian government since Hamas won legislative elections earlier this year.
Its hardline stance has landed it in an increasingly violent power struggle with Abbas and his Fatah party, the other main Palestinian faction. Since Hamas was sworn into office in March, Abbas has taken steps to curb its authorities.
The plan under discussion was formulated by Hamas and Fatah prisoners held in Israeli jails. But Hamas' exiled leaders, who make final decisions on policy, have refused to accept the proposal.
Hamas has been calling for more time to discuss the proposal and suggested changes in the language. But Abbas, who has avoided confrontation since taking office early last year, rejected Hamas' demands on Monday.
"If anyone wants to amend this document, then we will not reach any results," Abbas told reporters after meeting with European Union foreign policy chief Javier Solana.
Abbas has endorsed the plan as a way to end the crushing sanctions against the Palestinians and allow him to resume peace talks with Israel.
Hamas reacted angrily to Abbas' threats Monday.
Hamas spokesman Sami Abu Zuhri insisted that the talks must continue.
"You cannot raise the sword of ultimatum, you cannot raise the issue of a referendum while you are talking about dialogue," he told reporters in Gaza. He said calling a referendum meant circumventing the elected government, led by Hamas.
Many Palestinians are uneasy about the referendum, though polls show the document would be approved easily.
In Gaza, Mohammed Abu Seido, 30, a coffee shop cashier, said he would vote for the document, but he worried that Hamas would react with violence if it is approved.
"Hamas is already failing," he said.
The Palestinians have never held a referendum before, and officials said the vote would not be binding. But passing the referendum could give Abbas an important boost in his standoff with Hamas.
"It would bolster his legitimacy and give him power to go ahead with negotiations with Israel," said Azam al-Ahmad, a top Fatah official. Al-Ahmad also said Abbas, who wields considerable powers, would consider calling elections for president and parliament if Hamas did not abide by the results of a referendum.
Hamas says that accepting the plan would mean abandoning its principles. It also says a referendum is not necessary because Palestinian voters chose its political program in legislative elections just over four months ago.
The Palestinian infighting has turned deadly in recent weeks. Sixteen people have died in clashes between Hamas and Fatah loyalists, including five killed on Sunday. Five Palestinians were wounded in two clashes in the southern city of Khan Younis on Monday, security officials said.
Hamas has stuck to its tough line on Israel despite the crushing international boycott, which has left the government unable to pay the salaries of tens of thousands of civil servants for three months.
Hamas officials promised over the weekend to begin paying overdue salaries. But on Monday, the government failed to meet its pledge, causing widespread confusion in the Palestinian areas and prompting angry workers to storm one Gaza City bank.
Some banks advanced money to unpaid civil servants to help calm the situation. In some cases, people received money. In others, the promised payments failed to materialize. And tens of thousands of workers remained without paychecks, with the government as far away as ever from ending the boycott.
"These are people who don't have money to buy milk," said policeman Raed Abu Ghoneima, one of the protesters who stormed a Gaza City branch of the Arab Bank. "It has nothing to do with politics; it's about wages."
Also Monday, Israeli aircraft fired missiles at a car in the Jebaliya refugee camp next to Gaza city, the military said, killing two militants and wounding two bystanders, according to Palestinian hospital officials.
Israel said the main target, a militant from the renegade Popular Resistance committees, was involved in firing rockets at Israel. He died of his wounds.