The mistrial further clouds the legal picture for Merck, which had lost one Vioxx trial in Texas state court, but later won a case in New Jersey state court.
The drug maker faces more than 7,000 lawsuits claiming that for years it hid the risks of heart attack and stroke linked to its blockbuster drug. Analysts have said the company's legal costs to fight the cases could total several billion dollars.
The nine-member jury in federal court said on Saturday it had deadlocked after about two days of deliberations, but U.S. District Judge Eldon Fallon sent the panel back to try to reach a verdict on whether Vioxx had contributed to the fatal heart attack of a Florida man in 2001.
Lawyers in the case said Judge Fallon would meet with the lawyers on Friday to set a date for retrial, which could take place in February.
Judge Fallon previously said he planned to hear four types of Vioxx cases, including the Irvin case, which involved a heart attack suffered by a short-term user.
The upcoming cases involve a heart attack by a long-term Vioxx user, a stroke case and one from a patient who took drug after its label was altered to indicate there could be cardiovascular risks.
"There are many more trials to come both in federal and state courts throughout 2006. We remain committed to addressing these cases one by one over the coming years," Merck's general counsel, Kenneth Frazier, told a conference call.
Merck, whose shares were down more than 3 percent on Monday afternoon trading, pulled the drug off the market in September 2004 after it said it had become clear long-term usage could double users' risk of a heart attack or stroke.
One legal expert said the mistrial was a setback for Merck, since it involved a patient who used the drug for only a few weeks.
"I think it's not a good result for them," said Professor Carl Tobias of the University of Richmond's School of Law.
"I think most people expected Merck would win this case in federal court. Federal courts are more favorable for institutional defendants like Merck."
Analysts have said the upcoming cases in New Jersey state court that involve people who took Vioxx for longer than 18 months could favor the plaintiffs.
In the federal case, the jury had been asked to decide whether Vioxx contributed to the heart attack death of 53-year- old Richard "Dicky" Irvin Jr., a manager at a seafood distributor, who took the drug for less than a month for back pain.
Merck's lawyers argued Irvin had an existing circulatory problems and died after plaque in his arteries broke loose and caused a blockage that triggered his heart attack.
The Merck lawyers said they were disappointed by the mistrial, but would not alter their strategy for defending the company in future cases.
"The fact that we were unable to get a verdict the first time out I don't think is going to change our tactics," Phil Beck said.
Judge Fallon told the lawyers they may not contact the jurors involved in the mistrial and the nine members of the panel were not made available to talk to reporters.
Jere Beasley, lawyer for Irvin's widow, said new doubts about the studies Merck submitted to regulators to get approval for Vioxx, as well as the demotion of a key plaintiff witness after his testimony in the trial, would alter the legal landscape in the coming trial.
"Things have happened since the jury went out that are going to have a definite effect on this case," he told reporters.
Last week, Dr. Eric Topol, a cardiologist and the head of the nationally renowned Cleveland Clinic at the Lerner College of Medicine at Case Western Reserve University, lost his leadership position at the clinic days after his testimony calling Merck's behavior "repulsive" was played at the trial.
Also last week, the New England Journal of Medicine said Merck withheld information about the risky side effects of Vioxx when it presented the journal data from a key trial on the medicine's safety in 2000.
Merck said it disclosed the events to U.S. regulators in 2000 and publicly in 2001, including in several press releases.
The withdrawal of Vioxx, which was ultimately taken by more than 20 million people and generated more than $2.5 billion in sales for Merck in 2004, contributed to a $25 billion decline in the company's market capitalization.
The Texas widow who won the first case against the company in state court in August was awarded $253 million in compensatory and punitive damages, although that amount is likely to be trimmed to about $26 million because of Texas limitations on damage awards.
Shares of Merck, a component of the Dow Jones industrial average, were down $1.02 cents at $28.11 on the New York Stock Exchange. The stock is off more than 10 percent since the start of the year.