Updated

Cartagena, COLOMBIA -- The scandal involving nearly a dozen Secret Service agents being sent home because one allegedly tried to bring a prostitute into his hotel room has now widened to include five U.S. service members.

Military officials said Saturday the service members violated curfew and "may have been involved in inappropriate conduct ... alleged to have occurred in the same hotel where the recalled U.S. Secret Service agents were staying."

The service members were assigned to Joint Task Force Summit of the Americas in support of the Secret Service.

General Douglas Fraser, commander of USSOUTHCOM, said he is "disappointed by the entire incident and that this behavior is not in keeping with the professional standards expected of members of the United States military."

A White House spokesman said the president is aware that service members now appear to be involved.

The U.S. Secret Service said on Saturday it had put 11 agents on administrative leave to investigate their behavior, and apologized for the distraction the incident had caused.

A dispute in a hotel about paying for an extra "guest," possibly a prostitute, started the confrontation that ultimately led to sending home the agents covering detail of President Obama's trip in Colombia, a senior law enforcement official told Fox News' Ed Henry.

New details are emerging about the incident at the Hotel Caribe, where the Secret Service advance team was staying ahead of Obama's three-day visit to the country.

A senior law enforcement official said the dispute started when hotel employees witnessed inappropriate behavior.

An employee approached at least one Secret Service agent and demanding that he pay extra money for having an additional overnight guest in his room.

The agent balked, which eventually resulted in a confrontation and forced diplomatic intervention.

The official said the guest involved apparently was a prostitute. It remains unclear whether more agents and more prostitutes were involved.

A hotel employee told the Associated Press the agents were drinking heavily during their stay.

The senior law enforcement official stressed the matter is under investigation and officials are still trying to corroborate initial details.

The hotel tells Fox News they will not be commenting on the matter.

A new Secret Service crew was immediately rushed to Colombia to cover the agents who had been sent home.

What makes the case unusual is the large number involved. In previous incidents involving agents on a trip, typically only one or two personnel have been involved.

As is protocol with any White House foreign trip, there's a large contingency of federal agents covering multiple aspects of presidential security. Agents often arrive several days ahead of the president.

The embarrassing incident has quickly eclipsed issues addressed at the summit such as foreign trade, the economy; drug trafficking, immigration and collaboration among the 33 Western Hemisphere countries.

The beachfront hotel where the apparent incident occurred is also the same where other White House staff and the press are staying.

A Service Service spokesman said the alleged incident occurred prior to the president's arrival. The agency would not confirm the total number of people involved or whether the incident had to do with prostitution. Officials stressed that the president's security was never in any danger.

The White House has not yet commented and is directing all inquiries to the agency.

Prostitution is legal in Colombia in certain "tolerance zones"

"The Secret Service takes all allegations of misconduct seriously," agency spokesman Edwin Donovan said Friday. "This entire matter has been turned over to our Office of Professional Responsibility, which serves as the agency's internal affairs component."

Capitol Hill lawmakers have been briefed on the situation. Sources tell Fox News Senior Producer Chad Pergram that those involved were not on the president's detail.

There are different divisions of security positions within the Secret Service, including a uniformed division and one that provides presidential protection.

Former Washington Post reporter and author on a book about the agency, Ronald Kessler, first reported the allegations.