Students for a Free Tibet said three Americans were taken away after holding up a banner at a base camp on the Tibetan side of the mountain that said "One World, One Dream, Free Tibet 2008."
The fourth person detained by Chinese authorities was filming the protest, said the group's executive director Lhadon Tethong.
"One World, One Dream" is the slogan of the 2008 Beijing Olympics.
The activists, who included a Tibetan-American, were protesting China's bid to take the Olympic torch to the top of the world's tallest mountain on the border between Tibet and Nepal.
The International Olympic Committee plans to announce the route for the Olympic torch relay on Thursday.
"The Chinese government hopes to use the 2008 Olympic Games to conceal the brutality of its occupation of Tibet," Tethong said from the Nepalese capital, Katmandu.
Taking the torch up Mount Everest is seen by some as way for Beijing to underscore its claims to Tibet, the Himalayan region that China occupied in 1951 and continues to rule with a heavy hand.
One of the protesters, reached by cell phone, said they had been well treated but did not know how long they would be held.
"We were questioned separately by police and they took our passports away," said Kierstan Westby of Boulder, Colo.
She said they displayed the banner for about 30 minutes before local authorities took them away. "We are hoping they take us to the border and let us go."
Later calls to Westby's phone were answered by a message saying it was out of service.
Tethong identified the other three as Shannon Service and Laurel Mac Sutherlin of San Francisco and Tenzin Dorjee of New York.
She said more than 70 Chinese climbers were in the base camp preparing for a trial climb to see if it is possible to take a torch to the top of the 29,035-foot mountain.
"The International Olympic Committee has no business promoting the Chinese government's political agenda by allowing the torch to be run through Tibet," Tethong said.
Hein Verbruggen, head of the IOC Coordination Commission overseeing the Beijing Games, tried to stay above the fray.
"We don't want to be, as the IOC, involved in any political issues," Verbruggen said. "It's not our task. We are here for organizing the Games."
Tethong said Students for a Free Tibet has 650 chapters in more than 30 countries and has about 20,000 members.
U.S. Embassy spokeswoman Susan Stevenson said privacy laws prevented her from commenting on the detentions.