There may be as many as 5,000 people in the United States linked to Usama bin Laden's Al Qaeda terror network, intelligence officials estimate.

But the number of hardcore Al Qaeda members here who might actually do harm to Americans is in the low hundreds or even less, officials believe.

The 5,000 estimate includes all those in the "realm of suspicion" and those who may be aware of terrorist activities but not participate, one official said.

The FBI is searching for Americans and others suspected of advising Al Qaeda cells operating underground on U.S. soil and preparing for another attack, according to law enforcement officials.

Some of the suspected advisers are believed to be longtime U.S. citizens, fully immersed in American life and able to financially direct an attack without directly participating in it, the officials said.

Al Qaeda manuals recovered in Afghanistan suggest that terror operations have a "senior adviser or wise man" who does not take part attacks, said a law enforcement official who is familiar with the investigation.

While law enforcement looks broadly for terrorists, some FBI agents are working closely with Treasury agents to conduct a more specialized search for U.S. residents who might be working in that advisory capacity.

As part of the effort, federal investigators are conducting extensive checks into the backgrounds of longtime citizens who fall under suspicion, looking for operatives who may not have anything unusual in their immediate histories.

The agents are "looking for people who have an affinity toward or sympathy for those carrying out terrorist attacks and provide any kind of support," one law enforcement official said, speaking on condition of anonymity.

The search for terrorist operatives has frustrated members of the Muslim American community, who are pleased with the Justice Department's prosecution of backlash hate crimes but feel their own rights are being violated.

The FBI's counterterrorism team played a major role in the search of 14 homes and businesses in Virginia and Georgia in March.

No one was arrested. None of the law enforcement agencies involved would provide further information, saying affidavits filed in support of the search warrants are under seal in federal court.

Days after the raids, Laura Jaghlit, a high school English teacher from Fairfax Station, Va., described the raid on her home as "the most un-American thing I have ever seen."

She is still indignant.

"No one ever had to give us any explanation of why we were chosen for a search or what they were looking for," Jaghlit said. "We're just supposed to forget about it."

Ibrahim Hooper, a spokesman for the Council on American-Islamic Relations, said, "We understand and have been told that the FBI is looking for terrorist operatives of all ages and races, but what we're seeing is that people who are well-known and respected citizens are being treated like suspects."

"There are all of these vague references to threats without any evidence. It puts everybody on edge and makes the Muslim community suspect."

Law enforcement officials believe that limiting the search to those suspected of actively planning an attack might allow some who give terrorists support to remain hidden.

The possibility of terror advisers who don't fit the demographics of any of the Sept. 11 hijackers — young, Muslim men, who came to the United States in recent years — has led the FBI agents to seek out contacts in all parts of the Muslim community.

That includes older men and people of different demographics that might have gotten in contact with Al Qaeda later in life, one law enforcement official said.

The Associated Press contributed to this report.