WASHINGTON – Threats from Usama bin Laden (search) were detailed in television documentaries, books and all over the Internet in scores of languages, but the intelligence community largely ignored such "open source information" until it was too late, say former intelligence agents for the military and CIA.
Proponents told FOXNews.com that open source information can be an equally or more efffective resource for U.S intelligence. They say an over-reliance on "secret" information has caused an institutional bias against publicly-available information that could otherwise help identify terrorist threats across the globe before the damage is done.
"Everything we needed to know to prevent 9/11 was either known to elements of the U.S. government but not shared across agency boundaries, or openly published in foreign language media we chose to ignore," said Robert Steele, a former clandestine officer who now heads Open Source Solutions Network Inc., a global provider of open source networking and analysis training.
"[Al Qaeda] was publishing its plans and intentions publicly for years," Steele said. "All of the literature is quite clear."
Steele asserts that the U.S government is spending less than 1 percent — about $500 million — of the estimated $70 billion total yearly intelligence capabilities budget on gathering and using information in the public sphere. In a global society where the United States faces threats from all directions, this puts national security at a disadvantage, Steele said.
"The U.S. government has absolutely no idea what is going on in the provinces of countries, within tribes or at the ethnic and neighborhood level where the psychology and sociology of suicidal terrorism is best understood," he said.
Steele isn't the only one questioning why the government doesn't fully appreciate open source intelligence, which includes everything from public Web sites, news reports, blogs, books, radio and wire transmissions.
"I think we need to do more," said Hal Kempher, former military intelligence and now a California-based homeland security consultant. "One of the problems we have, particularly with state and local governments, is effectively scanning what's out there. Scanning to look for burgeoning trends ... in order to find anomalies."
Rep. Rob Simmons, R-Conn., chairman of the House Homeland Security Intelligence, Information Sharing and Terrorism Risk Assessment Subcommittee, led a hearing last month to look at what the government could do better.
"I believe that the Department of Homeland Security and the U.S. government need to do more to create open source products and integrate open source information into the DHS analytical product," he said during the June 21 hearing.
Despite an overwhelming majority opinion that mining open source information is a useful endeavor, Joseph Onek, a senior policy analyst at the Open Society Institute, warned lawwmakers that allowing too much access by the government into individual records could tread on civil liberties. Onek added that initiatives allowed in the Patriot Act, for instance, give government the ability to delve into personal information without a subject's knowledge and could end up unfairly targeting innocent groups.
"National security interests are also served by providing limited use immunity to people caught up in our anti-terrorism, data-mining efforts," he said.
Even with attempts to be sensitive about the use of personal data, intelligence experts say open source information is not getting the attention it deserves. They say an inherent bias has developed over the years among agents who resist using open sources, and the government tends to not take it seriously enough. Much of the prejudice stems from the effort employed in acquiring "secret" information and the specialized services that have built up around it.
"Many policymakers and other consumers are also a bit dazzled by the 'secret' stamp, so they tend to demand, for example, (to) respond more positively, to clandestinely-acquired material," said Tom Carroll, a consultant and former clandestine CIA agent based in California.
Thanks to the Internet, commercially accessible satellite imagery and even cell phones and faxes, an overwhelming amount of open source material is easily accessible, say the experts who spoke with FOXNews.com. Another problem is that the intelligence community has not yet figured out how to effectively harness it, said Joseph Mazzafro, a retired naval intelligence officer.
"For years and years, it used to be that if you wanted to know something about, let's say, the Soviet Union, you needed to penetrate something, there was a lot secrets, so you needed this network of classified, experienced collectors to get it," Mazzafro said.
"Then comes along the Internet and Cisco routers and a collapse of a totalitarian system, which all of a sudden puts stuff in plain sight for a lack of a better word. It's one of the things that have been missed by the intelligence community," he said.
Debate continues over how to go about collecting and analyzing open source intelligence. Some say a new bureau within one of the intelligence agencies or at the Department of Homeland Security (search) would be the best way to centralize, vet and distribute open source information.
Others say a network should be developed outside of any one central office, so that all agencies, including local and state homeland security interests, can access the information.
"The key is to bring all the available information, regardless of its origin or source, together for comprehensive and expert analysis and then, of course, to get that information to people who need it in real time so we can act upon it. That was the ultimate lesson of September 11," Homeland Security Committee Chairman Chris Cox, R-Calif., said at last month's hearing.
But Cox, who is planning a move from elected politics to the Securities and Exchange Commission (search), said such information should not be segregated in a separate entity or agency. A type of "Federal Bureau of Found Objects … that's exactly the sort of intelligence-specific balkanization that the Homeland Security Act seeks to remedy," Cox said.
Steele said a truly free system would look more like the Internet and would be completely accessible to the public not only for their own use but to make contributions. He said he believes this will not only inform agencies about homeland security interests, but will alert the global citizenry as well, making the world safer and more democratic.
"The alternative national intelligence community is emerging, and it will be both public and commercial in nature," Steele said, adding that he believes open source intelligence will overcome secret intelligence in the future.
Agreeing that open source intelligence has been ignored on many fronts, Mazzafro said he is not convinced it will completely overtake secret intelligence gathering.
"He's got a point," he said of Steele's claim. "But at the end of the day, you might have to send a spy in there and steal the stuff. It's just not that clean and simple."