Wikipedia Search Engine to Let Users Influence Web Results Rankings

Jimmy Wales, the founder of Wikipedia, has said that just as Internet users can contribute to articles on an open-source encyclopedia, so, too, will they be able to play a hand in the way the Web is searched.

Wales said that his open-source search engine, which will let users influence the ranking of search results, should be released by the end of the year. The engine will also allow users' computers to help crawl the hundreds of millions of pages on the Web, indexing the content of each.

A key component of the architecture of Search Wikia, as the project is called, is in place, Wales said, enabling users to download a piece of software to let individual computers crawl a small section of the Web.

The idea is that the combined power of thousands of small machines indexing the Web could outweigh the searching ability of the vast, centralized crawlers that sit behind engines such as Google's and Yahoo!'s.

"When I opened Wikipedia with only three articles in it, it was a pretty bad encyclopedia," Wales told the Washington Post. "That's where we're going to be in December with the search engine. We'll tell people upfront: 'It's not very good yet.' But we'll open it up to get feedback and community involvement to help us make it better."

Search Wikia would crawl the Web more quickly and efficiently than a traditional engine, he said, but it would also offer opportunities, "at various points along the search process, for people to engage, join and participate in the construction of the search results."

Participating users might, for instance, be able to tag sites using keywords and links that would work in conjunction with traditional algorithms to 'map out' the web in a more human way.

The engine would also expose the workings of the algorithm it used to rank pages in the interests of "openness and transparency," Wales said. Traditionally search engines have kept their algorithms secret.

Wales told a conference of developers in Portland, Ore., that Wikia, the start-up behind the project, has recently acquired Grub, a pioneering web crawler which would be opened up to the software community to make improvements.

LookSmart, the San Fransisco-based firm from which Wikia bought Grub, had agreed to deliver ads to Search Wikia when the engine launched, he said.

Not everyone has expressed enthusiasm for human-based search. Danny Sullivan, one of the editors of the 'Search Engine Land' blog, has said that a "human element" was crucial to the early success of Ask Jeeves, for instance, but that scaling quickly became a problem, and that "over time the machine has reigned supreme."

A comment on the Search Wikia blog said that an open-source search engine would be much more prone to abuse than a site such as Wikpedia, because the potential for for "bad guys" to exploit it for commercial gain was that much greater.

Smaller search engines that are struggling to compete with the larger providers should join the project, Mr Wales said. "A lot of second-tier players understand that competing with Google directly as independent proprietary projects, they'll never catch up," he said. "By banding together using open-source software, they can effectively compete with Google and improve their services."