Open Directory in Search of the Best of the Web
A grass-roots effort to harness the power of the masses has recently emerged to solve one of the most intractable problems of the Internet--indexing the vastness of the Web. The Open Directory project has mushroomed in the last year, suddenly becoming a credible competitor to the dominance of search leader Yahoo.
Created by two programmers at Sun Microsystems in their spare time in the summer of 1998, the Open Directory uses thousands of volunteers to create a master directory of the best of the Web. It now contains more than 1 million Web pages and has been adopted by such popular search sites as Lycos, HotBot, Netcenter and starting this month, America Online.
Much of the Open Directory’s sudden rise is due to its acquisition by Netscape Communications in November and AOL’s subsequent acquisition of Netscape in March. With those customers alone, the directory is a big-league player.
And as a free product offered to any Web site that wants it--even to search specialists--it has been adopted as a feature by 70 other search engines.
The approach has become one of the few viable ways for search directories to keep up with the explosive growth of the Web without resorting to automated methods that lack human judgment.
By leveraging the masses, the Open Directory (https://dmoz.org) has become one of the banner holders in the burgeoning “open content†movement that relies on Internet users to share their knowledge.
The movement draws its inspiration from the “open source†movement, in which programmers volunteer to create software that is given away. Just as the open-source operating system Linux has shaken up the software world and challenged Microsoft, the open-content model of the Open Directory is beginning to challenge the traditional powers of the Web.
“It’s a very big threat to Yahoo,†said Danny Sullivan, editor of Search Engine Watch, an online newsletter. “There’s no way they can ignore it. It’s a major change in the way search services are operating.â€
*
With Yahoo reporting quarterly earnings that went way beyond analysts’ estimates and its leading position on the Web with more than 33 million visitors each month, the challenge from the Open Directory has a lot of ground to cover.
Yahoo says its directory has never been about quantity, but about quality.
Srinija Srinivasan, editor in chief of Yahoo’s directories, said the company has succeeded over the years with its strategy of using a small staff of professional editors. She said only with a highly organized staff can it produce the cohesive, high-quality directory that people expect.
“Our goal from day one has been to deliver what users are looking for, and that doesn’t mean delivering every page on the Web,†said Srinivasan, also known as Yahoo’s “chief ontologist.†“The way I measure our success is whether people come back to our site, and they do.â€
But even Srinivasan conceded that as the Web grows, maintaining the directory--weeding out dead links, finding the best sites and indexing submissions from the public--gets harder and more time-consuming.
The problem that all search engines have faced in recent times is simply that the Web is growing too fast.
Automated search engines, such as AltaVista, Excite and Northern Light, have tried to keep up through a strategy of using software programs that “crawl†the Web, taking note of every site they stumble on.
The advantage of the crawling system is that it can encompass hundreds of millions of pages. The disadvantage is that their indexes grow so large that a search turns up thousands of results, most of which are irrelevant.
The other type of search engine relies on humans to select and index only the most useful sites, creating a smaller handpicked directory. The process takes time and requires people, but tends to yield higher-quality, more relevant results.
Srinivasan would not disclose how many editors work at Yahoo or the number of listings in its directory, but said her staff is largely made up of generalists, who are able to cover a variety of subjects.
About.com takes a slightly different approach, seeking out hundreds of specialists, each of whom is detailed to a specific category--doctors covering medicine and mechanics covering cars.
*
The Open Directory relies on thousands of volunteer editors, covering a wide range of knowledge and skills.
In recent years, there have been constant complaints against Yahoo about how long it takes to get a page listed in the directory and the volume of links in the directory that don’t work.
“The reality is that Yahoo just doesn’t have the time†to keep its directories current, said Sullivan of Search Engine Watch.
Rich Skrenta and Bob Truel of Sun, creators of the Open Directory, thought that by distributing the workload among thousands of volunteers, they could avoid the overload that Yahoo faces.
Each volunteer collects a list of Web sites under certain topics and maintains those categories by culling dead links and finding new ones. With enough volunteers, even the vastness of the Web could be handled.
Ted Casey, general manager of HotBot, which uses the Open Directory, said mobilizing large numbers of people connected by the Internet may be the only way to deal with the geometric growth of the Web, which has been nearly doubling every year.
The directory’s volume of listings, which will soon exceed Yahoo’s if it hasn’t already, and its free price tag has allowed search companies such as Lycos and HotBot to add the Open Directory to their regular Web-wide search.
It has also allowed new companies, such as Los Angeles-based Oingo, to create their own type of search product while using the listings in the Open Directory.
The result is that the directory has achieved a wide reach in remarkably short order.
“For the first time, there is a very real possibility that a Web site could get more traffic from being listed in the Open Directory than from Yahoo,†Sullivan said. “Yahoo became like a semiofficial guide to the Web. Suddenly the Open Directory could supplant them in that role.â€
Even if the Open Directory were bigger or better, said Yahoo’s Srinivasan, the company’s role as a central point of the Web has already moved far beyond mere search.
Yahoo is a Web portal now, offering free e-mail, free Web sites, stock quotes, news, instant messaging and personal scheduling.
“Yahoo was a search engine and directory 4 1/2 years ago because that’s what people wanted to do,†she said. “It’s still what people want to do, but it’s just one piece of the whole.â€
But the idea of mobilizing the masses to create Web content is beginning to spread beyond mere search as well.
Some of the most interesting Web companies to start up this year, such as Epinions.com and HotLinks, have adopted a similar strategy of using the Web masses, although none gives away the database of information it collects as the Open Directory does.
Epinions, which launched in August, is a collection of online product reviews written by consumers. The company gives reviewers a small royalty each time someone reads their reviews.
HotLinks is a search engine that collects its users’ browser bookmarks and lets others search them to find Web sites. HotLinks, which had its official launch last week, already has a directory listing about 500,000 pages.
Infoseek, the second-most-popular search site on the Internet, created a directory of its own last month, called the Go Guide, that is put together by users.
Casey, of HotBot, said the companies building on mass participation are, in many ways, simply reflecting the way the Web grew. Not a bad model to imitate.
“The open-content model is opening the door to other opportunities,†Casey said. “More and more, you will see ways that users will be able to contribute content.â€
(BEGIN TEXT OF INFOBOX / INFOGRAPHIC)
Building a Directory
The Open Directory, started in June 1998, has more than 1 million Web sites in its index-- considered near or above the volume of Yahoo’s 5-year-old directory-- and is fast becoming a viable candidate for the ultimate guide to the Web. A look at its growth:
* Nov. 18, 1998: Netscape acquires Open Directory (then called NewHoo)
4,700 contributors
84,000 sites
* March 17, 1999: AOL acquires Netscape
8,243 contributors
401,000 sites
* April 19, 1999: Lycos/HotBot license the Open Directory
9,298 contributors
453,000 sites
* June 24, 1999: Netscape launches new Netscape Search service with Open Directory
12,857 contributors
675,000 sites
* Oct. 6, 1999: AOL Search Open Directory for AOL and AOL.com
16,500 contributors
1 million sites
Source: Netscape Communications
