News Ticker

How the Search Engine Killed the Web Directory

By Rizwan Ahmad - 16 April 2013 No Comments
In order to understand why links are so important, it may help to understand the problem that search engines use links to solve. It's hard now for many of us to imagine the world without a World Wide Web, but by the end of 1993 there were only around 600 websites and most of those 600 sites were pretty thin; a handful of pages each. In August of that year, when there were just a couple of hundred websites, a well-known computer book publisher O'Reilly and Associates launched what was probably the first commercial directory of the Internet, GNN, the Global Network Navigator.
How the Search Engine Killed the Web Directory
Image is licensed under CC Attribution  
Much of the directory was based on the whole Internet catalogue. In effect, it was like a paper directory of websites posted on the web. In September, another directory appeared on the scene, W3 Catalogue. And in January of 1994, when there were around 600 websites, David and Jerry's Guide to the World Wide Web launched.

You'll know that directory better by its later name Yahoo! Yahoo! eventually became the world's top search directory and the world's most popular site. These were all web directories. In other words, they were list of websites with a little information explaining what each site contained. That was fine in the early days when the web simply didn't contain much information. But as the web grew, they became unwieldy.

If you're trying to find a particular Shakespeare sonnet, a directory will tell you which sites might contain the information, but they don't let you see what's in each site. They simply tell you what each site is about and it's up to you to go to the site to see if it contains what you're looking for. So the next step was the search engine, a system that created an index of pages within sites. There were various simple search engines early on, but perhaps the first true web search engine, a system that would allow use of the search through the text contained in web pages within index websites, was WebCrawler launched in April 1994.

A directory provides minimal information about a site. A search engine, though, lets user search pages within sites, a far more useful service. During 1995 and over the next few years, all sorts of other search engines appeared on the scene; Magellan, Excite, Infoseek, HotBot, Northern Light, and of course AltaVista which became hugely popular when it launched late in 1995. Finally, in 1998, Google appeared on the scene.

By the end of the decade, the writing was on the wall, search engines were the future and over time search engines would more or less kill off the directories. Even Yahoo! had to switch. In the year 2000, Yahoo! began using Google's index to provide search results to Yahoo! Searches; then gradually pushed the directory further down the page until they removed it from their homepage entirely. Yahoo! Directory still exists today of course, though most users have no idea where. But the search engines have problems of their own.

As the web grew to around two to three million websites and hundreds of millions of web pages by the time Google appeared on the scene, the problem of sorting through the starry amount of data was becoming overwhelming. Google was based on a revolutionary idea that you could figure out what a web page was about and whether it was a good match for someone's search query, not just by looking at the page itself but also by looking at links pointing to that page.

Those links could be inside the website within which the page was found, but could even be pointing to the page from other websites; sites that the owner of the reference web page might not even know existed. 
Rizwan Ahmad
About the Author:
This article is contributed by UK SEO Company and posted by Rizwan Ahmad Author and founder of, He is a tech blogger from India and he loves to share his thoughts by writing articles on this site to the different topics related to technology world,

No Comment to " How the Search Engine Killed the Web Directory "