How the Search Engine Killed the Web Directory

Much of the directory was based on
the whole Internet catalogue. In effect, it was like a paper directory of
websites posted on the web. In September, another directory appeared on the scene,
W3 Catalogue. And in January of 1994, when there were around 600 websites,
David and Jerry’s Guide to the World Wide Web launched.

You’ll know that directory better by
its later name Yahoo! Yahoo! eventually became the world’s top search directory
and the world’s most popular site. These were all web directories. In other
words, they were list of websites with a little information explaining what
each site contained. That was fine in the early days when the web simply didn’t
contain much information. But as the web grew, they became unwieldy.

If you’re trying to find a particular
Shakespeare sonnet, a directory will tell you which sites might contain the
information, but they don’t let you see what’s in each site. They simply tell
you what each site is about and it’s up to you to go to the site to see if it
contains what you’re looking for. So the next step was the search engine, a
system that created an index of pages within sites. There were various simple
search engines early on, but perhaps the first true web search engine, a system
that would allow use of the search through the text contained in web pages
within index websites, was WebCrawler launched in April 1994.

A directory provides minimal
information about a site. A search engine, though, lets user search pages
within sites, a far more useful service. During 1995 and over the next few
years, all sorts of other search engines appeared on the scene; Magellan,
Excite, Infoseek, HotBot, Northern Light, and of course AltaVista which became
hugely popular when it launched late in 1995. Finally, in 1998, Google appeared
on the scene.

By the end of the decade, the writing
was on the wall, search engines were the future and over time search engines
would more or less kill off the directories. Even Yahoo! had to switch. In the
year 2000, Yahoo! began using Google’s index to provide search results to
Yahoo! Searches; then gradually pushed the directory further down the page
until they removed it from their homepage entirely. Yahoo! Directory still
exists today of course, though most users have no idea where. But the search
engines have problems of their own.

As the web grew to around two to
three million websites and hundreds of millions of web pages by the time Google
appeared on the scene, the problem of sorting through the starry amount of data
was becoming overwhelming. Google was based on a revolutionary idea that you
could figure out what a web page was about and whether it was a good match for
someone’s search query, not just by looking at the page itself but also by
looking at links pointing to that page.

Those links could be inside the
website within which the page was found, but could even be pointing to the page
from other websites; sites that the owner of the reference web page might not
even know existed.