A Brief History of Search

Introduction

Although Google celebrated its tenth birthday only a few months ago. Search engines have existed in one form or another since 1990. It is interesting to see how search engines have evolved to cope with with the exponential growth of the internet.

Archie

The first search engine was called Archie written by Alan Emtage.  At this time the World Wide Web was in development. It was an FTP search engine and gathered its data by logging on to the sites that it knew about and indexed the data. A similar protocol to FTP called Gopher also spawned its own search engines, called Jughead and Veronica.

The First Web Crawler

The invention of the World Wide Web Project, started in 1990 and came in use in 1993 was a milestone in the history of the internet. Instead of logging into individual websites, the invention of hyper-text where clicking on a link could direct you to a page on the same computer or another computer on the other side of the world.  There was no longer a need to log into separate computers.

It also made it possible for a program to follow the links and index the web. The first web spider or crawler was invented. The World Wide Web Wanderer, was developed in June, 1993 and was able to index the entire web, storing only the URLs but no content information. The front-end, Wandex, display results but they were unfiltered and more or less just a list. Other robots followed, World Wide Web Worm, Jumpstation and the Repository Based Software spider.

The First Modern Search Engine

February 1993, also saw the birth of what why might say is the first truly modern search engine. The Excite was developed by six undergraduate students at Stanford university. It was the first search engine that made an attempt to index  content, using a statistical analysis of word relationships to makes search engines more efficient.

At this stage the computers was still underpowered but the number of www sites was increasing exponentially. The next significant development was called WebCrawler. Started in January of 1994, by Brian Pinkerton, a student at the University of Washington, it was the first search engine to use elements in the head section of the code such as the title and meta information as well as indexing the content of the page. The number of websites index in it database was around 4000 sites (2,250 – 10,000 www hosts around this time) [1] . WebCrawler has been acquired and resold by a number of companies including  AOL in 1995, Excite in 1997 and finally Excite with WebCrawler became owned by Infospace in 2001. The webcrawler still lives on as a meta search engine under the name Spidey.

Also created in 1994, Lycos introduced a number of developments including relevance retrieval, prefix matching, and word proximity in 1994. By 1996, Lycos was  indexing over 60 million documents; making it the largest search engine of the period.

While many search engines were created at this time including: Yahoo!, the next significant developments were introduced by Alta Vista in 1995. It provided advanced searching techniques and natural language support. It was also the first to offer multimedia search for photos, music, and videos.

Inktomi started in 1996 at UC Berkeley. In June of 1999 Inktomi introduced a directory search engine powered by “concept induction” technology. “Concept induction,” according to the company, “takes the experience of human analysis and applies the same habits to a computerized analysis of links, usage, and other patterns to determine which sites are most popular and the most productive.” Inktomi was purchased by Yahoo in 2003.

AskJeeves and Northern Light were both launched in 1997.

September 1998 saw the start of Google. Reportedly powered by Pigeons, the real reason for their success was their link-analysis algorithm, PageRank which calculates the authority of a page based on the authority of the number of links pointing to it. Since then Google has gone from strength to strength introducing a wide range of information-based services, including a product search, Froogle, Translation services, Books, Images, Video, Pay Per Click Advertising Adwords.

Windows entrepreneur, Bill Gates, failed to see the growth in the internet and as a result did not commit to search engines until 2003.  Early versions used the Inktomi search engine to serve queries.  Although later in 2005, Microsoft developed their own search engine database. Despite its late appearance,  it has managed to take the third largest share of search due to the enormous resources of Microsoft behind it and has continued to develop, changing its name to Live in 2006. The live search engine boasts a neural network algorithm.

Web Directories

The oldest directory was founded by Tim Berners-Lee, in 1991. the Virtual Library still exists today. Other directory also formed, such as the ELNET Galaxy. This had links for Telnet and Gopher protocols.

Yahoo also started life as a search directory in February, 1994 as a guide to the web. Built by two PhD students David Filo and Jerry Yang It expanded quickly and categories became subcategories. Today Yahoo incorporate search engine and has grown to take the second largest share of the search market serving around 20% of all search queries made.

1998 saw the introduction of the DMOZ open directory project, this is the largest human edited directory and is one of the most respected for search engine optimisation. Every listing is reviewed by a person before being allowed into the directory and it is one of the most important directories for search engine optimisation (SEO).

References

[1] http://www.nic.funet.fi/index/FUNET/history/internet/en/kasvu.html