History and invention of search engines

The first major search advance was Archie, which beginning in 1990 made it possible to search through a site's file directories. It was created in 1990 by Alan Emtage, Bill Heelan and J. Peter Deutsch, computer science students at McGill University in Montreal. The original intent of the name was “archives,” but it was shortened to Archie. The program downloaded the directory listings of all the files located on public anonymous FTP (File Transfer Protocol) sites, creating a searchable database of file names.

In 1991, Mark McCahill, a student at the University of Minnesota, effectively used a hypertext paradigm to create Gopher, which also searched for plain text references in files. Gopher made it possible to search through online databases and text files. Like Archie, they searched the file names and titles stored in Gopher index systems.

Then, Excite was born in February 1993 as a university project called Architext involving six undergraduate students at Stanford seeking to use statistical analysis of word relationships to improve relevancy of searches on the Internet.

In the same year the graphical Mosaic web browser improved upon Gopher’s primarily text-based interface. About the same time, in June 1993, Matthew Gray then at MIT developed Wandex, the first search engine in the form that is known search engines today. He produced what was probably the first web robot, the Perl-based World Wide Web Wanderer, and used it to generate an index called 'Wandex'. At first, he wanted to measure the growth of the web, and created the WWWW bot to count active web servers. Later, he upgraded the bot to capture actual URL’s.

In October of 1993 Martin Koster created an Archie-Like Indexing of the Web, or ALIWEB, in response to the Wanderer. ALIWEB crawled meta information and allowed users to submit pages they wanted indexed with their own page description.

Jump Station (released in December 1993) used a web robot to find web pages. Jump Station gathered info about the title and header from web pages and retrieved these by using a simple linear search. It was thus the first WWW resource discovery tool to combine the three essential features of a web search engine (crawling, indexing, and searching).

WebCrawler was the first search engine to be widely used, as well as the first to fully index the content on web pages, making every word and phrase searchable. It was developed at the University of Washington and launched in 1994.

Around 2000, Google's search engine rose to prominence. The company achieved better results for many searches with an innovation called PageRank. This iterative algorithm ranks web pages based on the number and PageRank of other web sites and pages that link there, on the premise that good or desirable pages are linked to more than others.

“Yahoo!” started as a traditional web directory in 1994 by two Stanford University graduates, then launching a search engine in 1995. By 2000, Yahoo was providing search services based on Inktomi's search engine. Yahoo! was created by Jerry Yang and David Filo. It started as a listing of their favorite websites but each entry, besides the URL, also had a page description. Yahoo! switched to Google's search engine until 2004, when it launched its own search engine based on the combined technologies of its acquisitions.
History and invention of search engines