ROUT, the first search engine, and NS, the first network news service

Lester Earnest

2017.08.12

[Note: it is conceivable that someone developed a search engine before this one but I have never heard of one. If you think you know of a search program that retrieved documents from a large collection using some combination of words that was initiated before March 1961, please send an email to les at cs.stanford.edu.]

In March 1961, based on my observations of Air Force System 438L, namely a misapplication by IBM of batch processing to military intelligence at the headquarters of the Strategic Air Command near Omaha, Nebraska, I initiated the development of what was evidently the first search engine, though that term did not come into use until about 40 years later. There were no online document collections nor general purpose computer networks then but it was clear to me that it would make sense to take texts directly into a computer from a telegraphic or other such channel and index them automatically so that they could be retrieved based on content. I initially proposed such a system under the name Project ROTGUT (Retrieval Of Teletype Generated Unformatted Text) but for some reason MITRE Corp. management didn’t like that name, so I shortened it to ROUT (Retrieval Of Unformatted Text).

The idea behind ROUT was to create an inverted index based on keywords. As new messages were received they would be searched for keywords and for each such keyword a list of all the documents that used it was compiled. Users could then specify the messages they were interested in by giving a Boolean combination of keywords, possibly with a date constraint, and all matching messages would be promptly retrieved. The retrieval process was quick because performing Boolean operations on document lists using “and”, “or”, or “not” operations promptly produced a list of qualifying texts.

Though my original motivation for creating ROUT was to deal with intelligence messages, for convenience I wanted to do the testing with unclassified material but couldn’t find any online document collections until I stumbled onto a solution. The Air Force’s Project Bluebook had been collecting reports of Unidentified Flying Objects (UFOs) for years and had baskets full of Teletype tape from all over the world and some of which made interesting reading. I understand that they are now available online in Facebook but I will never go there.

No sooner did I get that project started than I was reassigned to work at the Headquarters of the Central Intelligence Agency (CIA) in Langley, Virginia. I then followed the development of ROUT from afar. As far as I know there were no articles written about it in public media, though there were a number of unclassified (but proprietary) reports, which I still have:

L. Earnest, Intelligence message retrieval experiment (Project ROTGUT), MITRE Working Paper W-3779, 1961 Mar. 16.

C. Justice, W. Aldrich, H. Lynch, R. Rander, A description of the ROUT System, MITRE Working Paper W-5802, 1963 Jan. 22.

L. Earnest, Document Retrieval Development, MITRE Corp. Memo 600.5-33, 1963 Oct. 15.

J. Rial, Final report on the evaluation of the ROUT document retrieval system – Proj. 438L, MITRE Tech. Memo. TM-3869, 1963 Nov. 12.

NS News Service. Beginning in 1974 a program called NS (for News Service), created by Martin Frost with input from John McCarthy and me, began operating on our computer at the Stanford Artificial Intelligence Lab (SAIL). It indexed and stored stories from both the Associated Press and New York Times newswires and allowed users to search for recent stories using essentially the same retrieval scheme used in the ROUT search engine discussed above, namely Boolean combinations of keywords. There actually were two differences. Whereas ROUT used preselected keywords, NS regarded all words as keywords except for a short list of extremely common words such as “a, an, the.” Also, whereas ROUT had a thesaurus for searching synonyms of retrieval terms we didn’t provide that in NS.

M. Frost, Reading the wire service news, SAIL Operating Note 72.1, 1974 Oct. 7. http://www.saildart.org/NS.ME[S,DOC]6_blob

NS also allowed users to leave standing queries so that when a relevant story arrived it would automatically send an email notification to the requester. An auxiliary program called HOT allowed users to watch both newswires in real time as stories arrived and were displayed concurrently on separate parts of the user’s display.

The cost of providing this news service was rather low inasmuch as the newswires treated us as a college newspaper and charged only around $25 per month. NS was widely used by people on ARPAnet for general news information until one of the news services found out and forced us to provide it only to SAIL people. 

Aside from providing a general news service, NS played an important role in dealing with the Three Mile Island nuclear accident in 1979. The emergency response team at Lawrence Livermore Lab found that they needed up-to-the-minute information on developments at the site but couldn’t get it until we provided them with NS access.

Also, during the Tiananmen Square protests in 1989, Chinese students in the U.S. wanted to pass information to friends in China but there were no Internet connections there at that time. I set up NS to locate news about China and forward it to a student distribution list so that they could print the stories and fax them home.

Searching ahead. Library scientists soon started using computers to identify and located books and articles. For example, in 1975 Gerard Salton and colleagues developed a vector space model for automatic indexing but as far as I know systems such as that didn’t turn into network search services until later.

G. Salton , A. Wong , C. S. Yang, “A vector space model for automatic indexing,” CACM, v.18 n.11, p.613-620, 1975.11 http://dl.acm.org/citation.cfm?doid=361219.361220

After the creation of ARPAnet and Internet but before the Web, several search engines were developed including Archie, Veronica, Jughead, WAIS and Gopher. The Yahoo! search engine appeared in the early 1990s, shortly after Gopher. Altavista, developed by DEC, was evidently the first search engine to use a web browser and began operating at the end of 1995. It was later purchased by Yahoo!.

Google started later that decade and soon moved into a dominant position. Both Yahoo! and Google were spinoffs from Stanford University computer groups other than SAIL. Marissa Mayer, who reportedly came out of the Symbolic Systems group there and became an early Googler, then become CEO of Yahoo! but has since been displaced. A number of other search engines have appeared over the years and continue to compete with each other.