all but war is simulation:

the military-entertainment complex

Tim Lenoir

Stanford University

To appear in Configurations, Fall, 2000

The box office smash from spring 1999, The Matrix, projects a vision of a world in which "real" world objects are actually simulations emerging from streams of bits. Finding themselves pursued on a rooftop with no escape except a helicopter, the movie's hero, asks his guide, "Can you fly that thing?" "Not yet," she says, as she calls their home base systems administrator for software that uploads just in time.

In a similar vein, one of Intel's 1999 ads for the Pentium II processor articulates the consumer's desire for ever faster uploads and ultimately for fusing the digital and the real. As a skydiver plummets to earth alternating anxious glances between the camera and his chute, which appears one agonizing row of pixels at a time on the screen, the voiceover asks "Time for a Pentium II Processor?"

Such images are amusing fantasies. They are also reminders that we are becoming immersed in a growing repertoire of computer-based media for creating, distributing and interacting with digitized versions of the world. In numerous areas of our daily activities, we are witnessing a drive toward fusion of digital and physical reality: not the replacement of the real by a hyperreal—the obliteration of a referent and its replacement by a model without origin or reality—as Baudrillard predicted, but a new country of ubiquitous computing in which wearable computers, independent computational agent-artifacts, and material objects are all part of the landscape.

To paraphrase the description of the matrix by Gibson in Neuromancer, data is being made flesh.[1] These new media are reshaping the channels of our experience, transforming our conception of the "real," redefining what we mean by "community," and some would maintain, what we mean by our "selves."[2] As we come to entrust more of our lives to Internet communications and as we spend more time in virtual, electronic space, our notions of materiality and reality will inevitably change.

I am intrigued by the notion that we are on the verge of a new renaissance, which like the Renaissance of the fourteenth and fifteenth centuries is deeply connected with a revolution in information technology. That most celebrated Renaissance is frequently heralded as the birth of humanism. I sympathize with several contemporary theorists who characterize our renaissance as heralding a posthuman era in which the human being becomes seamlessly articulated with the intelligent machine. In the posthuman state, there are no demarcations between bodily existence and computer simulation, between cybernetic mechanism and biological organism.[3]

A minimal condition for a new, “post”-human condition would certainly be a fundamental shift in our notions of material reality. By exploring the recent history of what I am calling the military-entertainment complex I hope to suggest some of the pathways through which a so-called posthuman future might emerge. Our experience of materiality is deeply tied to technologies that affect how we experience space and time and how we use our bodies. Changes in these technologies have a profound impact on our sense of the real.

A sign of these posthuman times is the rapid fusion of the digital and the real going on around us, taking place in personal digital assistants, cell phones, and Palm Pilots™ (about to become wearable servers) that accompany us throughout the day. The sign is more clearly perceptible perhaps in technologies such as web-based personal shopping assistants that learn our preferences and then crawl the web in search of software upgrades, information, and commodities that define us as consumers of information.

The Fusion of the Digital and the Real

No less important for effecting these changes in our notions and experience of material reality will be the implementation of research and development efforts to embed information technologies in the world around us, in objects other than communications devices. For a generation we have been used to thinking of the computer as the symbol of the information revolution, but one way to think about our present stage within this revolution is that the computer is in fact disappearing. If developments funded by the military research agencies such as DARPA at several research universities and at organizations like Xerox PARC come to pass, that large box we are used to staring into all day will vanish. In its place will be a world filled with special purpose chips, “smart” devices, and agents that interact with us constantly. These agents and devices will not sit on our desktops, but rather will be embedded in wearable microdevices and implants, leading to a world of ubiquitous computing.

Since 1996, for instance, the DARPA Smart Modules program has been developing and demonstrating novel ways of combining sensors, microprocessors, and communications in lightweight, low-power, modular packages that offer war fighters and small fighting units new methods to enhance their situational awareness and effectively control their resources on the battlefield. Smart modules are integrated into personal and portable information products that sense, compile, analyze, display, compare, store, process, and transmit information. The resulting products create opportunities to exploit data-rich battlefield environments at the individual war fighter level. Instead of the normally limited set of information resources at the disposal of the individual war fighter (maps, compasses, hand-held global positioning systems) and limited connectivity (primarily voice radio) to information infrastructures, Smart Modules allow individuals to better perceive their environment (see, hear, and feel the electromagnetic spectrum), augment their ability to remember and make decisions through use of electronic devices, and provide mechanisms for connection to wireless distributed data networks. Modular information products are part of clothing, worn on a belt or put into a pocket. These products will capitalize on current rapid developments in micro electromechanical systems, head-mounted and small direct-view displays, optoelectronics, integrated sensors and video modules, energy storage, and low-power electronics.

DARPA’s “smart matter” programs go beyond the wearable modular communication devices and information systems described above. Smart Matter research is based in large part on MEMS (micro electromechanical systems), very small sensors and actuators that are etched into silicon or other media using photolithography-based techniques. Integrated with computation, these sensors and actuators form a bridge between the virtual and physical worlds, enabling structures to dynamically respond to conditions in their environment. Smart materials and structures mimic the natural world where animals and plants have the clear ability to adapt to their environment in real time. Designers and promoters of these “biomimetic” technologies dream about the possibilities of such materials and structures in the man-made world, engineering structures operating at the very limit of their performance envelopes and to their structural limits without fear of exceeding either. “Smart” structures could give maintenance engineers a full report on their performance history, as well as the location of any defect as it occurs. Furthermore, that same structure could be given the capacity to self-repair or the ability to counteract unwanted or potentially dangerous conditions such as excessive vibration.

Sutherland’s Holy Grail

The nexus between computer simulation and virtual reality for military purposes and the entertainment industry has a thirty-five year history tracing its origin to Ivan Sutherland's head-mounted display project.[4] The project usefully illustrates both the synergy between problem-focussed environments of industry and government-funded (military and otherwise) projects, and the less product-oriented research focus of university work that spills across disciplinary boundaries. In 1966 Sutherland moved from ARPA to Harvard as an associate professor in applied mathematics. At ARPA Sutherland had participated in implementing J.C.R. Licklider's vision of human-computer interaction, and he returned to academe inspired to pursue his own program of extending human capabilities.[5] One such project was his head-mounted display.

Funding for this project came from a variety of sources: military, academic, industry. The CIA provided $80,000, and funding support was also provided by ARPA, the Office of Naval Research, and Bell Labs. Bell Helicopter provided equipment. The Air Force provided a PDP-1 computer, while MIT Lincoln Labs, also under an ARPA contract, provided an ultrasonic head-position acoustic sensor. Sutherland's experiments built on the network of personal and professional contacts he had developed at MIT and at ARPA as well as on earlier work on head-mounted displays at the Bell Helicopter Company, centered on input from servo-controlled cameras which would move with the user's head and thus move the user's visual field. At Bell Helicopter Company, the head-mounted display was coupled with an infrared camera that would give military helicopter pilots the ability to land at night in rough terrain. An infrared camera, which moved as the pilot's head moved, was mounted on the bottom of a helicopter. The pilot's visual field was the camera's.

The helicopter experiments demonstrated that a human could become totally immersed in a remote environment through the "eyes" of a camera. With the viewer inside a building, a camera was mounted on the roof, with its field of view focused on two people playing catch. The viewer immediately responded to the motion of the ball, moving the camera to follow the game of catch by moving his head. Proof of the viewer's involvement in this remote environment came when the ball was thrown at the camera and the viewer ducked. When the camera panned the horizon, the viewer reported a panoramic skyline. When the camera looked down to reveal that it was "standing" on a plank extended off the roof of the building, the viewer panicked.[6]

In 1966, as an associate professor at Harvard, Sutherland and his student, Bob Sproull, took the "Remote Reality" vision systems of the Bell Helicopter project and turned them into "Virtual Reality" by replacing the camera with computer-generated images.[7] The first such computer environment was no more than a wire-frame room with the cardinal directions—North, South, East, and West—initialed on the walls. The viewer could "enter" the room by way of the West door, and turn to look out windows in the other three directions. What they called the "Head-Mounted Display" later became known as Virtual Reality.

Sutherland later recalled that at the time he formulated the head-mounted display project he was clear that there was no hope of immediately realizing it. But the project was important, he recalled, "as an 'attention focuser' which defined a set of problems that motivated people for a number of years." VR was a target impossible to reach. It provided a holy grail, "a reason to go forward and push the technology as hard as you could. Spinoffs from that kind of pursuit are its greatest value."[8]

In Sutherland's view, the most important spinoff from such projects were the students; the personal and professional connections supported future work in the area. Sociologists of science talk about the importance of "core sets" of individuals who define the intellectual and technological direction of a domain. Certainly the bevy of students Evans and Sutherland trained constitute one of the most dramatic examples of such a core set in the history of computer science. Among the students who worked on the "holy grail" of VR with Sutherland at Harvard were Charles Seitz, Robert Sproull, Ted Lee, Dan Cohen, and Quintin Foster. In 1968 Sutherland left for Utah, where he joined the Computer Science Department at the University of Utah founded by Dave Evans in 1965, the first computer science program to focus on graphics and graphical interfaces. Sutherland had known Evans from his ARPA days, and together they founded Evans & Sutherland Computer Corporation in 1968, which manufactured graphical display systems and constructed military flight and tank simulators under government contract. A number of Evans' and Sutherland's students worked on an ARPA-supported project on 3-D graphics, and several worked at Evans & Sutherland on simulations. Of the original Harvard group several came with Sutherland to form Evans and Sutherland, including Chuck Seitz who joined the faculty in 1970, and remained until 1973 when he moved to Cal Tech and founded Myricom with Dan Cohen, another of the original Harvard team who contributed to the head-mounted display. The interaction between the research on basic problems and development-directed hardware and software systems for government and military projects at E&S was an important feature of work at Utah.

At Harvard briefly and then from 1968-1974 at the University of Utah Sutherland set out a research program for work in interactive computer graphics that guided the field in much of its early development and continues to be relevant for the discussion of current trends in medical graphics.[9] For Sutherland the display screen was to be considered a window, through which the user looks at a virtual, conceptual 3-D universe. Sutherland’s program called for inventing ways to make the image in the window more and more realistic, until at last it becomes indistinguishable from the image in a real window, a real window augmented, that is, by “magical” powers of scaling, labelling, rotating and cross-sectioning.

In addition to visible realism Sutherland sketched two other directions. A second class of graphical applications related purely to representing abstractions, such as force fields, molecules, mathematical objects, and data graphing, for which visual realism is irrelevant. But in this context Sutherland considered that it would be useful to extend the domain of information available to the user by incorporating information from other sensory modalities. Sutherland coined the term “virtual worlds” for systems in which users are immersed in scenes created completely by computer graphics; and he urged that the goal of this work should be to make the objects in the scene “look real, sound real, feel real, and move realistically as the user interacts with them.”[10]

The third form of interactive graphics Sutherland outlined is one particularly relevant to current medical applications such as virtual surgery: namely, the ability to superimpose abstract representations on an object, as in cartography, where abstractions are superimposed on a realistic rendering of a geographical space. One of Sutherland's first attempts at practical application of the head-mounted display was in fact in pursuit of this third form of graphical interface. The first published research project deploying the head-mounted three-dimensional display engaged problems of representing hemodynamic flow in models of prosthetic heart valves; the goal of this research was to generate the results of calculations involving the application of physical laws of fluid mechanics, a variety of numerical analysis techniques in order to generate a synthetic object that one could walk towards, move around, or even into.[11]

The period from the late 1960s through the late 1970s was a golden era of computer graphics at Utah, and students of the Utah ARPA-funded program contributed to a number of exploratory systems in computer graphics and the identification of key problems for future work. Among these were various efforts to develop fast algorithms for removing hidden surfaces for color and 3-D graphics, a problem identified as a key computational bottleneck.[12] Two important contributions in this field were made by students of the Utah program, including an area search method by Warnock,[13] and a scan-line algorithm developed by Watkins which was constructed into a hardware system.[14] Perhaps the most important breakthrough came just at the close of the decade with Henri Gouraud's development of a simple scheme for continuous shading.[15] Unlike polygonal shading, where an entire polygon was shaded with a single level of grey, Gouraud's scheme involved interpolation of surface normals to describe continuous shading across a single polygon, and thus a closer approximation to reality. The effect made a surface composed of discrete polygons appear to be continuous.

The list of alumni from the Utah program in the years between 1968-1978 is impressive indeed. Below are listed a few members of this illustrious group and their accomplishments.

TABLE 1. Select Alumni of the University of Utah’s Computer Graphics Program

Name	Affiliation	Accomplishments
Alan Kay	Ph.D. 1969	Developed the notion of a graphical user interface at Xerox PARC, which led to the design of Apple MacIntosh computers. Developed Smalltalk. Director of Research, Disney Imagineering
John Warnock	Ph.D. 1969	Worked on the Illiac 4 Project, a NASA space flight simulator, and airplane simulators at Evans & Sutherland. Developed the Warnock recursive subdivision algorithm for hidden surface elimination. Founder of Adobe Systems, which developed the Postscript language for desktop publishing.
Chuck Seitz	Faculty 1970-73	Pioneer in asyncronous circuits. Co-designer of the first graphics machine, LDS-1 (Line Drawing System). Designed the Cosmic Cube machine as a research prototype that led to the design of the Intel iPSC. Founder of Myricom Corp.
Nolan Bushnell	B.S. 1969	Developed the table tennis game Pong in 1972, which launched the video game industry. Founder of Atari, which became the leading company in video games by 1982.
Henri Gouraud	Ph.D. 1971	Developed the Gouraud shading method for polygon smoothing - a simple rendering method that dramatically improved the appearance of objects.
Ed Catmull	Ph.D. 1974	Pioneer in computer animation. Developed the first computer animation course in the world. Co-founder of Pixar Animation Studios, a leading computer graphics company which has done work for LucasFilm and was recently involved in the production of the movie Toy Story. Received a technical Academy Award (with Tom Porter, Tom Duff, and Alvy Ray Smith) on March 2, 1996 in Beverly Hills from the Academy of Motion Picture Arts and Sciences (AMPAS) for "pioneering inventions in Digital Image Compositing".
Jim Clark	Ph.D. 1974	Rebuilt the head-mounted display and 3-D wand to see and interact with 3-dimensional graphic spaces. Former faculty at Stanford University. Founder of Silicon Graphics Inc., Netscape Communications Corporation, and most recently Healtheon.
Bui Tuong-Phong	Ph.D. 1975	Invented the Phong shading method for capturing highlights in graphical images by modeling specular reflection. Phong's lighting model is still one of the most widely used methods for illumination in computer graphics.
Henry Fuchs	Ph.D. 1975	Federico Gil Professor, University of North Carolina at Chapel Hill. Research in high-performance graphics hardware; 3D medical imaging; head-mounted display and virtual environments. Founder of Pixel Planes.
Martin Newell	Ph.D. 1975; Faculty 1977-79	Developed procedural modeling for object rendering. Co-developed the Painter's algorithm for surface rendering. Founder of Ashlar, Inc., which develops computer-assisted design software.
James Blinn	Ph.D 1978	Invented the first method for representing surface textures in graphical images. Scientist at JPL, where he worked on computer animation of the Voyager fly-bys.

The work of these individuals alone suggests the high level of fundamental research that was done at the University of Utah under federally sponsored projects in a variety of graphics fields, including surface rendering, simulations, computer animation, graphical user interface design, and early steps toward virtual reality.[16] The number of significant commercial firms generated by the members of this group is astounding. No less than 11 commercial firms, several of which ship more than $100 million in product annually, were the offspring of the Utah program.

Sustaining the Graphics Revolution

Many of these firms have their own research divisions and have contributed importantly to the fundamental research base in computer graphics (both hardware and software) that has been essential to the take-off of VR. But here, once again, the importance of long-term government support, particularly by DARPA, to sustaining innovative research directions emerges as clearly as in our earlier example. The case of Atari illustrates this point dramatically. Founded by Utah graduate in computer science, Nolan Bushnell, Atari at one point in it history was the fastest growing company in America. Started in 1972 with an initial investment of $500, Atari reached sales of $536 million in 1980. During the late 1970s and early 1980s Atari hosted exciting developments in software and chip design for the home entertainment market, and a joint venture with LucasFilm in 1982, in which Atari licensed and manufactured games designed by LucasFilm, established cross-pollination between videogames and film studios. Atari was also a center of developments in VR, and several of the pioneering figures in the VR field got their start at Atari. For instance, Warren Robinett, who has directed the head-mounted display and nanomanipulator projects at the University of North Carolina, Chapel Hill (discussed below), developed the extremely popular videogame Adventure at Atari from June 1977 through November 1979. Jaron Lanier, who developed the DataGlove in 1985, got his start by creating the videogame Moondust, the profits from which Lanier used to launch VPL-Research in 1984, the first commercial VR company.

In 1980 Atari created its own research center, directed by Alan Kay, who came over from Xerox PARC and assembled a stunning team of the best and brightest in the field of interface design and VR research. Kay's team at the Atari Research Lab included Brenda Laurel (who had been at Atari since 1979), Scott Fisher, who had studied with Nicholas Negroponte at MIT before coming to Kay's lab to work on visual displays and virtual reality, and William Bricken, a recent Ph.D. from Stanford in computer science and educational psychology.

But Atari fell on hard times. Having reached annual sales of $512 million in 1980, Atari registered $536 million in losses for 1983. The Atari Research Lab was, obviously, one of the casualties of the economic crash in the video game industry (and computer industry more generally). Most of the people working in VR at Atari either migrated to work in federally funded VR projects, like Jaron Lanier, who created VPL-Research in 1984 and landed a government contract to build the DataGlove for NASA. What emerges from this example is not that federal projects provided fortunate safety nets for failed industry initiatives, but more importantly that centers such as NASA Ames and UNC had the right mix of basic research and long-term vision to move the technology forward. Thus, Scott Fisher moved from Atari to NASA Ames where he directed the Virtual Environment Workstation Project and VR project. Joining Fisher were Warren Robinett and Brenda Laurel. As noted above Robinett eventually moved from NASA to Chapel Hill in 1989. William Bricken moved from Atari to Advanced Decision Systems, where he pioneered high-performance inference engines, visual programming systems, and instructable interfaces, then on to Autodesk Research Lab, where he developed the Cyberspace CAD prototype of virtual reality. Bricken then moved from industry to the University of Washington's Human Interface Technology Laboratory, where he designed and implemented the Virtual Environment Operating System and interactive tools of the VR environment.[17] There was little question that the continued development of virtual reality technology in the 1980s was not something industry was prepared to do on its own: indeed Lanier's failed efforts to market for Nintendo a consumer entertainment version of the DataGlove, called PowerGlove, demonstrated that the time was not yet right for a sustained industry push. Federal support was crucial to building the array of hardware and software necessary for industry to step in and move VR forward. The impressive synergism of federally funded projects and industry developments bringing about the emergence today of the new VR technologies in surgery and other fields would not have been possible without sustained federal funding in centers where the different components of VR work were developed in tandem. As several pioneers in the field observed in a 1991 senate hearing, the merging of the substantially different technologies at stake in virtual worlds could not be undertaken by commercial interests whose horizon of return on investment is short, particularly while the technologies at issue remained in a precompetitive situation for so many years.[18] It is instructive to explore how a sustained mixture of government, industry, and university-based research and development turned the dim portrait of the future depicted in these 1991 senate hearings into the extremely bright picture of the late 1990s.

By the mid-1980s it was universally acknowledged that the creation of virtual worlds technology depended upon developments in several fields, including computer architectures, processors, operating systems, and languages. DARPA funding played the crucial role in these initiatives. One critical turning point for enabling this next phase of development was the DARPA VLSI (very large systems integration) and reduced instruction set computing (RISC) programs begun in the late 1970s. For the first 15 years of its life, the microprocessor improved its performance by an impressive 35% per year. But these performance gains began to slow down, and increasing chip fabrication costs led DARPA program managers to be concerned about future growth. In 1976, they commissioned a RAND study on the problem.[19]

The study showed that the U.S. computer technology environment of the mid-1970s was characterized by (1) a tapering off in the rate of improvement in computer performance as the marginal costs rose and marginal gains from extending prevailing technologies declined; (2) extensive insulation of commercial microelectronics firms, concentrating on their own proprietary developments, from academic communities which were limited in their access to advanced equipment and industry technologies; and (3) exponential growth in the cost of equipment and of implementing device design, as industry concentrated on incremental efforts to pack more gates and transistors into semiconductor devices. The authors of the study also realized that university engineering and computer science departments were getting shut out of much of the microelectronics revolution because they couldn't afford the equipment necessary to manufacture silicon chips. Even those universities that could afford some equipment could never keep up with the rapidly advancing state of the art.[20]

It was in this environment that DARPA originated the VLSI RISC programs. Through his relations with the academic community going back to the early 1970s, Dr. Robert E. Kahn was aware of both the technology potentials of work being done at academic centers of excellence in computer science, and of the cost and limits placed on their ability to implement, validate, and demonstrate their work because of the proprietary practices of industry.[21] The VLSI and RISC programs were undertaken specifically to revitalize and tap creativity in the academic community, which had played an important role in earlier computer and semiconductor developments but which had a declining role by the mid-1970s due to its increasing distance from technology developments in industry. As a result of research at universities and industrial laboratories supported by the DARPA programs, performance gains began to increase by 1987 to about 55% per year -- a doubling of performance every 18 months.[22]

[Figure 1: The Development of Processing Power and Memory]

RISC processors have advanced the field of interactive graphics and contributed significantly to the development of VR. Silicon Graphics, co-founded by Jim Clark in 1982, was an early adopter of RISC processors and led in the recent development of high-end graphics, including virtual reality. Clark joined the Stanford engineering faculty in 1979 having done his Ph.D. with Ivan Sutherland on problems related to the head-mounted display. Clark worked with John Hennessy and Forrest Baskett on the Stanford VLSI program and was supported by DARPA for a project on the Geometry Engine, the goal of which was to harness the custom integrated-circuit technology of MIPS to create cost-effective high-performance graphics systems. In 1981 Clark received a patent for his Geometry Engine, the 3-D algorithms built into the firmware that allow the unit to serve up realtime interactive 3-D. The patent on the Geometry Engine formed the basis of Silicon Graphics founded in 1982 with Kurt Akeley, then a research assistant working with Clark at Stanford. Clark also invented the GraphicsLibrary,which is the graphics interface language used to program SGI's computers. These systems offered built-in 3D graphics capabilities, high speed RISC processors and symmetrical (multiple processor) architectures. The following year in 1983, SGI marketed its first graphics terminal, the IRIS 1000 graphics terminal.

The development of Silicon Graphics not only shows that federal funding initiatives have had major impacts on the economy. It also represents the contribution of commercial developments to the field of interactive graphics and VR. Silicon Graphics, Evans & Sutherland, HP, Sun Microsystems, DEC, and others have generated products enabling simulations of all sorts, scientific visualizations, and CAD programs for engineering. No less significant has been their contribution to the entertainment industry, particularly to the film and video game industries. Indeed, as I have noted above, the entertainment industry has been a major stimulus to graphics throughout its history, in providing sources not only of employment and markets for products but also of substantial research contributions.[23] The relationship between these different partners has been mutually enriching; the arrows of influence point in both directions.

Several spectacular examples of the contribution of the entertainment industry to graphics might be discussed here, but one of the most widely appreciated is RenderMan, developed by Pixar Animation Studios. Ed Catmull, another alumnus of the Utah graphics program in the 1970s, joined Alvy Ray Smith in the computer graphics lab at LucasFilm in 1979. Catmull and Smith had colloborated on the integrated alpha channel in 1977 at the New York Institute of Technology, a fundamental technology in computer graphics.[24] Smith then went on to direct the genesis scene of LucasFilm's Star Trek II, a sequence several minutes long generated by computer graphics depicting the spread of life across a new world. In the view of George Lucas and his organization, such work signaled that computer animation was finally coming of age as a tool for building movies. To realize the dream of constructing an entire film from computer-generated material, Smith and Catmull recruited a number of young computer-graphics talents to LucasFilm, among them, Loren Carpenter from the Boeing Company in Seattle, Washington, who had studied Mandelbrot's research and then modified it to create realistic fractal images. At the 1980 SIGGRAPH conference Carpenter had presented a stunning film entitled "Vol Libre," a computer-generated high-speed flight through rugged fractal mountains. In 1981 Carpenter wrote the first renderer for Lucasfilm, called REYES (Renders Everything You Ever Saw), which was the beginning of RenderMan.

In 1986 the computer graphics division of LucasFilm's Industrial Light and Magic was spun off as a separate company, Pixar, with Catmull as president and Smith as vice-president. Under their direction work continued at Pixar on developing a rendering computer. Pat Hanrahan joined the REYES machine group at Pixar in 1986. At the University of Wisconsin and then at the New York Institute of Technology Computer Graphics Laboratory, where he was Director of the 3D Animation Systems Group, Hanrahan published a number of path-breaking papers on methods of volume rendering, including papers on ray-tracing algebraic surfaces and beam-tracing polygonal surfaces. Hanrahan joined Robert Drebin and Loren Carpenter in developing the first volume-rendering algorithms for the Pixar image computer.[25] These algorithms were quite different from earlier approaches in that they created images directly from three-dimensional arrays without the intermediate steps of converting to standard surface representations such as polygons. Hanrahan was responsible for the interface as well as the rendering software and the graphics architecture of RenderMan.

The rendering interface of the system evolved into the RenderMan standard now widely used in the movie industry. The RenderMan standard describes everything the computer needs to know -- the objects, light sources, cameras, atmospheric effects, and so on -- before rendering a 3D scene. Once a scene is converted to a RenderMan file, it can be rendered on a variety of systems, from Macs to PCs to Silicon Graphics Workstations. This opened up many possibilities for 3D computer-graphics software developers. With RenderMan all the developer had to do was give the modeling system the capability of producing RenderMan-compatible scene descriptions. Once it did this, then the developer could bundle a RenderMan rendering engine with the package, and not worry about writing a renderer. Another strength of RenderMan is its "shaders," pieces of programming code for describing surfaces, lighting, and atmospheric effects. The spatial texture of an object is generated by the computer in 3D space. In contrast to most texture-mapping techniques which map the texture to the outside surface of the object, Hanrahan's procedural textures run completely through the object in 3D, so that if, for example, a cube of wood is sectioned, you see wood grain running through the whole cube. When the initial specification of RenderMan was announced, over 19 firms endorsed it, including Apollo, Autodesk, Sun Microsystems, NeXT, MIPS, Prime, and Walt Disney.

RenderMan was used in creating Toy Story, the first feature-length computer-animated film, the dinosaurs in Jurassic Park, the cyborg in Terminator 2, and numerous other major effects. But this powerful tool has not been limited to use in the film industry. It has also been an important tool in recent work on scientific visualization and volume rendering in a number of fields in science, engineering and medicine. Moreover, the hardware and software components are not the only things that have circulated between industry and academe. The people have circulated too. Thus Ed Catmull and Alvy Ray Smith moved from academic environments of NYIT and Berkeley (in Smith's early career) to LucasFilms, and Pixar; Pat Hanrahan, after starting at NYIT and then Pixar, moved back to an academic lab, first as an associate professor at Princeton, and more recently as professor at Stanford, where he has gone on to contribute to several areas of graphics, including development of applications of the Responsive Workbench, a 3D interactive virtual environment workspace, to areas of scientific visualization, architecture, and medicine. The work in Hanrahan's lab on the workbench has been a cooperative project between Stanford University and the GMD (the German Institute for Information Design), and has been supported by grants from Interval Research Corporation, DARPA (visualization of complex systems), and NASA Ames (virtual windtunnel). Equipment donations have been provided by Silicon Graphics and Fakespace, Inc.

Desire and the Cultural Imaginary

Through films such as Jurassic Park and Toy Story media industries have created desire for computer-generated imagery. Entertainment such as IMAX films, the Star Tours simulation ride at Disneyland, and more recently “Magic Edge” flight simulators all have contributed to creating the desire for sensory immersion experiences. The film Titanic is emblematic of our current desire for digital effects. James Cameron and his organization actually pursued digital effects as ends in themselves—indeed they drew upon effects generated by 19 different visual effects and graphics companies—stealing pride of place from older film techniques, stage effects, and models (which the film also employs to a limited extent). We have come to desire these effects even when the film could be made without them. The desire for “realism” in visual effects forms a feedback loop with whatever technologies are currently available, being inspired in part by them at the same time the imaginary inspires more extreme and exotic visions. The science-fiction novel Enders Game by Orson Scott Card provides an example of how this desire for the fusion of the digital and the real actually preceded the full availability of the technology. Enders Game centers on a boy-ninja who saves the world from aliens in a war game where the video game simulation becomes not only the training ground for real world warriors but the actual war itself. Originally written in 1977, years before flight simulators were invented, the training scenario in Enders Game has nonetheless so inspired military training programs that it was adopted as required reading by the Marine University in Quantico, Virginia. Graphics designers and computer scientists frequently cite science fiction as a source of inspiration. For example, Perlin and Goldberg of Disney Imagineering, the authors of Improv, a system for scripting autonomous interacting actors for virtual worlds, note the influence of Neil Stephenson’s description of the problems in constructing authoring tools for avatars in the Metaverse in his novel Snow Crash. Numerous programmers of contemporary (1999) video games and military flight simulators report the inspiration they have derived from this novel.[26]

Desire for realistic computer-generated images has combined with the stimulus to the computer graphics and hardware markets provided by exponential improvements in processors (Moore’s Law) and new chip architectures to fuel the growth of companies like Silicon Graphics, driving the prices of machines equivalent to first-generation Onyx workstations at over $20,000 to the price of powerful desktop computers around $5000. The potential markets for multimedia have stimulated the search for new architectures for image caching and compression techniques which can greatly reduce bandwidth and memory requirements of expensive high-end machines like the SGI InfiniteReality Engine with its tens of megabytes of graphics memory and multiple memory buses hundreds of bits wide, in order to bring high-end multimedia performance to PC prices.[27] A sense of the market forces driving this convergence of high-end computer architectures, graphical rendering hardware and software with low-end commercial markets for computer graphics ultimately bringing VR to everyone can be seen by Silicon Graphics' partnership with Nintendo to produce Nintendo64.

On August 23, 1993, Silicon Graphics, NEC, and Nintendo announced a partnership to build the world's most powerful game machine. Speaking to a crowd of analysts, news media, and industry pundits, Silicon Graphics founder and then CEO Jim Clark outlined an ambitious project, Project Reality, which he claimed would revolutionize the consumer electronics industry. Never one for understatement, Clark declared that Project Reality would harness the "combined computer power of hundreds of PCs" for less than $250. Clark's often-stated goal since he started the company, the plan called for Silicon Graphics to design two chips to form the heart of the system: the R4300i processor and the Reality CoProcessor (RCP). The R4300i processor, a low-cost, low-power MIPS RISC CPU, would handle the interaction with the game player and manage the game's control tasks. The RCP, a media-processing engine, would handle all the high-performance graphic and music-synthesis tasks. The R4300i processor team was already in place at MIPS, recently acquired by Silicon Graphics and staffed with experienced engineers. However, the Project Reality team, slated to design the RCP and write the software, had to be built from scratch. NEC fabricated the RCP chips on a totally new, state-of-the-art chip fab line in Japan, built at a cost of more than one billion dollars. The chips in Nintendo64 were the first microchips produced in volume using .35 micron semiconductor technology. Nintendo's partners, Silicon Graphics and NEC, succeeded in getting the world's most advanced semiconductor technology into a consumer product. Nintendo64, shipped in April of 1996, has been one of the most successful entertainment products in history. By the end of 1997 SuperMario64 had enabled Nintendo to capture a worldwide base of 6 million users with video game revenues breaking the $2 billion mark.

In 1997, Silicon Graphics CEO Ed McCracken explained the importance of this development in his letter of introduction to the Silicon Graphics booth at the National Broadcasters convention:

Through the years, many of you have asked why the entertainment market is critical to the success of Silicon Graphics. The answer is simple. Our entertainment customers drive our technological innovation. And technological innovation is the foundation of Silicon Graphics.[28]

Indeed, in the 12 months ending in March 1994, SGI reported revenues of $1.5 billion. In 1997 revenues were reported as $3.66 billion.[29] SuperMario was certainly super to SGI. Kurt Akeley, a cofounder of Silicon Graphics, echoed McCracken's sentiments to a group of SGI developers' at a meeting in Munich in the spring of 1998:

That's what Silicon Graphics has been about since 1982, when I was one of the people that started it. We've had a huge impact, with you, making that come true. We've done it in domains that seemed obvious at the time: computer-aided design scientific visualization, as well as domains that were not anticipated.

It's easy to imagine that we've affected more people directly with the technology in the Nintendo64 than we have collectively with all of our other computers. We've certainly sold more of them - by far - than all of the rest of the workstations we've done. So we've had an effect, not just in the technical domain, not just in the places that would have been fairly obvious to applied 3-D technology, but across the board-- in people's homes and in their lives, and we're going to continue doing that.[30]

By making the technology more affordable, by finding ways to scale it to large consumer markets, by aiming, in short, to make technologies like the RISC chip everywhere present, developments such as those illustrated by the research-entertainment nexus including Pixar, Silicon Graphics, and Nintendo have made use of imaging technology in science and medicine possible on a scale and at a pace that would not otherwise be imaginable.

Distributed Networks: SIMNET

In addition to the central role of the research-entertainment complex, the examples discussed in the preceding sections point to the importance of federal funding of university research as well as research in government-funded labs (primarily through DARPA contracts) as crucial in creating and sustaining the hardware developments critical to the fields of 3-D graphics, simulation technology and virtual reality. This is only half of the picture. Although networks are usually thought of apart from computer graphics, network considerations are in fact crucial to large-scale interactive 3-D graphics. Graphics and networks have become two interlocking halves of a larger whole: distributed virtual environments. Central to this work have been DARPA funding and the US Army's creation of SIMNET, the military's distributed SIMulator NETworking program.

Simulators developed prior to the 1980s were stand-alone systems designed for specific task-training purposes, such as docking a space capsule or landing on the deck of an aircraft carrier. Such systems were quite expensive, for example, more than $30-$35 million for an advanced pilot simulator system in the late 1970s, and $18 million for a tank simulator at a time when an advanced individual aircraft was priced around $18 million and a tank considerably less. High-end simulators cost twice as much as the systems they were intended to simulate. Jack A. Thorpe was brought into DARPA to address this situation based on a proposal he had floated in September 1978. Thorpe's idea was that aircraft simulators should be used to augment aircraft. They should be used to teach air-combat skills that pilots could not learn in peacetime flying, but that could be trained with simulators in large-scale battle-engagement interactions. Thorpe proposed the construction of battle-engagement simulation technology as a 25-year development goal.[31] Concerned about costs for such a system Thorpe actively pursued technologies developed outside the DoD such as video-game technology from the entertainment industries.[32] In 1982 Thorpe hired a team to develop a network of tank simulators suitable to collective training. The team that eventually guided SIMNET development consisted of retired Army Colonel Gary W. Bloedorn, Ulf Helgesson, an industrial designer, and a team of designers from Perceptronics of Woodland Hills, California, led by Robert S. Jacobs. Perceptronics had pioneered the first overlay of computer graphics on a display of images generated by a (analog) videodisc as part of a tank gunnery project in 1979.

The SIMNET project was approved by DARPA in late 1982 and began early in the spring of 1983 with three essential component contracts. Perceptronics was to develop the training requirements and conceptual designs for the vehicle simulator hardware and system integration; BBN Laboratories Inc, of Boston, which had been the principal ARPANET developer, was to develop the networking and graphics technology; and the Science Applications International Corporation (SAIC) of La Jolla, California was to conduct studies of field training experiences at instrumented training ranges at the National Training Center in Fort Irwin, California.

Affordability was the chief requirement Thorpe placed on the development of SIMNET components. Sticking to this requirement led to the most highly innovative aspects of SIMNET. Prior to the late 1980s simulators were typically designed to emulate the vehicles they represented as closely as engineering technology and the available funds permitted. The usual design goal was to reach the highest possible level of physical fidelity -- to design "an airplane on a stick," as it were. The SIMNET design goal was different. It called for learning first what functions were needed to meet the training objectives, and only then specifying the needs for simulator hardware. Selective functional fidelity, rather than full physical fidelity, was SIMNET's design goal, and as a result, many hardware items not regarded as relevant to combat operations were not included or were designated only by drawings or photographs in the simulator. Furthermore, the design did not concentrate on the armored vehicle per se. Rather, the vehicle simulator was viewed as a tool for the training of crews as a military unit. The major interest was in collective, not individual, training. The design goal was to make the crews and units, not the devices, the center of the simulations.[33] This approach helped minimize costs, thus making possible the design of a relatively low-cost device.[34]

An early crisis that threatened to undo the project was that the visual-display and networking architecture being developed by BBN would not support the SIMNET system concept within the limits of the low-cost constraints. Analyses and expert judgments, from both within and outside of DARPA, indicated that the planned use of available off-the-shelf visual-display technology would not support the required scene complexity within the cost, computer, and communications constraints set by the SIMNET goals. However a proposal from Boeing allowed Thorpe to take advantage of the new generation of DARPA-funded microprocessor advances in VLSI and RISC for development of a new low-cost microprocessor-based computer image generating technology for visual displays. The technology proposed by M. Cyrus of Boeing met the scene complexity ("moving models") requirements at acceptably low dollar and computational costs. Also, it permitted use of a simpler, less costly networking architecture. The proposed technology would use microprocessors in each tank simulator to compute the visual scene for that tank's own "virtual world," including the needed representations of other armored vehicles, both "friendly" and "enemy." The network would not have to carry all the information in the visual scenes (or potential visual scenes) of all simulators. Rather, the network transmission could be limited to a relatively small package of calibration and "status change" information.[35]

With these architecture and design elements in place SIMNET was constructed of local and long-haul nets of interactive simulators for maneuvering armored vehicle combat elements (MI tanks and M2/3 fighting vehicles), combat-support elements (including artillery effects and close air support with both rotary and fixed-wing aircraft), and all the necessary command-and-control, administrative and logistics elements for both "friendly" and "enemy" forces. A distributed-net architecture was used, with no central computer exercising executive control or major computations, but rather with essentially similar (and all necessary) computation power resident in each vehicle simulator or center‑nodal representation.[36]

The terrains for the battle engagements were simulations of actual places, 50 kilometers by 50 kilometers initially, but eventually expandable by an order of magnitude in depth and width. Battles were to be fought in real time, with each simulated element—vehicle, command post, administrative and logistics center, etc.‑being operated by its assigned crew members. Scoring would be recorded on combat events such as movements, firings, hits, and outcomes, but actions during the simulated battle engagements would be completely under the control of the personnel who were fighting the battle. Training would occur as a function of the intrinsic feedback and lessons learned from the relevant battle-engagement experiences. Development would proceed in steps, first to demonstrate platoon‑level networking, then on to company and battalion levels, and later perhaps on to even higher levels.

Each simulator was developed as a self-contained stand-alone unit, with its own graphics and sound systems, host microprocessor, terrain data base, cockpit with task-training-justified controls and displays only, and network plug-in capability (Figure 2). Thus, each simulator generated the complete battle-engagement environment necessary for the combat mission training of its crew. For example, each tank crew member could see a part of the virtual world created by the graphics generator using the terrain data base and information arriving via the net regarding the movements and status of other simulated vehicles and battle effects. The precise part of the virtual world was defined by the crew member's line of sight—forward for the tank driver, or from any of three viewing ports in a rotatable turret for the tank commander.

The visual display depended primarily on the graphics generator resident in each simulator. This computer image generation (CIG) system differed in several important characteristics from earlier CIG systems. First, it was microprocessor-based (vs. large mainframe or multiple minicomputer based), and therefore relatively low in cost (less than $100,000 per simulator visual-display subsystem, vs. more than $1 million per visual channel; typical flight simulators have at least five visual channels). Secondly, it was high in environmental complexity with many moving models and special effects, but low in display complexity with relatively few pixels, small viewing ports, and a relatively slow update rate of 15 frames per second (vs. the opposite with earlier CIG systems and the technology being developed to improve and replace them). The development of the essentially unique graphics generator for SIMNET was a principal factor in permitting the system to meet the low-cost-per-unit constraint of the plan.

Figure 2. Architecture of a Single M1 (Abrams Tank) Simulator in SIMNET

(From J.A. Thorpe, "The New Technology of Large Scale Simulator Networking: Implications for Mastering the Art of Warfighting," in Proceedings of the 9th Interservice/Industry Training Systems Conference, Nov. 30-Dec. 2, 1987, American Defense Preparedness Association, 1987, p. 495.)

The architecture of the microprocessor-based graphics generator permits anyone or any simulator so equipped to connect to the net. This, combined with the distributed computing architecture of the net, provides an extremely powerful and robust system. New or additional elements can be included simply by "plugging into" the network. Once connected to the net, simulators transmit and receive data "packets" from other simulators or nodes (such as stations for combat-support or logistics elements), and compute their visual scenes and other cues (such as special effects produced by the sound system). Because the data packets need to convey only a relatively small amount of information (position coordinates, orientation, and unique events or changes in status), the communications load on the net and the increase in load with the addition of another simulator are both quite modest. Also, where updating information is slow in coming from another simulator, its state can be inferred, computed, and displayed. Then, when a new update is received, the actual-state data are used in the next frame, and any serious discontinuity is masked by the receiving simulator's automatic activation of a transition-smoothing algorithm. Should a simulator fail, the rest of the network continues without its contribution. Thus, network degradations are soft and graceful.

The prototypes and early experiments with SIMNET elements were carried out between 1987-89, and the system was made operational in January 1990. The Army bought the first several hundred units for the Close Combat Tactical Trainer CCTT system, an application of the SIMNET concept, the first purchase of a system that would eventually contain several thousand units at a total cost of $850 million.[37]

From DARPA to Your Local Area Network

Throughout the period examined here a key characteristic of federal funding of university research through agencies such as the NSF, NASA, and NIH, as well as through defense department agencies such as IPTO and DARPA has been the interest in sustaining imaginative, exploratory, often “holy grail” research expanding the frontiers of knowledge.

But as examples such as the VLSI program suggest, support of federal agencies has also been directed toward seeing that the products of federal research funding get transferred to technologies in service of both national defense and the commercial sector. For most of the period covered to this point—up to the end of the 1980s—policy discussions about these goals—of seeing that research served national defense and that it ultimately benefited the commercial sector—were either kept rigidly separate or delicately balanced in a complicated dance.

With the end of the Cold War, a stronger emphasis was placed during the 1990s on running a fiscally efficient military built on the practices of sound business and of making military procurement practices interface seamlessly with commercial industrial manufacturing processes. With pressure to reduce military spending applied by the Federal Acquisitions Streamlining Act of 1994, the Department of Defense remodeled policies and procedures on procurement (through DOD Directives 5000.1 and 5000.2) that had been in place for over 25 years. Among the policies the new directives established was a move away from the historically based DOD reliance on contracting with segments of the US technology and industrial base dedicated to DOD requirements, moving instead by statutory preference toward the acquisition of commercial items, components, processes and practices. In the new mandated hierarchy of procurement acquisition, commercially available alternatives are to be considered first, while choice of a service-unique development program has the lowest priority in the hierarchy. DOD components were directed to a