Morton Heilig, pioneer in virtual reality research

Above: Morton Heilig shooting his award-winning art film "Once," courtesy of actress Marta Kristen (playing "Humanity" on the right).

Morton Heilig (1926-1997) was an American cinematographer, theorist and inventor whom scholars credit with originating studies of virtual reality through his work in immersive multimedia, and whose theory and inventions were influential in later designs of virtual reality devices.

CINEMA OF THE FUTURE

Heilig was inspired by the Cinerama, a technique that used three cameras to project movies onto an arced widescreen, expanding the area of viewing space for the audience. In his 1955 essay, "The Cinema of the Future," Heilig writes that Cinerama, as well as 3-D films, which had only recently entered widespread usage, was a logical step in the evolution of art: "The really exciting thing is that these new devices have clearly and dramatically revealed to everyone what painting, photography and cinema have been semiconsciously trying to do all along -- portray in its full glory the visual world of man as perceived by the human eye" (p. 244). Heilig wanted to expand this replication of reality beyond the available senses of sight and sound to create the future cinema after which he named his essay.

Heilig begins his study of the cinema of the future, not by examining the relationship between film and viewer, but by trying to learn, "how man shifts his attention normally in any situation" (1955/2002, p. 249). Heilig's study of attention deals with which of each of the five senses are being used at any given point in an experience. He tells us that senses are monopolized in the following proportions (p. 247):

• Sight: 70%


• Hearing: 20%


• Smell: 5%


• Touch: 4%


• Taste: 1%

Heilig also identified the various organs that were, "the building bricks, which when united create the sensual form of man's consciousness": the eye with 180 degrees of horizontal and 150 degrees of vertical three dimensional color view; the ear, able to discern pitch, volume, rhythm, sounds, words and music; the nose and mouth detecting odors and flavors; the skin registering temperature, pressure and texture (1955/2002, p. 245).

The goal of future cinema, said Heilig, was to replicate reality for each of these senses. He advocated theaters that would make us of magnetic tape with separate tracks for each sense -- the intensity of odors to be piped in through air conditioning systems, moving pictures that extended beyond the peripheral vision, stereophonic sound delivered by dozens of speakers. Sensory stimulations would be provided in the proportions at which they naturally occurred (as listed above). Attention would be directed and maintained not through use of cinematic tricks, but by mimicking the natural focusing functions of the human eye. Through the creation of multi-sensory based theatric, man would better learn to manipulate the "sense materials," to develop and arrange greater forms of art (1955/2002, p. 251).

With these procedures, based on the natural biology of man and applied to the methodology of art, Heilig believed that, "The cinema of the future will become the first art form to reveal the new scientific world to man in the full sensual vividness and dynamic vitality of his consciousness" (1955/2002 p. 251).

Unfortunately, Heilig's ideas met with the same problem that he cites in his essay as delaying the development of 3D and films with sound: short-sighted financiers unwilling to provide funding. Before writing the essay, Heilig had already pitched his ideas in Hollywood, had them roundly rejected, and moved to Mexico City where he found them better received (Packer & Jordan, 2002, p. 240). In fact, "The Cinema of the Future" was originally published in Spanish, in the Mexican architectural journal Espacios (later reprinted in English by MIT in 1992).

SENSORAMA

Above: Advertisement for Heilig's Sensorama, courtesy of Scott Fisher's Telepresence.

After his theories about immersive theatre fell on mostly deaf ears, Heilig decided to take things into his own hands, developing (while completing his Communication Master's at the Annenberg School at the University of Pennsylvania) a device that would put his theories into practice. Heilig called the device, which resembled a 1980s-era arcade game, the Sensorama Simulator.

In his patent for the Sensorama (filed in 1961), Heilig stresses the pedagogical potential for the device, discussing, for instance, the armed services, who, "must instruct men in the operation and maintenance of extremely complicated and potentially dangerous equipment, and it is desirable to educate the men with the least possible danger to their lives and to possible damage to costly equipment" (p. 9). The default experience that shipped with the short run of Sensorama, however, was not a replication of the battlefield, but a series of journeys, including a motorcycle ride through Brooklyn (complete with seat vibrations mimicking the motor of the bike, the smell of baking pizza, wind from strategically placed fans, voices of people walking down sidewalks) and a view of a belly-dancer (with cheap perfume).

Howard Rheingold, in his 1991 book Virtual Reality: Exploring the Brave New Technologies, describes his own usage of the Sensorama, 30 years after its invention:

By virtue of its longevity, it was a time machine of sorts. I sat down, put my hands and eyes and ears in the right places, and peered through the eyes of a motorcycle passenger at the streets of a city as they appeared decades ago. For thirty seconds, in southern California, the first week of March, 1990, I was transported to the driver's seat of a motorcycle in Brooklyn in the 1950s. I heard the engine start. I felt a growing vibration through the handlebar, and the 3D photo that filled much of my view came alive, animating into a yellowed, scratchy, but still effective 3D motion picture. I was on my way through the streets of a city that hasn't looked like this for a generation. It didn't make me bite my tongue or scream aloud, but that wasn't the point of Sensorama. It was meant to be a proof of concept, a place to start, a demo. In terms of VR history, putting my hands and head into Sensorama was a bit like looking up the Wright Brothers and taking their original prototype out for a spin. (p. 50)

Sensorama failed to catch on, and faded away due to financial issues, but, writes Joseph Kaye in 2001, the machine still remains the pinnacle in some aspects of immersive experience, notably in utilization of olfactory stimulus.


TELESPHERE MASK

Above: Telesphere mask, courtesy of Morton Heilig.com.

Even before the Sensorama, however, Heilig had worked on another patent that would have also proved influential on future virtual reality designers, had they heard of it: the Telesphere mask, awarded a patent in 1960 under the name, "Stereoscopic-Television Apparatus For Individual Use."

The Telesphere mask was a sort of head-mounted version of the Sensorama that would allow for wrap-around views, stereo sound, and air currents that could blow at different velocities or temperatures, and could carry smell (Heilig, 1957). The Telesphere was the first patented head-mounted apparatus designed to convey virtual views to the user, predating Ivan Sutherland's influential "Head-Mounted Three-Dimensional Display," nicknamed the "Sword of Damocles" by half a decade. The mask was built as a prototype, but Rheingold surmised in 1991 that, "If it had not been for the vicissitudes of research funding, Morton Heilig, rather than Ivan Sutherland, might be considered the founder of VR" (p. 46).

...And shortly after, backed by exposure from Rheingold's first-hand reports of experiencing Heilig's work, Heilig's reputation as a pioneer in virtual realities studies and technology grew. He spoke in late 1991 at the CyberArts International Conference, where his speech was received as, "a fascinating picture of a man whose vision far outpaced those with the resources to bring his ideas into reality...there was no doubt that Heilig deserves respect for his pioneering work" (Czeiszperger & Tanaka, 1992, p. 93).

LASTING INFLUENCE

Rheingold was turned onto Heilig's work by a technologist named Scott Fisher, who had the Sensorama poster displayed above in his office at the NASA Ames Research Center (1991, p. 51). Fisher's work as founding Director of the Virtual Environment Workstation Project (VIEW) at NASA led to work that helped create the head-mounted viewer and glove that has been associated with virtual reality since the mid-1980s.

Fisher cited Heilig's work with the Sensorama as an influence in his 1991 essay, "Virtual Environments, Personal Simulation & Telepresence," and discussed the major shortcoming of the device: the fact that the user couldn't independently move or change their viewpoint (p. 3). These decisions were made, as Heilig discussed in his essay above, by the director of the Sensorama film.

Above: Aspen Interactive Movie Map

Earlier, Fisher had worked with a project called the Aspen Movie Map that utilized ideas started by Heilig's Sensorama and expanded them to allow for autonomy on the part of the user. The project involved the virtualization of Aspen, Colorado, in which users could freely tour the city, moving forward and backwards down streets (similar to Google Maps Street View) and even inside some buildings. In one configuration, Fisher writes, the user was surrounded on all sides by views of Aspen, similar to Heilig's wish for an image that would fill the whole of a viewer's vision (1991, p. 3).

Above: Diagram describing the capabilities of the VIEW Project with illustration of headset and glove, courtesy Telepresence.

At NASA, Fisher was a pioneer in what became to be known as "telepresence," or the concept of "being there" without actually being there. The headset and glove developed in the VIEW labs allowed for the user to have three senses virtually stimulated: sight, sound and touch. A headset with stereo speakers, similar to the Telesphere, allowed views in three dimensions that responded to motion of the user's head. The glove allowed for feelings of pressure, as mandated by Heilig's essay, that would allow the user to, for instance, remotely grasp things with a robot arm that would relay tactile responses (Fisher, 1991, p. 4).

Fisher envisioned numerous uses for virtual environments displayed and controlled with technologies like that produced by VIEW: architects could tour their creations before they were built, surgeons could practice on "electronic cadavers," or, as Heilig had envisioned, audiences could participate in full immersion theatre experiences (Fisher, 1991, p. 5).

Through Fisher's development of Heilig's ideas in the form of virtual reality and telepresence theory and practice, Heilig's pioneering work places him at the head of a branch of immersive technologies science that continues to make breakthroughs in VR technology today.

Before his reconsideration as a founding father of virtual realities, however, he made his money in designing theatres for Disney amusement parks, theatres that were like large-scale versions of the Sensorama with vibrating seats, arced widescreens, and 360-degree sound. Theatres and attractions based on Heilig's designs still exist in most Disney parks today, including a recently opened ride called "Soarin' Over California," that seems like a large-scale version of Sensorama, complete with the smell of oranges and pine trees being piped in at appropriate times through vents in viewers' seats.

Heilig remained fascinated by virtual realities and cyberspace, and believed in the potential of sensory-immersive art until his death in 1997. Responding to an essay that touched upon his work in a 1994 edition of Computer Graphics, Heilig delivered what he called his "final word" on the subject:

One hears endless debates about which form will survive--passive movie entertainment or computer-based virtual reality? I believe BOTH will because both satisfy normal human needs. There are times when people want to relax--to surrender themselves into the hands of a talented "experience conveyer" (years ago, I would have said "storyteller") who has something wonderful and interesting to share. At times like this, the last thing the viewer wants is to "make decisions."

On the other hand, there are times when one wants to explore--to be left alone--to examine a new world and savor a new experience in one's own way at one's own pace. So vive la difference! (p. 130)

PERSONAL THOUGHTS

Heilig's thought about exploration on one's own time reminded me of the recent crossovers in virtual reality and video games. Though we're still not at a point where one can slip on a Telesphere mask, there exists a video game in which one can, as in Heilig's Sensorama, hop on a (stolen) motorcycle and drive around Brooklyn, exploring (and presumably killing prostitutes) at one's own leisure. And certainly through videogames, we have experienced spatial sound and tactile vibrations through controllers and specially wired chairs. Thousands of people who have never visited New York City in real life have been able to experience Grand Theft Auto IV: Liberty City. But is that the art, the "cinema of the future," that Heilig would have wanted?

Heilig's work with scents has still not come to fruition in the digital age. I turned up an article from a 1999 issue of Wired Magazine discussing a company called DigiScent and a device called iSmell, but the latest news on the company involves layoffs and a lack of funding. And this projection of a smelly internet of the future was made a decade ago -- consider the changes that have molded the web since then.

For me, the most interesting part of this essay, and what drew me to choosing Heilig was this notion of the rejuvenation of a pioneer. Because Heilig worked outside of the academic/laboratory/military model, his inventions and theories were ignored or forgotten for thirty years, although key participants in virtual reality research, like Fisher, were pushing for his inclusion. Only six years before his death was Heilig celebrated widely for his achievements. In the meantime, from various interviews and personal accounts I've read, he had become a sort of cantankerous old man, with his old dreams covered under tarps in his garage. With Rheingold's visit, he reportedly became interested again in seeing his dreams achieved, and lived in a time when the technology could actually lead to them coming to fruition.

WORKS CITED

Czeiszperger, M. & Tanaka, A. (1992). Cyberarts international. Computer Music Journal, 16(3). Retrieved September 14, 2008, from JSTOR Database.

Fisher, S. (1991). Virtual environments, personal simulation & telepresence. Retrieved from http://www.itofisher.com/PEOPLE/sfisher/VirtualSimPresence-pdfrev.pdf.

Heilig, M. (2002). The cinema of the future. In R. Packer & K. Jordan (Eds.) Multimedia: From Wagner to virtual reality (pp. 239-251). New York: W. W. Norton & Company. (Original work published in 1955)

Heilig, M. (1994). Mort Heilig's response. Computer Graphics, 28(2). Retrieved September 12, 2008, from ACM Portal Database.

Heilig, M. (1957). U.S. Patent No. 2955156. Washington, D.C.: U.S. Patent and Trademark Office.

Heilig, M. (1961). U.S. Patent No. 3050870. Washington, D.C.: U.S. Patent and Trademark Office.

Kaye, J. (2004). Making scents: Aromatic output for HCI [Warning: PDF]. interactions, 11(1). Retrieved September 12, 2008, from ACM Portal Database.

Platt, Charles. (1999). You've got smell! Wired, 7(11). Retrieved from http://www.wired.com/wired/archive/7.11/digiscent.html.

Rheingold, H. (1991). Virtual reality: Exploring the brave new technologies. New York: Summit Books.

BONUS MATERIAL

Mort Heilig entry @ IMDB

Heilig's patent for a wicked looking skateboard

Episode of crazy kids' show called "Diver Dan" directed by Mort Heilig in his pre-VR days