Thoughts on Virtual Reality

by piero scaruffi
Cognitive Science and Artificial Intelligence | My book on A.I. | My book on consciousness | Human 2.0 | Bibliography and book reviews | Contact/feedback/email
(Copyright © 2021 Piero Scaruffi | Terms of use )

Essays on this page:
  • "From Play to Simulation"
  • "From Simulation to Immersion"
  • What is "New" about "New Media"?
  • Further reading (in chronological order):

Thoughts on Virtual Reality: 1. "From Play to Simulation"

While in industrial societies work and p lay are u sua lly regarded a s mutually exclusive con cepts, the ir interrelation s come to the fore in societies who se membe rs ha ve a rathe r large amoun t of leisure time and d ispo sab le income. As more empha sis is pu t on creative forms of lab our, the b ounda ries be tween work and leisure time become b lurred . Johan Huizinga's book "Homo Ludens" (1938) is usually considered the first major analysis of "play" and of games. Huizinga saw play as more than just an amusement activity confined to the early years of one's life. He saw it as a fundamental process in creating meanings and shaping culture, in the self-organization, or autopoisesis, of a society. However, this fundamental process is kept in a separate category, the category of irrational and somewhat useless activities, neatly removed from ordinary life, for example from "work", if nothing else because it requires its own space and time, and special rules apply during the game: it is confined in a "magic circle". Three decades later Marshall McLuhan in "Understanding Media" (1967), living in the age of television and consumerism, saw games as cultural mirrors, "faithful models of our culture".

A few years after Huizing legitimized the study of play, game theory became popular in a completely different world, the mathematical world. This field of mathematics was born when John von Neumann and Oskar Morgenstern, published the book "Theory of Games and Economic Behavior" (1944). Game theory, in essense, builds mathematical models of behavior between rational agents. It was applied to evolution (e.g. John Maynard Smith) and to economics (eleven game theorists have won the economics Nobel Prize in economics, starting with John Nash, Reinhard Selten and Janos Harsanyi in 1994). While they were not talking about the same kind of "game" and "play" (not the ludic kind), there was a connection in the way that games usually take place in an abstract world and the players are supposed to play it according to rules, therefore "rationally".

Gregory Bateson mediated between the two concepts of play in "A Theory of Play and Fantasy" (1955). He hinted that play is crucial in training people for communication and that the emergence of play in nature may have been crucial for the evolution of higher forms of communication in mammals, primates and then humans. He saw play as existing in a limbo between reality and fantasy, meaning and absence of meaning, rationality and irrationality. And perhaps precisely the fact that it inhabits that limbo is what gives play the power to train us for future interactions with members of our species. One can conceive that "limbo" as a meta-level of communication. William Stephenson, in "The Play Theory of Mass Communication" (1967), saw play precisely as a meta-activity in which the player plays with the very nature of communication.

This may explain why we enjoy reading "bad news" (particularly sex scandals and crime) more than we enjoy reading good news. Good news are not "play": they are what one expects, according to the rules in force. Bad news can be understood only by setting yourself in a world in which those rules don't apply and other rules apply. And finding the right set of alternative rules is also the way to explain the bad news, for example why someone committed a heinous crime or why a man cheated on his wife. Looking for the right set of rules is a game in itself. In order to understand an unexpected situation we need to interpret that situation as a game and figure out the rules of that game.

Something similar can be said of culture and entertainment in general. When we read a novel, we "play" with the novel (again, a meta-level of communication). Paraphrasing something that John Fiske wrote in his book "Television Culture" (1987), we play with a text insofar as we create our own meaning of what the text means. We look for the rules of the game that produced that text, and, since we cannot know for sure what they were, we make our own. The author had in mind a game between the text and the reader: the reader turns it into the game of discovering what that game is. We make our own meaning out of reading the text. Ditto if it's a poem or a movie. A literary critic typically "invents" his own interpretation of a novel, and we are all critics.

There is traditionally an approach to games that juxtaposes them to texts as polar opposites: texts being "linear", because one reads them sequentially from beginnig to end, and games being "nonlinear" because they are interactive. However, one can argue that texts are also "games" and games are also "texts", or, better, texts can be "played" (are playable) and games can be "read" (are readable). Playability and readability are two dimensions of a diagram in which texts tend to be high on the dimension of readability and games tend to be high on the dimension of playability, but they are not mutually exclusive. After all, a novel or a film are both a narrative artifact and a ludic act; and a game's rules and purpose can be summarized in a narrative "plot". By the same token, a poem is "playable" because its meaning is often ambiguous and a game is poetic because of its emotional resonance. We experience the game both ludically and narratively because we tell stories about the game.

Hence it is more appropriate to talk about a continuum, a spectrum, of playability rather than two polar ends of the spectrum. The computer game, the game created on a computer, certainly moves dramatically towards the playability pole, and the reason is not only the storytelling and interface flexibility enabled by the computer.

Gonzalo Frasca in "Simulation versus Narrative" (2003) pointed out a natural relationship between games and computers. A computer game is a sort of simulation system. And at the same time the computer is a simulation machine, as Turing showed in 1936 (every computer is a "universal Turing machine"), coincidentally at the same time that Huizinga started thinking about play. Turing (with a thought experiment) devised a universal computing machine that can simulate any Turing machine, and that universal computing machine was implemented in the late 1940s as the digital electronic computer. Hence, computers were born as generators of simulations. Therefore they are the natural platform for games. Games blossomed in the age of the computer the same way that rock music (a rebellious form of music) blossomed in the age of radios and records. If rock music was the most important cultural phenomeno to emerge out of the mass adoption of the radio and the record, the videogame could be the most important cultural phenomenon to emerge out of mass computerisation.

Traditionally, a representation has been either a static element (a painting, a photograph, a poster of something) or a narrative. That's how we normally communicate. However, there exists an alternative to representation: simulation. I have two ways to teach you what happens in a situation: i can tell you in a series of sentences or i can create a model of that situation and immerse you in it. Before the invention of the computer, it is difficult to model a complex system. When in 1906 Lizzie Magie started selling the board game "The Landlord's Game" (later renamed "Monopoly" in 1932), she was trying to teach players a lesson in economics by immersing them in a simulation. A tabletop game like "Dungeons & Dragons", first published in 1974 by Gary Gygax and Dave Arneson, doesn't simulate the real world but simulates an imaginary world in which there are still people, money and, alas, wars. The computer made it possible to create simulations of complex situations. While both representation and simulation can express the behavior of the living beings involved, in a simulation those living beings enact their role all the time and react in real time to actions taken by the player. When scientists started studying systems that are too difficult to express in terms of linear equations, they started using simulations. A linear equation can predict the future state of a simple system (for example, where an arrow will fall), but only a simulation can predict (with a degree of approximation) the future of a complex system (for example, tomorrow's weather). Because it models the behavior of the agents, a simulation lends itself to interaction: the player is not simply a reader or a spectator, the player can actually manipulate the simulation. In a sense, that's the whole point of the simulation: the "what happens if". Note that simulations never end, unlike narratives (although in videogames there is usually a goal, a definition of what it means "to win", and therefore an end). It took a while to realize that a simulation can also be the venue for play.

Ian Bogost in "Persuasive Games" (2007) argued that the simulations created by videogames constitute a veritable "rhetorical" tool, just like the verbal nuances of the ancient art of rhetoric, and therefore videogames are a persuasive medium. As Gonzalo Frasca wrote: "simulations can express messages in ways that narrative simply cannot."

As Espen Aarseth wrote in texts like "Doors and Perception - Fiction vs. Simulation in Games" (2007), videogames set in a virtual world combine two aspects: the explorative and the configurative; meaning that the user is offered two capabilities: to explore the virtual world and to modify it. While the modification (the ability of shaping the world) is easy to understand as a difference with the narrative media, the exploration is a no less crucial component: exploring a virtual world is not like being taken steps by step into the world of a novel or of a film. Even better if the exploration requires a degree of reasoning, equivalent to solving a puzzle. It is not a coincidence that the labyrinth is a very common setting for games.

Board games like "Monopoly" are actually interesting because they blur the distinction between play and reality within the game itself. Even more exciting in this sense is the live-action game "Assassin", popular on college campuses in the USA, because the gameplay takes place in real ordinary life, in the real ordinary world, over the course of days or weeks, 24 hours a day, and the players don't know who else is playing; and it is debatable if a computer game can ever achieve this kind of "estrangement".

Television also created interesting swaps of virtual and physical worlds. The television documentary "An American Family" (1971), the first "reality" show, followed an ordinary middle-class family: the virtual world of the show is actually the real world, not simulated at all, but indirectly the show simulates the reality of the spectator (the spectator has become the spectacle).

Paul Levinson in "Toy, Mirror and Art" (1977) argued that all media start out as toys, before they become representational media ("mirrors"), and finally media of artistic expression. Videogames seem to fit the description of this three-stage transformation.

Their success often depends on the "mirror" stage, when they resonate with the weltanschauung of a nation. Evidence that the study of games is still far from understanding what makes players click is the fact that it is so hard to predict which games will become popular across the world or even within a nation. Take the Korean series "The Squid Game" that was streaming on Netflix in 2021 (directed by Dong-hyuk Hwang and somewhat similar to Takashi Miike's 2014 horror movie "As The Gods Will", in turn adapting the first installment of the manga series written by Muneyuki Kaneshiro and illustrated by Akeji Fujimura in 2011). At its core was a deadly survival game played by desperate people that consisted in the players playing old children's games. It became an overnight sensation possibly because it reflected social anxieties that not even sociologists and psychologists, let alone game theorists, could understand well.

They became a "mirror" at a time when Jean Baudrillard was dissecting postmodern society as blinded by the models and maps that had created a gap with the real world: "The territory no longer precedes the map, nor does it survive it. It is nevertheless the map that precedes the territory-precession of simulacra-that engenders the territory". He viewed society as confusing the real and the signs of the real, as blurring the distinction between the real and the artificial. Pre-modern societies were organized around symbolic exchange, modern societies around production, and post-modern societies around "simulation" (back then mainly television). Within the latter, simulations create a "hyperreality" of semiotic spectacles (like Disneyland and the shopping mall). The consumer society is no longer a consumer of goods but now also a consumer of signs.

What was missing in Baudrillard's analysis was that the "territory" itself has been dramatically compromised by human civilization over the centuries. One can argue that we have been living in a "virtual world" since we moved to cities: the urban world of paved roads and brick buildings is built on top of the our original world, the natural environment, which is still visible only in a few city parks. And within cities we have further "virtual worlds", like Disneyland (1955) in Los Angeles. A physical virtual world: One can argue that the "Burning Man" festival, originally organized in 1990 by Kevin Evans and John Law organize, creates a "virtual world" (an ephemeral one) in the Black Rock Desert. And so reality isn't quite divided into the virtual and the physical, but it is a continuum, a gradient, between these two extremes. The virtual worlds built inside the computer are a continuation of a process that humans started a long time ago of creating increasingly more artificial worlds on top of the natural world (in a sense replicating the multi-layered structure of their brain's neocortex).

Thoughts on Virtual Reality: 2. "From Simulation to Immersion"

Virtual reality did not originate in the world of games but became popular in such a world. Outside of games, virtual reality is still a rarity. Ignoring this fact can lead to a misinterpretation of what virtual reality does and is. First and foremost, it is a game, it is a form of playing.

Virtual reality is not the first "immersive" media. In fact, "immersion" and "presence" have long been goals of the visual arts. One could argue that the medieval cathedral (a multimedia experience if ever there was one) was precisely conceived as an "immersive" experience: the faithful literally entered a virtual world when passing through the portal. The camera obscura and the magic lantern (both of which mixed physical and projected worlds) were popular attractions before the invention of cinema, as was the painted panorama. Charles Wheatstone's Stereoscope of 1832 (the sensation of the Great Exhibition of London in 1851), Auguste and Louis Lumiere's Photorama for the 1900 Paris Exposition, Edwin Link's flight simulator of 1929, William Gruber's and Harold Graves's View-Master (exhibited at the 1939 New York World's Fair), Fred Waller's Cinerama (first exhibited in New York in 1952), and Morton Heilig's Sensorama (first used to make short 3D stereophonic films in 1962) were all pre-computer attempts at an immersive experience. What changed after the advent of the electronic computer was the digital simulation. As computer became faster, cheaper and smaller, digital simulation became a much simpler way to design immersive experiences.

In 1817 English poet Samuel Taylor Coleridge coined the terms "willing suspension of disbelief" to explain how we get absorbed into a narrative text: we, as readers, tend to overlook implausible elements in a plot because we are inclined (genetically programmed?) to construct narrative coherence. When we experience storytelling in modern media like television and cinema (or in a novel), our default state is "suspension of disbelief". Virtual reality is a medium in which the default state is belief instead of suspension of disbelief. Virtual reality aims for "presence": the brain of the user (even if fully aware of being the user of a virtual-reality system) comes to believe that virtual reality is reality, and reacts as if it were reality (for example, feeling fear or joy). The user is not a spectator but a participant, an explorer, a deus-ex-machina. The setting and the body movements are clearly not the ones of the physical world, but the emotions are. Despite the fact that the user is wearing a cumbersome headset and knows perfectly well the room in which the experience takes place, virtual reality succeeds if the user "feels" that the projection in the headset is the reality out there. In fact the emotions can even be stronger. Hence the frequently touted correlation between virtual reality and psychedelics in countercultural salons.

One interesting aspect of the transition from "suspension of disbelief" to "belief" is what it does to the attention span. The headset-wearer is forced to focus on the sensory experience, which is exactly the opposite of what has been happening with all the other media. Experiencing content is psychologically different from consuming content. We live in a world in which the "attention span" has been constantly reduced. The younger generations are proud that they are "multi-tasking" all the time, while the older generations complain that this multi-tasking and this short attention span are creating the "shallows" (the title of an influential book by Nicholas Carr), consumers of information with only a shallow perception of the world, people with a very superficial understanding of the issues, consumers of information who cannot read a poem or a philosophical essay because it requires concentration. Virtual reality can be the perfect antidote to this trend because virtual reality is a medium that "forces" absolute focus on the content. Virtual reality creates a new dimension of human-machine interaction, one in which the machine is not a distraction but a tool to concentrate. We had interaction based on keyboards, mouse, voice and touch. Virtual reality expands interaction to body movements. Immersion is a distraction from the physical reality; but at a meta-level immersion is the opposite of distraction.

Thoughts on Virtual Reality: 3. What is "New" about "New Media"?

The art world has embraced the expression "new media" as if it identified a category of art the way "painting" or "sculpture" do. In most cases the "new media" artist is an artist who employs "new" technologies (and sometimes not even so new, if you ask a technologist). But there is nothing new about art using technology: all art has always used technology, and typically the most advanced technology. A poet who writes a poem using ink and paper is using technology. Painters were chemists and architects were engineers. Except for oral poetry and mime, art cannot exist without technology: it uses technology to manifest itself.

Terminology like "new media" is not only meaningless but also grounded in the logic of "old media": if it doesn't fit in the old categories of painting, sculpture, photography, cinema, etc, then it is "new media". Each medium was new compared with the preexisting ones.

On the other hand, there have been artistic trends that can be considered truly "new", like audience participation (displacing the old dogma of separation between producer and consumer) and interactivity (displacing the old museum dogma of "Do not touch!"). These elements become fully expressed in artworks that embrace simulation and gamification.

"Collaboration" has traditionally been conceived as collaboration between artists, but that too is grounded in the logic of "old media". When collaboration becomes co-creation (among any number of creators, artists or not), then one can truly experience something "new". I suspect that art based on simulation can open a whole new dimension to co-creation.

Further reading (in chronological order):

Alan Turing: "On Computable Numbers, With an Application to the Entscheidungsproblem" (1936)
Johan Huizinga: "Homo Ludens" (1938)
Gregory Bateson: "A Theory of Play and Fantasy" (1955)
William Stephenson: "The Play Theory of Mass Communication" (1967)
Marshall McLuhan: Understanding Media (1967)
Norman Holland's article "The Willing Suspension of Disbelief Revisited" (1967)
Paul Levinson: "Toy, Mirror and Art" (1977)
Philip Dick: "How to Build a Universe that doesn't Fall apart two days Later" (1978)
Jean Baudrillard: "Simulacra and Simulation" (1981)
Anthony Niesz & Norman Holland: "Interactive Fiction" (1984)
John Fiske: "Television Culture" (1987)
Howard Rheingold: Virtual Reality (1991)
Brenda Laurel: "Computers as Theatre" (1991)
Chris Chesher: "Colonizing Virtual Reality: Construction of the Discourse of Virtual Reality, 1984-1992" (1994)
Edward Casey: "The Fate of Place" (1997)
Marie-Laure Ryan: "Cyberspace, Virtuality, and the Text" (1999)
Katherine Hayles: "How We Became Posthuman" (1999)
Jesper Juul: "A clash between game and narrative" (1999)
Geoff King and Tanya Krzywinska: "ScreenPlay - Cinema/Videogames/Interfaces" (2002)
Julian Kucklich: "The Playability of Texts vs the Readability of Games" (2003)
Gonzalo Frasca: "Simulation versus Narrative" (2003)
Eric Zimmerman and Katie Salen: "This is not a Game: Play in Cultural Environments" (2003)
Nick Bostrom: "Are You Living in a Computer Simulation?" (2003)
Noah Wardrip-Fruin & Pat Harrigan: "First Person - New Media as Story, Performance, and Game" (2004)
Anthony Ferri: "Willing Suspension of Disbelief" (2007)
Ian Bogost: "Persuasive Games" (2007)
Alexander Galloway & Eugene Thacker: "The Exploit - A Theory of Networks" (2007)
Espen Aarseth: "Doors and Perception - Fiction vs. Simulation in Games" (2007)
Frans Mayra: "An Introduction to Game Studies" (2008)
Julian Bleecker: "Design Fiction" (2009)
David Kirby: "The Future is Now: Diegetic Prototypes and the Role of Popular Films in Generating Real-world Technological Development" (2010)
Jeremy Bailenson & Jim Blascovich: "Infinite Reality" (2011)
Veli-Matti Karhulahti: "Suspending Virtual Disbelief" (2012)
Nelson Zagalo, Leonel Morgado, Ana Boa-Ventura: "Virtual Worlds and Metaverse Platforms" (2012)
Bjarke Liboriussen: "Collective Building Projects in Second Life" (2012)
Erkki Huhtamo: "Illusions in Motion" (2013)
Rizwan Virk: "The Simulation Hypothesis" (2019)
See also A History of Virtual and Augmented Reality
See also the timelines of VR and AR (including the arts)
Back to the index