Making Sense of Mixed Reality
Mixed Reality is a term being used to describe contemporary head mounted displays like the Microsoft HoloLens and the Magic Leap One, which support physical and digital objects being viewed and interacted with in real time. But the roots of Mixed Reality go much deeper and can maybe even help us understand reality better.
Mixed Reality by Milgram, Takemura, Utsumi, and Kishino (1994)
The Mixed Reality Continuum
A diagram first put forth by Milgram, Takemura, Utsumi, and Kishino (1994) is an approach to show the continuum which defines Mixed Reality, which can draw from the real environment, from augmented parts of the environment, or from purely virtual / digital entities. It highlights the idea that most experiences fall somewhere in the center mass of the diagram, with very few experiences operating at the far ends of the spectrum. Our perceptions of reality are an amalgam of our senses and the build-up of our experiences and biology. This is why optical illusions can be so disconcerting, they actively conflict with the story we have created through our sense of what is real. The same is true of the right side of the diagram, few experiences rely on purely virtual sensations, ignoring the floor, gravity, or other basic building blocks on what we consider to be the real. This relationship of our senses and the stories our minds form about what is happening are already a type of Mixed Reality. Our current relationship to the ecosystems of technology in our spaces has us constantly shifting between real and virtual elements, but as Mixed Reality takes hold, it promises to reduce that friction and return us to more holistic ways of living.
Where do we spend our lives?
But let’s start more close to home, in the physical places which represent where we live our lives. According to the Bureau of Labor Statistics, American’s spend their time between personal care, sleeping, working, eating, cleaning, learning, religious services, and leisure. But where do these activities take place?
In the kitchens, living rooms, bedrooms, trains, parks, stores, museums, airports, factories, farms, nightclubs, restaurants, schools, hotels, elevators, forests, boats, ships, grocery stores, libraries, cars, hospitals, banks, patios, and break rooms that make up our built and natural environment. Every experience we have is based in a place, and much of our interaction with technology happens through screen portals in our personal computing devices to virtual spaces.
But where are you?
You are reading this through a digital device that connects you to hundreds of virtual places, some of which are very meaningful to you. Amazon, Facebook, YouTube, Etsy, WebMD, Box, Github, Grubhub, Tumblr, Twitter, Instagram, BBC, Quickbooks, Spotify, and many more all vying for your attention and creating powerful relationships to people, to services, and to ourselves. Our senses in the physical world can become overwhelmed by our focus on these services and make us feel removed from the present situation. The activity of many of our social spaces have transformed with the ubiquity of mobile computing. This pull between your dining experience and your mobile activity is because you believe in both of them, but they are in near constant competition for our attention. How did mobile activity get so prevalent so quickly?
From Mainframe, to Personal Computer, to Mobile Computing
Content is King, but Context is Scale
In 1996, Bill Gates was famously quoted when talking about the coming internet revolution that "Content is King". His observation was that just as in broadcast, the bulk of the business quickly moved from people making the medium possible, to the creators of the content of that new medium. Hand in hand with this observation is the fact that putting computation where people need it increases the scale greatly. Initial mainframe computers in the halls of Academia and Business were powerful additions to business, but when the personal computer revolution came it make having a computer at home commonplace, and with it came more content and more scale. But in this scenario you still had to go to where the computer was to accomplish your tasks, whether they were leisure or crunching spreadsheets. Mobile computing, especially with the integration of GPS, let us take our computing devices to where we were. That ability to use your computer in the context of a specific place unlocked another order of magnitude in the scale of computation. That scale jump is likely to happen again when we can integrate our computation even more closely into the environments which we live and work. The vision of Mixed Reality is to combine our physical and virtual experiences into a unified whole which take advantage of the benefits from both.
“Software is eating the world.”
In 2011 Marc Andreessen, the Venture Capitalist, wrote an influential article in the Wall Street Journal which described how software eats the world. In the article he articulates how nearly every business was becoming (or should at least think about becoming) a software business. But how does software eat the world? It has been a step by step process. This includes all of types of media which exist in our environments. The first step was to consume the paper of the environment.
Image from Microsoft of screen based lineup of computing devices.
Software ate all the paper
Every piece of paper that exists in an environment can now be replaced by a screen. In 1991 Mark Weiser wrote about the oncoming ubiquity of computing and asked "How many tabs, pads, and board-sized writing and display surfaces are there in a typical room? Look around you: at the inch scale include wall notes, titles on book spines, labels on controls, thermostats and clocks, as well as small pieces of paper. Depending upon the room you may see more than a hundred tabs, ten or twenty pads, and one or two boards...Hundreds of computers in a room could seem intimidating at first, just as hundreds of volts coursing through wires in the walls did at one time. But like the wires in the walls, these hundreds of computers will come to be invisible to common awareness. People will simply use them unconsciously to accomplish everyday tasks."
The majority of what can be done on paper can now be done by screens. Any paper size you can imagine has a flat screen equivalent, from post-it sized hand held computing devices, to whiteboard scale projections and poster sized high resolution animated displays. Incidentally, this doesn't mean paper goes away, it means that it is free to explore it's other qualities of texture, cultural context, and uniqueness. This follows the same pattern of what happened to painting after the creation of photography. Printed paper and design will be used in novel ways we don't expect. However the properties of paper going digital have changed how processes, communications, and roles work within every industry.
Software is eating the objects
Now the software begins to inhabit the objects around us, giving digital life to formerly analog electronics. From thermostats, to speedometers, to refrigerators, to parking meters, the software reshapes our objects from the inside out. In some cases it mimics that traditional interaction we have had with the object, and in others it is completely rethought.
