How Virtual Reality Headsets Work

Virtual reality is the act of tricking your brain and visual senses into believing that you are immersed in another space.  This requires the perception, not the reality, that as we look around, or move our heads up, down, or from side to side, that the new world we are in appears as it should.

The surprising truth is that while our brains are good at detecting significant issues, we don’t require perfection in order to be visually satisfied that we are in a different world.  From a technical standpoint, this is a huge part of why virtual reality is feasible in 2016, as true 360-degree viewing is much more difficult than the solutions major players have come up with.
Let’s break this down.

Hardware Basics

Today’s virtual reality headsets come in two main flavors: external computing, like the Oculus Rift:

…and internal computing, like Google Cardboard:

The key difference is where the graphics calculations are being performed relative to the screen(s).  In the instance of the Rift, the rendering is performed on an external PC, connected to the headset by an HDMI cord, and in the instance of a system like Google Cardboard, the screen and computer are actually the same device (your phone).

In addition to the screen, both types include lenses, which are basically high powered glasses which allow you to clearly view a screen which is extremely close to your eyes—approximately two to three inches away.  If you’re curious enough to prove to yourself that these are necessary, try holding a detailed image that close to your face and see how clearly you can make out various features.  Even though the images in virtual reality appear to be far away, at times, the screen is a constant distance from your eyes.

Now that we’ve gone over the basics of the hardware, let’s talk about the remaining technology that is required to complete the experience, and compensate for our brain’s natural mistrust of what it is perceiving.

Each of our eyes requires a different image

If you’ve ever taken a right-eye/left-eye dominance test, you’ll understand this right away.  If you haven’t, quickly put your index finger about 5 inches in front of your face, and then close each eye one at a time.

Did you notice that what you see changes?  One of your eyes will perceive something quite similar to what you see when both eyes are open—this is called your dominant eye—and the other will be significantly different.

 What this means for a virtual reality headset is that we need to project two different images, one per eye, and then let the brain combine them in order to create the perception of a 3D environment.

 There is one additional wrinkle, however, as the farther away our focal point (what we’re specifically looking at), the less of a difference there is between what each eye sees. To prove this to yourself again, look at something farther away, and repeat the one eye at a time experiment, and note the subtle difference this time.

The rendering engines in virtual reality systems perform all of these calculations in real-time and then send the correct images to each eye.

Head tracking technology

The next aspect of virtual reality is taking into account the position of our head, which is one key component of what our eyes will perceive in a scene.

A virtual reality headset needs to adjust as our head moves, or the scene in front of us would “feel” like it is following us as we move.
 
To counter this, if we turn our head 45 degrees to the left, the scene needs to “move” 45 degrees to the right in our field of view.  Because the screen itself is strapped to our head, this is done by the computer which is rendering the scene.

The same goes for moving our head up and down.  The way this is accomplished is through similar technology to the iPhone’s gyroscope, which allows for physical device movement tracking.

3D sound

While not technically a part of the visual technology in virtual reality, sound plays a key role in creating a believable environment.  Think about the difference between a 3d audio-visual system for watching movies, where a battle scene can have an explosion occur behind you, the viewer, versus all of the sound being mapped to a stereo sound system coming from your TV.

 It makes a big difference while watching a movie, and an even bigger difference while fully immersed in virtual reality.

The potential, however, is enormous, as these systems can now track not only your relative position to an audio source (like an explosion), but also the physical orientation of your head and ears relative to that sound.

 If you spin around suddenly, you are now facing the explosion, and will perceive it to be directly in front of you, as opposed to behind you.  Spinning around on your couch during a movie wouldn’t do much.

Eye tracking

A relatively untapped source of innovation in virtual reality is eye tracking, which is different from head tracking in that our eyes can independently look around a scene, or more  critically, focus on a particular depth of field within a scene.

Currently, VR headsets render everything crisply and clearly, but human vision is relatively blurry at every depth other than the depth of the object that you are currently looking at.  On the whole, this does not appear to dramatically reduce our brain’s willingness to believe what it is seeing, but it definitely is a source for even greater reality.

So that’s that!  Next time you’re playing a virtual reality enabled game, you’ll be the expert in the room.