Not Just a Fad: Virtual Reality Really Does Benefit the Military

Photo by Lt. Megan Chester

IT’S EASY TO SEE THE COST SAVINGS OF VIRTUAL REALITY TRAINING—BUT IS
IT AS EFFECTIVE AS, OR EVEN BETTER THAN, OTHER TYPES OF TRAINING? THE
SCIENCE SO FAR SUGGESTS THE ANSWER TO BOTH QUESTIONS IS YES.

By Dr. Karl F. Van Orden, Dr. Jamie Lukos, Dr. Robert Gutzwiller, and Heidi Buck

Augmented reality (AR) and virtual reality (VR) systems are increasing in quality and decreasing in price in incredibly short time spans.¹ It is estimated that mixed reality (MR) technologies will have 95 million users by 2020, and that number more than triples by 2025.² The military takes an active interest in exploring the advantages of these technologies because of their great potential to help warfighters orient and train for missions in challenging environments, and operate with greater efficiency.

Many current applications of VR have focused on training, where an attempt is made to recreate situations individuals face in tactical environments. One clear example is flight simulators, which have grown in complexity and capability to the point of becoming immersive, challenging, and a valid means of learning how to perform a real-world task. The Federal Aviation Administration regularly provides guidance and directives on flight simulator evaluations and qualifications for pilots of commercial carriers, including how they are used in training and how they represent accurate scenarios involving stalls, recovery maneuvers, inclement weather, and dangerous conditions. Simulator time is put toward basic pilot rating achievement, and the acceptability of these hours is increasing. VR systems can make this training less expensive and more portable.

Space and Naval Warfare Systems Center Pacific has established the Battlespace Exploitation of Mixed Reality (BEMR) lab to demonstrate the art of the possible for applying AR and VR technologies to Navy-relevant areas of interest such as training, maintenance, and new user interfaces for a variety of operational environments. The laboratory has demonstrated capability concepts in the areas of flight deck training, shipboard deck gunner targeting and firing instructions, development of 3D models of actual shipboard spaces for installations and modifications (based upon LIDAR scanning), and specific maintenance training.

In the near future, with the enabling capability of VR, fleet maintenance workers will have the ability to gain expertise in highly skilled tasks that now require significant time and travel to expensive training facilities, and will use AR glasses to complete highly detailed tasks without the need to tediously search through paper manuals. Ship commanding officers and department heads will have instant visibility of critical information to enhance decision making and situational awareness. AR will allow personnel standing watch on the bridge of a ship to see track information fused with their view of the ships (or lights at night) out on the horizon. Special warfare personnel will prepare for and execute missions using VR and AR technologies, equipped with vital information—including navigation and enemy position data—whereas today’s warfighters have no direct or fused access.

Beyond the novelty of AR and VR visualization approaches, and the variety of applications outlined above, are there good perceptual and cognitive reasons for using them? Are there scientifically sound reasons to embrace MR technologies, or are we simply enamored with the latest superficial display technologies?

We describe a few of the most important underlying cognitive, sensorimotor, and perceptual processes associated with the success of current technologies and make the case for further exploitation of VR and AR technologies in the military. Although AR technologies appear to have tremendous utility, the rationale for their use is less understood at this time.

Emotional Engagement
An important advantage of virtual reality compared to traditional visual displays is the greater experience of presence, or “engagement.” Greater engagement is typically associated with greater excitement, anxiety, or fear. Virtual reality produces more intense emotional responses to evocative images such as snakes, spiders, or faces, than traditional display methods, such as computer monitors.³ VR seems to increase the salience of stimuli and experiences, perhaps by reducing extraneous distractions, in keeping with applied research on attention and learning.⁴ ⁵ In addition, once in VR one cannot “look away” from the virtual world without taking the device off, removing the capacity for some types of distraction. VR may come with some costs (e.g., motion sickness). If poorly designed, augmented AR displays may also create distractions when overlaid on the real world, such as visually blocking items of interest.⁶

The emotional dimensions associated with VR experiences are important, as we know from decades of work in the neurosciences that experiences containing greater emotional qualities form more durable and lasting memories in the brain.⁷ The feeling of presence in VR—as compared to traditional display modes—also has been shown to result in cortical activation responses in the prefrontal brain regions related to visual processing and self-reflective thoughts, regardless of the actual visual stimuli.⁸ This biological response is critical if VR is to be used as a training approach for military missions by producing more memorable mental representations of complex processes and procedures.

The greater salience of virtual environments, and subsequent engagement in them, has been central to its use in clinical rehabilitation of service members with post-traumatic stress disorder (PTSD). Patients are first carefully reintroduced via the virtual battlefield by clinical psychologists, who monitor them for adverse stress reactions. The virtual world can be made more or less stressful with various sights, sounds, and even smells. This more immersive technique allows the patient to learn to regulate their emotions in a controlled and constructive way, and, in doing so, successfully desensitize the brain from the traumatic events that produced PTSD.⁹ This mode of treatment has been found to be highly effective when compared to traditional treatment models, and has also been used for reducing fear of flying, suggesting its further potential against phobias and other psychological phenomena.¹⁰

Movement Coordination
Constraints surrounding space, budget, travel, and other factors have shifted the majority of today’s military training to noninteractive demonstrations on computer displays. Notably, a lack of interaction violates principles of skill learning especially for complex tasks. Physically engaging in a task or environment is generally more conducive to learning and memory retrieval (remembering what you learned) than static or observational exercises. To that end, VR technologies allow for far more interaction, as they are less resource constrained, remove dangerous conditions, and allow for failures with immediate feedback. For example, learning a new complex motor skill often requires personal, human-in-the-loop familiarity. Although some learning can and does occur through observation, skill consolidation (the strengthening in your performance over time) as well as motor performance are better when participants learn through doing versus observing— particularly for motor sequence timing.¹¹ During learning there is evidence that the area of the brain activated during the task shifts from premotor cortex and cerebellum during initial learning, to the frontal cortex during retention.¹² This shift in the cortical activation patterns through physical practice is critical for training effectiveness and retention. VR allows users to interact and move in the environment, building and practicing those skills—and this has been shown to amplify learning rates, particularly when learning novel tasks.¹³ VR in sensorimotor rehabilitation after stroke, traumatic brain injury, or amputation, has also shown improvements for motor rehabilitation.¹⁴ ¹⁵ For recovery of function in the areas of gait, hand and arm coordination, and balance, there are critical periods where more intensive therapy can make a positive difference, and VR enables more flexible and accessible complex treatment protocols.¹⁶

Spatial Perception
Another area in which VR is improving our lives is in architecture, where it is helping to overcome our perceptual limitations. For example, architects are sometimes surprised by the structural outcome of their designs, in that the space doesn’t have the same feel or look that they had expected. This modeling-to-reality mismatch may have to do with well-known perceptual distortions—the flattening of objects that fall perpendicular to the line of sight. A number of studies have found that the tilt of objects is typically perceived or underestimated as flat to the observer.¹⁷ ¹⁸ This tilt results in the common misperception that distant hills appear far steeper than they really are when viewed from up close.¹⁹ Others have found this effect exists for smaller nearby objects as well.²⁰

Architects are deriving significant benefit from modeling and experiencing newly designed structures in VR to better understand how spaces will look and feel to inhabitants, discover how spaces will look at different times of day, and share designs with others without needing to be in the same physical space. These benefits will also be recognized by maritime architects and shipboard installers/ maintainers in the near future, as it is widely known that individual ships of the same class (even consecutive ships from the same shipyard) have physical differences between them despite being constructed from the same plans. VR has an important advantage in that it allows an observer to experience angular dimensions of objects and spaces by observing them from different perspectives. There are already commercial applications to support this purpose, which basically work by ingesting architectural drawing and converting them into virtual space. In addition, the capability to create three-dimensional models from various scanning technologies continues to grow, allowing high-fidelity visualization of unique and complex environments.²¹

Summary
The use of VR technologies for gaming and entertainment has grown tremendously, subsequently driving down cost and enabling significant improvements in performance. The opportunity to apply MR technologies to numerous military applications now is feasible and justified given current perceptual, sensorimotor, and cognitive findings. VR experiences are more engaging, memorable, and perceptually real compared to other methods of training or familiarization. AR displays will soon allow individuals to navigate and gain important information in an entirely “heads-up” manner, without having to look at ancillary displays. Lastly, VR display technology, coupled with 360-degree film technology, will allow military personnel to experience rare or unusual events and become better prepared for, and less surprised by, what they see. Though the future is promising, we still must be cautious in application, as no technology is a cure-all, and a great deal of MR technology still needs to be created, tested, and applied in a user-centered design fashion.

Notes
1. Brynjolfsson, E., and A. McAfee, The Second Machine Age: Work, Progress, and Prosperity in a Time of Brilliant Technologies (New York: W.W. Norton & Co., 2014).

2. Bellini, H., W. Chen, M. Sugiyama, M. Shin, S. Salam, and D. Takayama, “Profiles in Innovation: Virtual and Augmented Reality—Understanding the Next Computing Platform,” Equity Research (13 Jan 2016), pp. 3-28.

3. Estupian, S., F. Rebelo, P. Noriega, C. Ferreira, and E. Duarte, “Can Virtual Reality Increase Emotional Responses (Arousal and Valence)? A Pilot Study,” in Marcus, A. (ed.), Design, User Experience, and Usability: User Experience Design for Diverse Interaction Platforms and Environments (Springer International Publishing, 2014), pp. 541-49.

4. Sweller, J., P. Ayres, and S. Kalyuga, Cognitive Load Theory (New York: Springer, 2011), http://doi.org/10.1007/978-1-4419-8126-4.

5. Wickens, C. D., S. Hutchins, T. Carolan, and J. Cumming, “Attention and Cognitive Resource Load in Training Strategies,” in A. F. Healy and L. E. Bourne (Eds.), Training Cognition: Optimizing Efficiency, Durability, and Generalizability (New York: Psychology Press, 2012), pp. 67-88.

6. Wickens, C. D., and A. L. Alexander, “Attentional Tunneling and Task Management in Synthetic Vision Displays,” The International Journal of Aviation Psychology, vol. 19, no. 2 (2009), pp. 182–99, http://doi.org/10.1080/10508410902766549.

7. Cahill, L., and L. McGaugh, “Mechanisms of Emotional Arousal and Lasting Declarative Memory,” Trends in Neurosciences, vol. 21 (1998), pp. 294-99.

8. Baumgartner, T., D. Speck, D. Wettstein, O. Masnari, G. Beeli, and L. Janncke, “Feeling Present in Arousing Virtual Worlds: Prefontal Brain Regions Differentially Orchestrate Presence Experience in Adults and Children,” Frontiers in Human Neuroscience, vol. 2 (2008), pp. 1-12.

9. Reger, G. M., K. M. Holloway, C. Candy, B. O. Rothbaum, J. Difede, A. A. Rizzo, and G. A. Gahm, “Effectiveness of Virtual Reality Exposure Therapy for Active Duty
Soldiers in a Military Mental Health Clinic,” Journal of Traumatic Stress, vol. 24 (2011), pp. 93-96.

10. Wiederhold, B. K., R. Gevirt, and M. D. Wiederhold, (2009) “Fear of Flying: A Case Report Using Virtual Reality Therapy with Physiological Monitoring,” CyberPsychology and Behavior, vol. 1 (2009), pp. 97-103.

11. Black, C. B., and D. L. Wright, “Can Observational Practice Facilitate Error Recognition and Movement Production?” Research Quarterly for Exercise and Sport, vol. 71, no. 4 (2000), pp. 331-39.

12. Lafleur, M. F., P. L. Jackson, F. Malouin, C. L. Richards, A. C. Evans, and J. Doyon, “Motor Learning Produces Parallel Dynamic Functional Changes During the Execution and Imagination of Sequential Foot Movements,” Neuroimage, vol. 16, no. 1 (2002), pp. 142-57.

13. Mulder, T., S. Zijlstra, W. Zijlstra, and J. Hochstenbach, “The Role of Motor Imagery in Learning a Totally Novel Movement,” Experimental Brain Research, vol. 154, no. 2 (2004), pp. 211-17.

14. Mirelman, A., I. Maidan, T. Herman, J. E. Deutsch, N. Giladi, and J. M. Hausdorff, “Virtual reality for gait training: can it induce motor learning to enhance complex walking and reduce fall risk in patients with Parkinson’s disease?” The Journals of Gerontology Series A: Biological Sciences and Medical Sciences (2010), glq201.

15. Sveistrup, H., “Motor Rehabilitation Using Virtual Reality,” Journal of Neuroengineering and Rehabilitation, vol. 1, no. 1 (2004), p. 1.

16. Adamovich, S. V., G. G. Fluet, E. Tunik, and A. S. Merians, “Sensorimotor Training in Virtual Reality: A Review,” NeuroRehabilitation, vol. 25 (2009), pp. 1-21.

17. Gibson, J. J., and J. Cornsweet, “The Perceived Slant of Visual Surfaces—Optical and Geographical,” Journal of Experimental Psychology, vol. 44 (1952), pp. 11-15.

18. Ooi, T. L., Wu, B., & He, Z. J., “Perceptual space in the dark affected by the intrinsic bias of the visual system,” Perception, vol. 35  (2006), 605-624.

19. Proffitt, D. R., Bhalla, M., Grossweiler, R., & Midgett, J., “Perceiving geographical slant,” Psychonomic Bulleting & Review, vol. 2 (1995), 409-428.

20. Durgin, F. H., Li, Z., & Hajnal, A., “Slant perception is categorically biased: Evidence for a vertical tendency,” Attention, Perception, and Psychophysics, vol. 72 (2010), 1875-1889.

21. Harguess, J., Bilinski, M., Nguyen. K.B., Powell, D., “Tactical 3D model generation using structure-from-motion on video from unmanned systems,” Proc. SPIE 9468, Unmanned Systems Technology XVII, 94680F (May 22, 2015); doi:10.1117/12.2178426.

About the authors:
Dr. Van Orden is the senior technologist for decision optimization systems at the Space and Naval Warfare Systems Center Pacific (SSC Pacific). Dr. Lukos is a computational neuroscientist, Dr. Gutzwiller is a cognitive scientist, and Heidi Buck is a senior engineer and the director of the Battlespace Exploitation of Mixed Realities Laboratory, all at SSC Pacific