Energy matters, Holospatial is a new 360 projection perspective for consumption- with Stopford Energy

With Stopford Energy and the SEND program we analysed the cost and energy saving capabilities of just *some* of the Holospatial applications. The idea was in focusing on just a couple of our integrated programs, how much time, energy and expenses can be saved- as a base line- something that all organisations can relate with.

We focused on communications and training, delivering both remotely. The measure we decided to focus on for carbon measurements were trees, we love trees here at Holospatial- especially the big ancient ones!

Stopford Energy delivered a cost-saving calculator that not only calculated cost-savings but it also counted by measurements of trees how much carbon your organisation can or has saved in x period of time.

This is a tool, and depending how well you use it with Holospatial 360 projection technology deliverables it's a bit of a weapon, because Holospatial enables you to change the way you do normal processes- site visits, risk-mitigation, project review, analytics and scenario based training, when combined and expanded throughout an organisation Holospatial projection rooms, walls and immersive spaces deliver an unlimited scope of savings- and we'll help guide you all the way, with immersive strategies and stats to show the savings, high-value and rapid ROI.

Learn more by contacting the team quoting SEND to get instant access to our carbon-cost calculator via makespacework@holospatial.co.uk

How virtual reality is redefining soft skills training- a Pwc Study.

Example pattern for desktop

Summary

7 minute read

The VR advantage

Employers are facing a dilemma: Their workforce needs to learn new skills, upgrade existing capabilities or complete compliance training, but may not be able to do so in person given the current environment. Yet, training is especially important now, with employees so keen to gain skills, and it may become even more critical when workers start returning to a changed workplace. So how can employers deal with the challenge?

One solution to this training problem comes from an unexpected place: virtual reality (VR).

VR is already known to be effective for teaching hard skills and for job skills simulations, such as a flight simulator to train pilots. But many employees also need to learn soft skills, such as leadership, resilience and managing through change.1:35Play Video

Tech effect

So how does VR measure up as a training tool for these and other soft skills?

PwC set out to answer this question with our study of VR designed for soft skills training. Selected employees from a group of new managers in 12 US locations took the same training — designed to address inclusive leadership — in one of three learning modalities: classroom, e-learn and v-learn (VR).

The results? The survey showed that VR can help business leaders upskill their employees faster, even at a time when training budgets may be shrinking and in-person training may be off the table, as people continue to observe social distancing.

VR learners were:
Statistics: VR learners

Five top findings about the value of VR in soft skills training

Here are five takeaways that can help you support your employees’ digital learning needs:

1. Employees in VR courses can be trained up to four times faster

US employees typically spend only 1% of their workweek on training and development, so employers need to be sure that they use that time productively. That’s where VR can help.

What took two hours to learn in the classroom could possibly be learned in only 30 minutes using VR. When you account for extra time needed for first-time learners to review, be fitted for and be taught to use the VR headset, V-learners still complete training three times faster than classroom learners. And that figure only accounts for the time actually spent in the classroom, not the additional time required to travel to the classroom itself.

Time to complete training
Pie chart: Employees VR courses

2. VR learners are more confident in applying what they’re taught

When learning soft skills, confidence is a key driver of success. In difficult circumstances, such as having to give negative feedback to an employee, people generally wish they could practice handling the situation in a safe environment. With VR, they can.

Because it provides the ability to practice in an immersive, low-stress environment, VR-based training results in higher confidence levels and an improved ability to actually apply the learning on the job. In fact, learners trained with VR were up to 275% more confident to act on what they learned after training — a 40% improvement over classroom and 35% improvement over e-learn training.Bar chart titledImprovement in confidence discussing issues and acting on issues of diversity and inclusion after the trainingDiscussing issuesActing on issuesClassroom166%198%E-learn179%203%VR245%275%Source: PwC VR Soft Skills training Efficacy Study, 2020

3. Employees are more emotionally connected to VR content

People connect, understand and remember things more deeply when their emotions are involved. (We learned that during the VR study and multiple BXT experiences, where we gathered different viewpoints and worked together to identify what matters most.) Simulation-based learning in VR gives individuals the opportunity to feel as if they’ve had a meaningful experience.

V-learners felt 3.75 times more emotionally connected to the content than classroom learners and 2.3 times more connected than e-learners. Three-quarters of learners surveyed said that during the VR course on diversity and inclusion, they had a wake-up-call moment and realized that they were not as inclusive as they thought they were.Bar chart titledAverage emotional connection felt to learning contentClassroom4.29E-learn5.29VR20.43Source: PwC VR Soft Skills training Efficacy Study, 2020

4. VR learners are more focused

Today’s learners are often impatient, distracted and overwhelmed. Many learners will not watch a video for its duration, and smartphones are a leading cause of interruption and distraction.

With VR learning, users are significantly less distracted. In a VR headset, simulations and immersive experiences command the individual’s vision and attention. There are no interruptions and no options to multitask. In our study, VR-trained employees were up to four times more focused during training than their e-learning peers and 1.5 times more focused than their classroom colleagues. When learners are immersed in a VR experience, they tend to get more out of the training and have better outcomes.

Comparison chart: How focused are VR learners?

5. VR learning can be more cost-effective at scale

In the past, VR was too expensive, complicated and challenging to deploy outside of a small group. Today, the cost of an enterprise headset ecosystem is a one-time fee of less than $1,000, and these units can be managed like any other enterprise mobile device and can be used repeatedly to deliver training. Studios of all sizes are developing compelling content, while vendors are creating software packages to enable non-VR developers to create their own content in a cost-effective way. Elsewhere, some big learning-management-system players are enabling VR content to be easily integrated into their platforms.

The value VR provides is unmistakable when used appropriately. In our study, we found that, when delivered to enough learners, VR training is estimated to be more cost-effective at scale than classroom or e-learning. Because VR content initially requires up to a 48% greater investment than similar classroom or e-learn courses, it’s essential to have enough learners to help make this approach cost-effective. At 375 learners, VR training achieved cost parity with classroom learning. At 3,000 learners, VR training became 52% more cost-effective than classroom. At 1,950 learners, VR training achieved cost parity with e-learn. The more people you train, the higher your return will likely be in terms of employee time saved during training, as well as course facilitation and other out-of-pocket cost savings.

Training modality cost per learner
Line graph: Training modality cost per learner

Building a blended learning curriculum

While VR will not replace classroom or e-learn training anytime soon, it should be part of most companies’ blended learning curriculum. VR learning differentiates itself by combining the elements of a well-planned BXT experience: business expertise to tackle challenges, a human-centered experience and the right technology to boost productivity without sacrificing quality. Ideally, an entire team would take this training and then have follow-up discussions to determine how they can apply the learned skills in their jobs.

VR can help people make more meaningful connections by allowing learners to practice skills that help them relate to diverse perspectives in the real world. For example, PwC developed a VR soft skills course that enables executives and staff to practice new sales approaches. Learners get to make a pitch to a virtual CEO, but if they rely on business-as-usual sales techniques, the virtual CEO asks them to leave her office. However, if learners apply skills that demonstrate how they can bring value to the CEO’s company, they get a “virtual contract” at the end of the conversation.

The simplicity of this technology is another good reason to start using VR at scale in your organization. In the study, our team was able to provision, deploy and manage a large fleet of VR headsets with a very small team. That success makes it easy to imagine a day when all employees will be issued their own headsets, along with the requisite laptops, on their first day on the job. That would be a truly new way of working.

Achieving Presence Through Evoked Reality

Jayesh S. Pillai1*, Colin Schmidt1,2 and Simon Richir1

The following report collates a variety of information and perspectives on our multiple realities and how these can impact an immersive experience but more so the human experience which is so critical to long-lasting and user-focused digital experiences that improve memorization, understanding and engagement as a whole.

The sense of “Presence” (evolving from “telepresence”) has always been associated with virtual reality research and is still an exceptionally mystifying constituent. Now the study of presence clearly spans over various disciplines associated with cognition. This paper attempts to put forth a concept that argues that it’s an experience of an “Evoked Reality (ER)” (illusion of reality) that triggers an “Evoked Presence (EP)” (sense of presence) in our minds. A Three Pole Reality Model is proposed to explain this phenomenon. The poles range from Dream Reality to Simulated Reality with Primary (Physical) Reality at the center. To demonstrate the relationship between ER and EP, a Reality-Presence Map is developed. We believe that this concept of ER and the proposed model may have significant applications in the study of presence, and in exploring the possibilities of not just virtual reality but also what we call “reality.”

Introduction

Research on presence has brought to our understanding various elements that certainly cause or affect the experience of presence in one way or another. But in order to evoke an illusion of presence, we in effect try to generate an illusion of reality different from our apparent (real world) reality through different mediations like Virtual Reality. The attempt to evoke an illusory reality is what brought researchers to think about presence in the first place. “Reality,” despite its being a major concept, is most often either overlooked or confused with other aspects that affect presence. To study presence we must first understand the reality evoked in one’s mind. It is this illusion of reality that forms a space-time reference in which one would experience presence. It is evident from the research in the field of virtual reality, that if a medium is able to create a convincing illusion of reality, there will certainly be a resultant feeling of presence. Various theories have been proposed, to explore and define the components of this mediated presence. We aim to abridge those theories in an efficient manner. Moreover, studies in the field of cognition and neuroscience confirm that the illusion of reality can as well be non-mediated (without the help of external perceptual inputs), that is purely evoked by our mind with an inception of corresponding presence. One of the most common but intriguing example of a non-mediated illusion of reality would be – a dream. This self evoking faculty of mind leading to the formation of presence is often neglected when observed from the perspective of virtual reality.

Sanchez-Vives and Slater (2005), suggest that presence research should be opened up, beyond the domain of computer science and other technologically oriented disciplines. Revonsuo (1995) proposed that we should consider both – the dreaming brain and the concept of Virtual Reality, as a metaphor for the phenomenal level of organization; they are excellent model systems for consciousness research. He argues that the subjective form of dreams reveals the subjective, macro-level form of consciousness in general and that both dreams and the everyday phenomenal world may be thought of as constructed “virtual realities.”

According to Revonsuo (2006), any useful scientific approach to the problem of consciousness must consider both the subjective psychological reality and the objective neurobiological reality. In Virtual Reality it’s not just the perceptual input and the technical faculties that contribute to a stronger illusion of reality but also various psychological aspects (Lombard and Ditton, 1997Slater, 20032009) relating to one’s emotion, attention, memory, and qualia (Tye, 2009) that help mold this illusion in the mind. In the case of non-mediated illusion of reality like dreams or mental imagery, the perceptual illusion is generated internally (Kosslyn, 19942005LaBerge, 1998). The dream images and contents are synthesized to fit the patterns of those internally generated stimulations creating a distinctive context for the dream reality (DR; Hobson and McCarley, 1977Hobson, 1988). Whether mediated or non-mediated, the illusion of reality is greatly affected by the context. “A context is a system that shapes conscious experience without itself being conscious at that time” (Baars, 1988, p. 138). Baars describes how some types of contexts shape conscious experience, while others evoke conscious thoughts and images or help select conscious percepts. In fact it’s a fine blend of perceptual and psychological illusions (explained in section The Illusion of Reality) that leads to a strong illusion of reality in one’s mind. We attempt to explore this subjective reality that is the fundamental source of experience for presence.

Presence and Reality

With the growing interest in the field of Virtual Reality, the subject of presence has evolved to be a prime area of research. The concept of presence, as Steuer (1992) describes, is the key to defining Virtual Reality in terms of human experience rather than technological hardware. Presence refers not to one’s surroundings as they exist in the physical world, but to the perception of those surroundings as mediated by both automatic and controlled mental processes.

Presence

Presence is a concept describing the effect that people experience when they interact with a computer-mediated or computer-generated environment (Sheridan, 1992). Witmer and Singer (1994) defined presence as the subjective experience of being in one environment (there) when physically in another environment (here). Lombard and Ditton (1997) described presence as an “illusion of non-mediation” that occurs when a person fails to perceive or acknowledge the existence of a medium in his/her communication environment and responds as he/she would if the medium were not there. Although their definition confines to presence due to a medium, they explained how the concept of presence is derived from multiple fields – communication, computer science, psychology, science, engineering, philosophy, and the arts. Presence induced by computer applications or interactive simulations was believed to be what gave people the sensation of, as Sheridan called it, “being there.” But the studies on presence progressed with a slow realization of the fact that it’s more than just “being there.” We believe that presence, whether strong or mild is the result of an “experience of reality.”

In fact “presence” has come to have multiple meanings, and it is difficult to have any useful scientific discussion about it given this confusion (Slater, 2009). There can be no advancement simply because when people talk about presence they are often not talking about the same underlying concept at all. No one is “right” or “wrong” in this debate; they are simply not talking about the same things (Slater, 2003). On the general problems in conveying knowledge due to the intersection of the conceptual, material, and linguistic representations of the same thing, there exists an attempt to explain the workings of communication and its mishaps (Schmidt, 1997a,b2009), which clearly states that scientists must always indicate which representation they speak of. In this article, we are mainly speaking about the phenomenon, which is the experience of presence.

Reality

The term “reality” itself is very subjective and controversial. While objectivists may argue that reality is the state of things as they truly exist and is mind-independent, subjectivists would reason that reality is what we perceive to be real, and there is no underlying true reality that exists independently of perception. Naturalists argue that reality is exhausted by nature, containing nothing supernatural, and that the scientific method should be used to investigate all areas of reality, including the human spirit (Papineau, 2009). Similarly a physicalist idea is that the reality and nature of the actual world conforms to the condition of being physical (Stoljar, 2009). Reality is independent of anyone’s beliefs, linguistic practices, or conceptual schemes from a realist perspective (Miller, 2010). The Platonist view is that reality is abstract and non-spatiotemporal with objects entirely non-physical and non-mental (Balaguer, 2009). While some agree that the physical world is our reality, the Simulation Argument suggests that this perceivable world itself may be an illusion of a simulated reality (SR; Bostrom, 2003). Still others would endeavor to say that the notion of physical world is relative as our world is in constant evolution due to technological advancement; also because of numerous points of view on its acceptation (Schmidt, 2008). Resolving this confusion about theories on reality is not our primary aim and is however beyond the scope of this study. So we reserve the term “Primary Reality” to signify the reality of our real world experiences, which would be explained later in this paper.

The Illusion of Reality

The factors determining the experience of presence in a virtual environment have been explored by many in different ways. For example, presence due to media has previously been reviewed as a combination of:

• Perceptual immersion and psychological immersion (Biocca and Delaney, 1995Lombard and Ditton, 1997).

• Perceptual realism and social realism (Lombard and Ditton, 1997).

• Technology and human experience (Steuer, 19921995).

• Proto-presence, core-presence, and extended-presence (Waterworth and Waterworth, 2006).

• Place illusion and plausibility illusion (Slater, 2009).

To summarize, the two main factors that contribute to the illusion of reality due to media are (1) Perceptual Illusion: the continuous stream of sensory input from a media, and (2) Psychological Illusion: the continuous cognitive processes with respect to the perceptual input, responding almost exactly how the mind would have reacted in Primary Reality. Virtual reality systems create highest levels of illusion simply because it can affect more senses and help us experience the world as if we were inside it with continuous updated sensory input and the freedom to interact with virtual people or objects. However other forms of media, like a movie (where the sensory input is merely audio-visual and there is no means to interact with the reality presented) can still create a powerful illusion if it manages to create a stronger Psychological Illusion through its content (for example a story related to one’s culture or past experiences, would excite the memory and emotional aspects). One of the obvious examples illustrating the strength of Perceptual illusion is a media that enforces stereoscopic view enhancing our depth perception (the illusion works due to the way our visual perception would work otherwise, without a medium). The resultant of the two, Perceptual Illusion and Psychological Illusion evokes an illusion of reality in the mind, although subjectively varying for each person – in strength and experience.

The Concept of “Evoked Reality”

We know that it’s not directly presence that we create but rather an illusion in our minds as a result of which we experience presence. When we use virtual reality systems and create convincing illusions of reality in the minds of users, they feel present in it. This illusion of reality that we evoke through different means in order to enable the experience of presence is what we intend to call “Evoked Reality (ER).” To explore this experience of presence we must first better understand what ER is.

As deduced earlier, all the factors influencing presence would essentially be categorized as Perceptual Illusion and Psychological Illusion. We believe that every media in a way has these two basic elements. Thus ER is a combined illusion of Perceptual Illusion and Psychological Illusion. This combined spatiotemporal illusion is what evokes a different reality in our minds (Figure 1) inducing presence.FIGURE 1

Figure 1. Spatiotemporal illusion due to mediation: reality so evoked generates the experience of presence

Evoked Reality

Even though the terms like telepresence and virtual reality are very recent, their evidence can be traced back to ancient times. The urge to evoke reality different from our Primary Reality (real world reality) is not at all new and can be observed through the evolution of artistic and scientific media throughout history. “When anything new comes along, everyone, like a child discovering the world, thinks that they’ve invented it, but you scratch a little and you find a caveman scratching on a wall is creating virtual reality in a sense. What is new here is that more sophisticated instruments give you the power to do it more easily. Virtual Reality is dreams.” Morton Heilig. (as quoted in Hamit, 1993, p. 57).

From Caves to CAVEs

Since the beginning of civilizations, man has always tried to “express his feelings,” “convey an idea,” “tell a story” or just “communicate” through a number of different media. For example, the cave paintings and symbols that date back to prehistoric times may be considered as one of the earliest forms of media used to convey ideas. As technology progressed media evolved as well (Figure 2) and presently we are on the verge of extreme possibilities in mediation, thus equivalent mediated presence.FIGURE 2

Figure 2. Evolution of media: from caves to CAVEs

We all like to experience presence different from our everyday happenings. To do so, we basically find methods to create an illusion of reality different from the reality that we are familiar with. With the help of different media we have already succeeded to evoke a certain amount of presence and we further aim for an optimum level – almost similar to our real world. Every form of mediation evokes a different kind of illusory reality and hence different degrees of presence. In the early examples of research in presence, studies were conducted based on television experiences before Virtual Reality became a more prominent field of research (Hatada and Sakata, 1980). While some types of media evoke mild illusion of presence, highly advanced media like Virtual Reality may evoke stronger presence. “But we must note that the basic appeal of media still lies in the content, the storyline, the ideas, and emotions that are being communicated. We can be bored in VR and moved to tears by a book” (Ijsselsteijn, 2003). This is precisely why the reality evoked (by media) in one’s mind depends greatly on the eventual psychological illusion, although it may have been triggered initially by a perceptual illusion. Media that could evoke mild or strong presence may range from simple paintings to photos to televisions to films to interactive games to 3D IMAX films to simulation rides to immersive Virtual Reality systems.

Evoked Reality

Evoked Reality is an illusion of reality, different from our Primary Reality (Physical Reality as referred in previous studies). ER is a transient subjective reality created in our mind. In the case of ER due to media, the illusion persists until an uninterrupted input of perceptual stimuli (causing perceptual illusion) and simultaneous interactions (affecting the psychological illusion) continue to remain. The moment at which this illusion of ER breaks due to an anomaly is when we experience what is called a “Break in Presence (BIP)” (Slater and Steed, 2000Brogni et al., 2003). Thus a BIP is simply an immediate result of the “Break in Reality (BIR)” experienced. Different kinds of media can evoke realities of different qualities and different strengths in our minds for different amount of time. It’s an illusion of space or events, where or during which we experience a sense of presence. Thus, it is this ER in which one may experience Evoked Presence (EP).

Evoked Presence

Depending on the characteristics of ER, an experience of presence is evoked. To be more specific this illusion of presence created by ER, we would like to refer to as EP. In this paper, the term “EP” would imply the illusion of presence experience (the sense of presence), while the term “presence” would be reserved for experience of presence in its broad sense (real presence and the sense of presence). EP is the spatiotemporal experience of an ER. We could say that so far it’s through the media like highly immersive virtual reality systems, that we were able to create ER that could evoke significantly strong EP.

Media-Evoked Reality and Self-Evoked Reality

As we saw before, ER is a momentary and subjective reality created in our mind due to the Perceptual Illusion and Psychological Illusion imposed by a media. It is clear that due to ER induced through media like Virtual Reality we experience an EP. This illusion of reality evoked through media, we would like to call “Media-Evoked Reality” or Media-ER.

As mentioned earlier, it’s not just through the media that one can evoke an illusion of reality. The illusion can as well be endogenously created by our mind evoking a seemingly perceivable reality; whether merely observable or amazingly deformable; extremely detailed or highly abstract; simple and familiar or bizarrely uncanny. Thus to fully comprehend the nature of presence, we must study this category of ER that does not rely on media. In fact, we always or most often undergo different types of presence without mediation. Sanchez-Vives and Slater (2005) proposed that the concept of presence is sufficiently similar to consciousness and that it may help to transform research within domains outside Virtual Reality. They argue that presence is a phenomenon worthy of study by neuroscientists and may help toward the study of consciousness. As rightly put by Biocca (2003), where do dream states fit in the two pole model of presence (Reality-Virtuality Continuum)? The psychological mechanisms that generate presence in a dream state have to be at least slightly different than psychological mechanisms that generate presence in an immersive, 3D multimodal virtual environment. Dreaming, according to Revonsuo (1995) is an organized simulation of the perceptual world and is comparable to virtual reality. During dreaming, we experience a complex model of the world in which certain types of elements, when compared to waking life, are underrepresented whereas others are over represented (Revonsuo, 2000). According to LaBerge (1998), theories of consciousness that do not account for dreaming must be regarded as incomplete. LaBerge adds, “For example, the behaviorist assumption that ‘the brain is stimulated always and only from the outside by a sense organ process’ cannot explain dreams; likewise, for the assumption that consciousness is the direct or exclusive product of sensory input.” It is very clear that one can think, imagine, or dream to create a reality in his mind without the influence of any media whatsoever. This reality evoked endogenously, without the help of an external medium, we would like to call “Self-Evoked Reality” or Self-ER (implying that the reality evoked is initiated internally by the mind itself).

Ground-breaking works by Shepard and Metzler (1971) and Kosslyn (19801983) in the area of Mental Imagery provide empirical evidence of our ability to evoke images or imagine stimuli without actually perceiving them. We know that Perceptual and Psychological Illusion are factors that affect Media-ER and corresponding EP. We believe that Self-ER essentially has Psychological Illusion for which the Perceptual element is generated internally by our mind. By generally overlooking or occasionally completely overriding the external perceptual aspects (sensorimotor cues), our mind endogenously creates the Perceptual Illusion required for the ER. It’s evident in the case of dreaming which according to LaBerge (1998), can be viewed as the special case of perception without the constraints of external sensory input. Rechtschaffen and Buchignani (1992) suggest that the visual appearance of dreams is practically identical with that of the waking world. Moreover, Kosslyn’s (19942005) work show that there are considerable similarities between the neural mappings for imagined stimuli and perceived stimuli.

Similar to Media-ER, one may feel higher or lower levels of presence in Self-ER, depending on the reality evoked. A person dreaming at night may feel a stronger presence than a person who is daydreaming (perhaps about his first date) through an on-going lecture with higher possibilities of BIRs. According to Ramachandran and Hirstein (1997) we occasionally have a virtual reality simulation like scenario in the mind (although less vivid and generated from memory representations) in order to make appropriate decisions in the absence of the objects which normally provoke those qualities. However, the vividness, strength, and quality of this internally generated illusion may vary significantly from one person to another. For example, the intuitive “self-projection” phenomenon (Buckner and Carroll, 2007; personal internal mode of mental simulation, as they refer to it) that one undergoes for prospection will certainly differ in experience and qualia from another person. It is a form of Self-ER that may not be as strong or prolonged as a picturesque dream, but strong enough to visualize possible consequences. It is clear that ER is either the result of media or induced internally. This dual (self and media evoking) nature of ER directs us toward a fresh perceptive – three poles of reality.

Three Poles of Reality

As we move further into the concept of ER and EP, we would like to define the three poles of reality to be clearer and more objective in the explanations that follow. Reality, as discussed earlier (in subsection Simulated Reality), has always been a term interpreted with multiple meanings and theories. To avoid confusion we would like to use an impartial term – “Primary Reality,” which would refer to the “experience” of the real world (or what we call physical world). It is the spatiotemporal reality in our mind when we are completely present in the real world. It would mean that any reality other than Primary Reality is a conscious experience of illusion of reality (mediated or non-mediated), or more precisely – ER.

Presence and Poles of Reality

Inherited from early telerobotics and telepresence research, the two pole model of presence (Figure 3) suggests that presence shifts back and forth from physical space to virtual space. Research on presence has been dominated ever since by this standard two pole psychological model of presence which therefore requires no further explanation.FIGURE 3

Figure 3. The standard two pole model of presence

Biocca (2003) took the study of presence model one step further. According to the model he proposed, one’s spatial presence shifts between three poles of presence: mental imagery space, the virtual space, and the physical space. In this three pole graphic model, a quasi-triangular space defined by three poles represented the range of possible spatial mental models that are the specific locus of an individual user’s spatial presence. His Model of presence attempted to offer a parsimonious explanation for both the changing loci of presence and the mechanisms driving presence shifts. Though the model explained the possibilities of presence shifts and varying levels of presence, it is vague about certain aspects of reality. It did not clarify what happens when we experience an extremely low level of presence (at the center of the model). How or why do we instantly return to our Primary Reality (in this model – Physical Space) as soon as a mediated reality or a DR is disrupted (Even though we may have entirely believed to be present in the reality evoked during a vivid dream)? Moreover it took into account only the spatial aspects but not the temporal aspects of shifts in presence.

We would like to define three poles of reality from the perspective of ER. The Three Pole Reality Model (Figure 4) may help overcome the theoretical problems associated with presence in the standard two pole model of presence as well as the model proposed by Biocca. According to us it’s the shifts in the type of reality evoked that create respective shifts in the level of presence evoked. For example if one experiences a highly convincing ER during a virtual reality simulation, he/she would experience an equivalently strong EP until a BIR occurs. The three poles of reality that we define are:

• DR (Threshold of Self-ER)

• Primary Reality (No ER)

• SR (Threshold of Media-ER)FIGURE 4

Figure 4. Three pole reality model

Primary reality

Primary reality refers to the reality of our real world. In Primary reality, the experience evoking stimulation arrives at our sensory organs directly from objects from the real world. We maintain this as an ideal case in which the stimulus corresponds to the actual object and does not deceive or misinform us. For instance, imagine yourself running from a tiger that is chasing you. It’s very near and is about to pounce on you. You scream in fear, and wake up to realize that you are safe in your bed, like every morning. You know for sure that this is the real world and the chasing tiger was just a part of the DR that your mind was in, some time before. So, Primary Reality is our base reality to which we return when we are not in any ER. In other words, when a BIR occurs, we come back to Primary Reality. Thus, as we can see in Figure 5, any point of reality other than Primary Reality is an ER. We could say that it’s this Primary Reality that we rely on for our everyday activities. It’s the reality in which we believe that we live in. Our experiences in this Primary Reality may form the basis for our experiences and expectations in an ER. For example, our understanding of the real world could shape how we experience presence in an immersive virtual reality environment, or even in a Dream. We could suppose that it’s the Primary Reality in which one believes this paper exists, or is being read.FIGURE 5

Figure 5. Three poles of reality: evoked reality constantly shifts between them

Simulated reality

In the case of Media-ER, an experience similar to Primary Reality is attempted to be achieved by interfering with the stimulus field, leading to an illusion of reality. For example virtual reality uses displays that would entirely mediate our visual perception in a manner that our head or eye movements are tracked and updated with appropriate images to maintain this illusion of receiving particular visual stimuli from particular objects. SR would be the most compelling and plausible reality that could ever be achieved through such mediations. It would be the reality evoked in our mind under the influence of a perfectly simulated virtual reality system. It’s the ultimate level that virtual reality aims to reach someday. At the moment an immersive virtual reality system, like flight simulators would be able to create ER considerably close to this pole. Its effectiveness is evident in the fact that pilots are able to perfectly train themselves being in that ER created by the simulator, helping them eventually to directly pilot a real plane. However, in the hypothetical condition of a perfectly SR our mind would completely believe the reality evoked by the simulation medium, and have no knowledge of the parent Primary Reality (Putnam, 1982Bostrom, 2003). In this state, it would be necessary to force a BIR to bring our mind back to Primary Reality. A Perfect SR is the Media-ER with strongest presence evoked and will have no BIRs.

Dream reality

In the case of Self-ER, the external perceptual stimuli are imitated by generating them internally. DR is an ideal mental state in which we almost entirely believe in the reality experienced, and accept what is happening as real. It does not return to the Primary Reality unless a BIR occurs. For instance, in the case of our regular dreams, the most common BIR would be “waking up.” Although internally generated, dream states may not be completely divorced from sensorimotor cues. There can be leakage from physical space into the dream state (Biocca, 2003). The experienced EP during a strong Dream can be so powerful that even the possible anomalies (causing BIRs) like external noises (an alarm or phone ringing) or even elements from physical disturbances (blowing wind, temperature fluctuations) may be merged into the DR, so as to sustain this ER for as long as possible. A Perfect DR is a Self-ER with the strongest presence evoked and will have no BIRs (similar to SR on the media side).

Presence Shifts and Presence Threshold

We are often under the effect of either Media or Self-ER. Imagine that we are not influenced by any mediation, nor any kind of thoughts, mental imagery, or dreams and our mind is absolutely and only conscious about the Primary Reality. In such an exceptional situation we would supposedly feel complete presence in the Primary Reality. Thus we presume that this perfect Primary Reality-Presence (or “real presence” as some may call) is the threshold of presence one’s mind may be able to experience at a point of time. It is clear that we can experience presence either in Primary Reality or in an ER. We cannot consciously experience presence in two or more realities at the same time, but our mind can shift from one reality to another voluntarily or involuntarily, thus constantly shifting the nature and strength of the presence felt. As pointed out by Garau et al. (2008), presence is not a stable experience and varies temporally. They explain how even BIPs could be of varying intensities. They also try to illustrate using different presence graphs the phenomenon of shifting levels of presence with the course of time and how subjective the experience is for different participants. Media like virtual reality aims to achieve the Presence Threshold at which one’s mind might completely believe the reality evoked. Though we have not however achieved it, or may never do, theoretically it’s possible to reach such a level of SR. Similarly if one experiences a Perfect Dream without any BIR, he/she would be at this threshold of presence exactly like being in the Primary Reality. SR and DR are the two extreme poles of reality at which the EP is at its threshold. These presence shifts due to the shifting of reality between these poles is something that we seldom apprehend, although we always experience and constantly adapt to them. In the following section we attempt to represent this phenomenon with a schematic model that would help us examine presence and reality from a clearer perspective.

Reality-Presence Map

Based on the three poles of reality and Presence Threshold we would like to propose the Reality-Presence Map (Figure 6). This map is a diagram of the logical relations between the terms herein defined. At any point of time one’s mind would be under the influence of either a Media-ER or a Self-ER when not in the Primary Reality (with no ER at all). Between the poles of reality, ER would constantly shift evoking a corresponding presence EP. As we can see in the map there is always a sub-conscious Parent Reality-Presence corresponding to the EP. This Parent Reality-Presence is very important as it helps our mind to return to the Primary Reality once the illusion of ER discontinues (or a BIR occurs). For a weaker EP, the Parent Reality-Presence is stronger (although experienced sub-consciously). When the ER manages to evoke very strong presence, the strength of Parent Reality-Presence drops very low (almost unconscious) and we start to become unaware of the existence of a Primary Reality; which is what an excellent immersive virtual reality system does. The shifting of presence is closely related to our attention. As soon as our attention from the ER is disrupted (predominantly due to interfering external perceptual elements), our attention shifts to the parent reality-presence sliding us back to Primary Reality (thus breaking our EP).FIGURE 6

Figure 6. Reality-presence map.

At the extreme poles, we would experience an Optimum Virtual Presence in a SR and similarly an Optimum Dream Presence in a DR. At these extreme points one may completely believe in the illusion of reality experienced almost or exactly like it is our Primary Reality, without the knowledge of an existing Parent Reality. At such a point, possibly a very strong BIR should be forced to bring one back to the parent Primary Reality. Experiencing a strong DR is one such example which many would relate to. During a very compelling but frightening dream, “waking up” acts as a very strong BIR, helping in the desperate attempt to leave the DR. After such a sudden and shocking change in reality most often our mind takes time to adjust back to the Primary Reality where everything would slowly turn normal and comforting.

Whenever there is an ER, the EP part of the presence (in the map) is what has our primary attention, and thus is the conscious part. Hence, the higher the EP, the lesser we are aware of our parent reality. Evidence of the sub-conscious Parent Reality-Presence can be observed in our experience of any media that exists today. Many studies have shown that in virtual environments, although the users behaved as if experiencing the real world, at a sub-conscious level they were certain that it was indeed “not” real. BIPs (that are used to measure presence) are in fact triggered by shifts in attention from the virtual world to the real world. For instance, virtual reality systems that help visually surround us completely with a virtual environment, elevates our presence (compared to a panorama view or television with visible frame boundaries) as our chances of shifting attention toward the real world drastically reduce in such higher levels of immersion (Grau, 2004Slater, 2009). Since ER is a subjective feeling, it can never be measured or even compared truthfully. This is the reason why we depend on the measurement of presence EP to determine if a system creates a stronger or weaker ER. Since the strength of presence itself is relative, the best way to measure is to compare between systems in similar context. “The illusion of presence does not refer to the same qualia across different levels of immersion. The range of actions and responses that are possible are clearly bound to the sensorimotor contingencies set that defines a given level of immersion. It may, however, make sense to compare experience between systems that are in the same immersion equivalent class” (Slater, 2009).

A major task for empirical consciousness research is to find out the mechanisms which bind the experienced world into a coherent whole (Revonsuo, 1995). This map provides a framework where the various experiences of ER could be mapped. Note that this map is not a “graph” that shows the strength of EP as directly proportional to the strength of ER. In fact it would help us represent every possible kind of ER as a point fluctuating between the two extreme poles of reality, with its respective strength of EP. We may refer to ER as stronger or weaker, when its qualia evoke stronger or weaker EP respectively. The Reality-Presence Map shows that if we can skillfully manipulate these qualia of ER (although subjective to each individual) bringing it closer to either of the two extreme poles, we may be able to evoke higher levels of EP. We should also note that, in order to introduce its basic concept, the Reality-Presence Map is presented here in a flattened two-dimensional manner. In the later sections we will illustrate how this map attempts to account for different experiences which were unable to be explained by previous presence models.

Subjectivity of Evoked Reality

As a matter of fact, the same mediation can create different subjective ER for different users depending on their personal traits. For example, two users reading the same book, or playing the same video game, or using the same Virtual Reality system would experience presence in an entirely different manner. EP (especially evoked by a medium) may be affected by one’s knowledge related to the context, degree of interest, attention, concentration, involvement, engagement, willingness, acceptance, and emotional attributes making it a very subjective experience. This is precisely why it is difficult to evaluate the efficiency of a particular Virtual Reality system by means of presence questionnaires. In fact many researchers confuse few of these terms above, with the concept of presence.

Therefore, to locate ER on the map, we have to examine “presence.” In fact finding reliable ways to measure presence has been a pursuit among many virtual reality and communication media researchers. In order to lead to testable predictions, we would rely on currently evolving measuring and rating systems, so as to determine an objective scale for presence (from Primary Reality to each extreme pole). Presently existing measuring techniques include questionnaires like “presence questionnaire” (Witmer and Singer, 1998Usoh et al., 2000), ITC-SOPI questionnaire (Lessiter et al., 2001), SUS questionnaire (Slater et al., 19941995), analysis of BIPs (Slater and Steed, 2000Brogni et al., 2003), objective corroborative measures of presence like psycho-physiological measures, neural correlates, behavioral measures, task performance measures (Van Baren and Ijsselsteijn, 2004), to mention a few. We can certainly predict the positions of different everyday experiences for a person in general (Figure 7); however it could be tested in the future only using above mentioned methods of measuring presence.FIGURE 7

Figure 7. An example range of Media-ER and Self-ER experiences mapped on reality-presence map, for an individual, that would occur at various points in time.

In virtual reality, distinction between “presence” and “immersion” has been made very clear previously in (Slater, 19992003). Though immersion (which is discussed extensively in the domain of virtual reality) is one of the significant aspects of EP, it falls under the technical faculty of a mediated system. “Immersion (in perceptual sense) provides the boundaries within which Place Illusion can occur” (Slater, 2009). Detailed aspects of presence related to immersive virtual reality are also discussed in (Slater et al., 2009). The characteristics like involvement, engagement, degree of interest, emotional response, may seem similar to presence, but are in fact different elements that may influence or be influenced by EP. The psychological impact of content, i.e., good and bad, exciting and boring, depends to a large extent on the form in which it is represented (Ijsselsteijn, 2003). Thus one of the most important aspects of Media-ER is its context. In most cases it forms a reference in one’s mind to how they may experience ER and hence the presence evoked. For example, in some contexts, especially in art and entertainment, it would invoke a “genre” that plays a major role in its communication. The context (whether artistic expression, communication, entertainment, medical application, education, or research) should be a core concern while designing a Virtual Reality System, in order to bring about a subjectively higher quality of ER. A descriptive account on the importance of context in Self-ER is given by Baars (1988). With examples of different sources and types (perceptual and conceptual) of contexts, he demonstrates how unconscious contexts shape conscious experience. In addition, he explains the importance of attention, which acts as the control of access to consciousness. Attention (in both Media-ER and Self-ER) can direct the mind toward or away from a potential source of qualia. The experience of an ER therefore depends also on the voluntary and involuntary characteristics of one’s attention.

According to the concept, our presence shifts continuously from one ER to another and does not require passing through Primary Reality to move from one side to another. This map does not provide a temporal scale per se. However in future (with the advancements in presence measurement techniques), the map can be used to trace presence at different times to study the temporal aspects of presence shifts.

Evoked Reality within Evoked Reality

There is an important question that arises now. How can we account for our thoughts or mental imagery experiences during VR simulations, games, movies, or most importantly books? It is the phenomena of experiencing Self-ER during a Media-ER experience.

Self-ER within media-ER

Whenever we experience an ER, our mind is capable of temporarily presuming it as the parent reality and reacting accordingly. The better the ER and stronger the EP, the easier it is for our mind to maintain the illusion. In such states Media-ER is experienced as a temporarily form of Primary Reality, and we are able to experience Self-ER within it. In fact that is the core reason why virtual reality systems and virtual environments work. This phenomenon is clearly displayed in such experiences, where the users require thinking, planning, and imagination in order to navigate in the virtual world, just like they would do in the real world. Below, it is demonstrated how this phenomenon may be represented with respect to the Reality-Presence Map (Figures 8 and 9). This scenario will ultimately be classified under Media-ER.FIGURE 8

Figure 8. An example of how Media-ER would temporarily act as a version of primary reality

Figure 9. An example of presence shift due to Self-ER within Media-ER (for e.g., thinking within a virtual environment).

Self-ER triggered during media-ER

“Self-ER within Media-ER” should be distinguished from the phenomenon of “Self-ER triggered during Media-ER.” This is similar to a well-known case of Self-ER – the phenomenon of mind-wandering that temporarily detaches us from the Primary Reality. It is otherwise known as “task unrelated thought,” especially with respect to laboratory conditions. Smallwood et al. (2003) define it as the experience of thoughts directed away from the current situation. It is in fact a part of (and closely related to) our daily life experiences (Smallwood et al., 2004McVay et al., 2009). Although studies on mind-wandering are principally focused on shifts between Self-ER and tasks relating to Primary Reality (falling under usual case of Self-ER experience – Figure 10), we propose that they are applicable to similar cases in Media-ER as well. It has been suggested that this involuntary experience may be both stable and a transient state. That means we can experience a stable EP during mind-wandering or an EP oscillating between the Self-ER, Media-ER, and the Primary Reality.FIGURE 10

Figure 10. The usual case of presence shift from primary reality to Self-ER

Therefore, when an unrelated Self-ER is triggered while experiencing a Media-ER (or when Self-ER within Media-ER traverse the presence threshold and becomes unaware of the Media-ER itself), it should be considered under the case of Self-ER (Figure 11).FIGURE 11

Figure 11. An example of presence shift toward Self-ER triggered during Media-ER.

Discussion

Our attempt was a novel idea, to fit together different concepts regarding presence into a single coherent graphical representation. Although this concept of ER and EP along with the proposed map provides us a simplified way to look at reality and presence, it raises plenty of questions. Can the experience of an altered state of consciousness (ASC) like hallucination, delusion, or psychosis due to mental disorders be a kind of Self-ER? Revonsuo et al. (2009) redefines ASC, as the state in which consciousness relates itself differently to the world, in a way that involves widespread misrepresentations of the world and/or the self. They suggest that, to be in an ASC is to deviate from the natural (world-consciousness) relation in such a way that the world and/or self tend to be misrepresented (as evident in reversible states like dreaming, psychotic episodes, psychedelic drug experiences, epileptic seizures, and hypnosis). According to Ramachandran and Hirstein (1997) we have internal mental simulations in the mind using less vivid perceptual attributes, in the absence of the regular external sensory inputs. If they possessed full-strength perceptual quality, that would become dangerous leading to hallucinations. They argue that in cases like temporal lobe seizures, this illusion (Self-ER) may become indistinguishable to real sensory input losing its revocability and generating incorrect sense of reality (creating a permanent ER situation that makes it difficult to return to Primary Reality). So can hallucinations due to Self-ER be compared to Augmented Reality due to Media-ER?

In contrast to Presence, is there an “Absence” and do we experience that? If so, how? Can it be compared to a dreamless sleep? Can Presence Threshold itself be subjective and differ from person to person? With reference to the Reality-Presence Map, is there a possibility of an experience analogous to uncanny valley when ER is nearest to the two extreme poles? Is this the reason why many experience anomalies during exceptionally vivid nightmares or lucid dreams? Similarly on the Media-ER side, can simulator sickness due to inconsistencies during virtual reality simulations be compared to this phenomenon? Other than the obvious difference between Media-ER and Self-ER that was discussed before, they have another main differentiation. In most cases of Media-ER, multiple users could share the experience of a common ER at the same time (naturally, with subjective differences, especially due to psychological illusion). While in the case of Self-ER, every person’s mind experiences unique ER. Thus a Dream is typically an individual experience (as far as our present technological advancements and constraints suggest), while SR may be shared.

Furthermore, the Reality-Presence Map helps us investigate into potential ideas on Reality, for instance the possibility of Simulation within a Simulation (SWAS). The Map could be extended to and be applicable for any level of reality, in which we believe there’s a Primary Reality – the base reality, to which we return to in case of absence of any form of ER. Let’s imagine that someday we achieve a perfect SR. As per our proposition, one’s mind would accept it as the Primary Reality as long as the experience of presence continues (or till a “BIR” occurs). It would imply that at such a point, one can experience presence exactly as in the Primary Reality. In this perfect SR if one experiences Media-ER (e.g., virtual reality) or Self-ER (e.g., dream), as soon a BIR occurs they return back to it since it’s the immediate Parent Reality. Figure 12 attempts to illustrate such a situation with DR and SR as two orthogonal Poles of Reality. Similarly in the Self-ER side, one’s mind could experience a Dream within a Dream (DWAD). When one wakes up from such a dream, he could find himself in the parent DR from which he would have to wake up again into the Primary Reality. Can this be how people experience such false awakenings [a hallucinatory state distinct from waking experience (Green and McCreery, 1994)]? Figure 13 attempts to illustrate such a situation of DWAD.FIGURE 12

Figure 12. Simulation within a simulation

Figure 13. Dream within a dream

In fact it makes us curious about the even bigger questions. Can there be an ultimate reality beyond Primary Reality or even beyond the scope of this map. The Simulation argument claims that we are almost certainly living in a computer simulation (Bostrom, 2003), in which case what we believe to be our Primary Reality might itself be a SR [similar to Brains in a vat scenario (Putnam, 1982)]. Metzinger (2009) proposes that our experience of the Primary Reality is deceptive and that we experience only a small fraction of what actually exists out there. He suggests that no such thing as “self” exists and the subjective experience is due to the way our consciousness organizes the information about outside world, forming a knowledge of self in the first person. He claims that everything we experience is in fact a SR and the on-going process of conscious experience is not so much an image of reality as an “ego tunnel” through reality. So, is our Primary Reality in fact the base reality? Or are we always under an ER of some kind? Figure 14 attempts to put together different levels of reality as a Reality Continuum. It would make us wonder if it’s probable, to how many levels would one be able to go? Do we already visit them unknowingly through our dreams? Would the levels of reality in the figure be represented as a never ending fractal structure? In any case, will we be able to understand someday all these aspects of our experience of reality?FIGURE 14

Figure 14. Reality continuum (illustrating the levels of reality).

Conclusion

In this paper we explored presence and different elements that contribute to it. Presence is not just “being there” but a combination of multiple feelings and most importantly “experiencing the reality.” The two main factors affecting presence due to mediation are Perceptual Illusion and Psychological Illusion. These factors evoke an illusion of reality in our mind in which we feel presence. We are constantly subjected to such illusions of reality, during which we experience presence differently from that of our apparent real world. This illusion of reality is called ER.

Evoked Reality is not just media-evoked but can also be self-evoked. Media-ER may range from the mild effect of a painting to an extremely plausible immersive Virtual Reality experience while a Self-ER may range from a simple thought to an exceptionally believable DR (the strength of ER may not necessarily be in the same order, as it depends on one’s qualia and personal characteristics). This dual nature of ER led us to define three poles of reality: primary reality – the unaltered and unmediated Real World, SR – the ultimate Media-ER (a perfect Virtual Reality condition) and DR – the ultimate Self-ER (a perfect dream condition). Thus ER is an illusion of reality formed in our mind, which is different from Primary Reality. It’s a combined illusion of space and events, or at least one of them. It is in this ER, one would experience presence. Thus EP is the spatiotemporal experience of an ER.

The proposed Reality-Presence Map attempts to graphically illustrate the concept of ER and EP. This map provides a framework where the various experiences of ER could be mapped. The subjectivity of ER qualia and how these subjective factors affect Media-ER and EP were explained. The idea of Presence Threshold was also explored which formed the basis for different levels of EP and temporal Presence Shifts. Different possibilities like SWAS and DWAD conditions were discussed with respect to the proposed model. However certain elements still demand clarifications to fill in the theory. The concept presented here is an inception of a potential future research. We believe that ER and the proposed Reality-Presence Map could have significant applications in the study of presence and most importantly in exploring the possibilities of what we call “reality.”

The full report including references can be found here

Human Factors Research in Immersive Virtual Reality Firefighter Training: A Systematic Review

Steven G Wheeler1Hendrik Engelbrecht1 and Simon Hoermann1,2*

The following report details a deep study into immersive VR training systems focused on high-risk environments, the use of HMD's as well as projection environments and considers the huge variety of variables to be accounted for in one of the most risky and wide-ranging training scenarios imaginable. This makes the report a real standout reading for anyone looking to implement VR training solutions via projection, HMD or most ideally- both. Enjoy!

Immersive virtual reality (VR) shows a lot of potential for the training of professionals in the emergency response domain. Firefighters occupy a unique position among emergency personnel as the threats they encounter are mainly environmental. Immersive VR therefore represents a great opportunity to be utilized for firefighter training. This systematic review summarizes the existing literature of VR firefighting training that has a specific focus on human factors and learning outcomes, as opposed to literature that solely covers the system, or simulation, with little consideration given to its user. An extensive literature search followed by rigorous filtering of publications with narrowly defined criteria was performed to aggregate results from methodologically sound user studies. The included studies provide evidence that suggests the suitability of VR firefighter training, especially in search and rescue and commander training scenarios. Although the overall number of publications is small, the viability of VR as an ecologically valid analog to real-life training is promising. In the future, more work is needed to establish clear evidence and guidelines to optimize the effectiveness of VR training and to increase reliable data through appropriate research endeavors.

1 Introduction

Virtual reality (VR) technology has been evolving rapidly over the past few years. VR is making its way into the consumer market with affordable headsets in a variety of price ranges and research in the domain of the application of VR is at a record pace (Anthes et al., 2016).

Previous studies suggest that VR is a valuable training tool in the medical, educational, and manufacturing domains, such as the training of laparoscopic surgery (Alaker et al., 2016), in cognitive behavior therapy (Lindner, 2020), the creation of empathy in the user (Kilteni et al., 2012Shin, 2018), or as a teaching tool in the manufacturing domain (Mujber et al., 2004). Research in the field of military applications has used VR successfully for the treatment of adverse mental conditions (Rizzo et al., 2011) as well as increasing mental preparedness of soldiers (Wiederhold and Wiederhold, 2004Stetz et al., 2007) (known as stress inoculation training). VR has also been successfully used to teach correct safety procedure in hazardous situations (Ha et al., 2016Oliva et al., 2019Ooi et al., 2019).

VR enables users to be placed into a believable, customizable, and controllable virtual environment. Due to this, there is great interest in the educational domain thanks to the possibility of virtual worlds enabling experiential learning. As defined by Kolb (1984), experiential learning is achieved through the transformation of experience into knowledge. There has been considerable interest in applying virtual worlds for experiential learning; see, for example, Jarmon et al. (2009) or Le et al. (2015).

Applying this to the firefighting context, the possibility of enabling experiential learning in a virtual space is a great opportunity for hands-on training that does not need to be reliant on the personnel, resources, and budget for training firefighters. VR might therefore enable cost-effective and frequent training for a large variety of scenarios. Due to its immersive properties, VR is gaining traction in the training of high-risk job domains. Stimulating the feeling of presence, virtual environments can arouse physiological responses as indicators of stress on par with real-life arousal (Wiederhold et al., 2001Meehan et al., 2003), which shows promise for VR possibly being an ecologically valid analog to real-life training exercises. Firefighter trainees are faced with a multitude of environmental hazards making the use of VR for training a natural extension of what has been shown in other domains. Yet, with the variety of threats faced, the difference in skills needed and the mental demands seemingly unique, the effectiveness of VR training for firefighting needs to be looked at as an independent investigation.

This article explores and analyzes the field of firefighter VR training using a systematic search procedure. To obtain relevant research that enriches the pool of evidence in this domain, the researchers are purposefully restricting the analysis to research pertaining to the domain of human factors with the goal of assessing the impact on end-users within the target population.

2 Definitions

2.1 Immersive and Non-Immersive Virtual Reality

For this article, the definition for immersive VR concerns itself with the direct manipulation of the environment using input and visualization devices that respond to the natural motion of the user (Robertson et al., 1993). Several researchers have shown that non-immersive, monitor-bound simulations offer possibilities for training firefighters [see, for example, (St Julien and Shaw, 2003Yuan et al., 2007van Berlo et al., 2005)]. However, as immersive VR technology has many distinctive properties and brings with it many unique challenges and considerations—for example, the issue of cybersickness (LaViola, 2000) or the challenge of creating effective input methods in VR (Choe et al., 2019)—we argue that it needs to be treated as a separate inquiry. Therefore, VR setups utilizing head-mounted displays and CAVE systems (Cruz-Neira et al., 1993) are the focus of this inquiry, and desktop monitor-bound simulations are not within the scope of this investigation.

2.2 Presence

Presence is the result of immersion in the virtual environment where the user feels a sense of being physically part of the virtual world as if they have been transported to another place, independent from the current real-world location (Slater and Usoh, 1993Lombard and Ditton, 1997). Due to this, VR has been shown to be able to stimulate similar responses and behavior in reactions to hazards and risks as they would in real-life (Alcañiz et al., 2009). As such, effective transmission of presence has been found to make VR a safe and effective medium to train personnel in high-risk situations (Amokrane et al., 2008) and, therefore, is an important factor to consider in the discussion of firefighting training—a job domain with a high level of risk to the personnel.

2.3 Ecological Validity

Differing from both immersion and presence, we judge ecological validity to refer to how representative the virtual activities are of their real-life counterparts (Paljic, 2017). As the main focus of this inquiry is specifically looking at VR as a predictive tool for training, we deem it important to consider the ecological validity of each study to judge its efficacy in real-world applications. This is not to be confused with simply considering the physical fidelity, or graphical realism, of the virtual environment, which has been shown to have a limited impact on the user experience (Lukosch et al., 2019). Rather, this article directly considers the input methods used, the equivalent real-world equipment and the relevance of the virtual task to real-world situations.

2.4 Training, Aids, and Post-Hoc Applications

This article looks into the application of training, i.e., the acquisition of mental and physical skills, prior to the usage of such skills in the real world. This means that applications only for the use during deployment are not part of the inquiry, since this review is strictly on the potential for acquisition and training of skills and not the improvement of the execution with the usage of VR technology. The same principles apply to post-hoc applications, which concern themselves with either the treatment or post-incident analysis of factors resulting from the work itself. While there is an overlap between post-hoc applications used to reinforce skills that have already been executed and trained, the focus of these applications is not on the acquisition and maintenance of skills through VR, but represents a combination of approaches. We argue that this, while naturally a part of future inquiries, introduces too much noise into the validation of training in this domain.

2.5 Human Factors Evaluations

In this systematic review, the term “human factors” is being used in relation to the evaluation of behavioral and psychological outcomes of training applications. The term thereby extends functionality considerations beyond a mere systems perspective; the literature that only focuses on the purely functional aspects of training execution in the virtual environment, without considering the end-user, is excluded from this investigation. We aim to clarify this due to some work conflating functionality evaluations with training effectiveness. In these cases, the effect of virtual training execution on the user is often not specifically considered. The successful completion of a virtual task alone is often deemed as proof to the ecological validity of simulation. The impact of integrating existing training routines into virtual worlds needs a holistic investigation that encompasses functional, as well as psychological and behavioral outcomes for assessing their effectiveness in the human factors domain.

3 Population Considerations

3.1 Emergency Response and VR Research

There has been a lot of interest in VR technology for the training of emergency response employees. For example, the development of VR disaster response scenarios has gained popularity [see, for example, (Chow et al., 2005Vincent et al., 2008Sharma et al., 2014)] since it enables cost-effective training of large-scale exercises and offers immersive properties that are difficult to replicate in desktop monitor-bound training.

The term emergency response is an umbrella term that describes any profession that works in the service of public safety and health often under adverse or threatening conditions. Included under this umbrella term are professions such as emergency medical technicians, police officers, or firefighters. While these are all distinct professions, there is an overlap in the kind of situations all three encounter, such as traffic accidents or natural disasters. Hence, research in this domain is often grouped under this umbrella term, with generalizations being made across the entire domain.

While there is an overlap in skills and mental demands, the findings in one area should not be generalized with undue haste to other areas. Emergency medical technicians (EMTs) are primarily faced with mental strains in the form of potentially traumatizing imagery (e.g., in the form of heavily injured patients) at the scene. While there can be threats to EMTs during deployment, sprains and strains are most common and injury rates are potentially lower than those of other emergency response occupations (Heick et al., 2009). The skills needed are largely independent of the environment, as they apply to the handling of the patient directly.

Police officers, on the other hand, often deal with very direct threats in the form of human contact. Suspects, or generally people causing a disturbance, can pose a threat to the officer if the situation gets out of control. The environmental threats faced only account for a small fraction in the case of, for example, traffic accidents or disaster response, with the risk of injury being highest for assaults from non-compliant offenders (Lyons et al., 2017). Similarly to EMT’s, the skills needed are not completely independent of the environment, but interpersonal contact plays the main factor in the everyday life of the police officer when it comes to occupational threats.

This review concerns itself with the application of VR training for firefighters exclusively. The work environment of firefighters is hypothesized to be unique due to the nature of the threats and the skills applied being heavily dependent on the interaction with the environment. Firefighters work in an environment full of dangers. Fire, falling objects, explosions, smoke, and intense heat are only some of the large variety of environmental threats faced (Dunn, 2015). In 2017 alone, a total of 50,455 firefighters were injured during deployment in the United States. Furthermore deployment resulted in 60 deaths in 2017. Even during training itself, 8,380 injuries and ten deaths were recorded in 2017 (Evarts and Molis, 2018Fathy et al., 2018). Numerous threats are faced by firefighters, and with high potential risk to life and well-being, ecologically valid training is necessary. Training in an environment that adequately represents environmental threats faced during deployment is vital to learning skills.

While a transfer of knowledge gained in any emergency response research can be valuable for informing system design in other areas, the independent aggregation of results remains important for obtaining evidence that can be used as a building block for future work. A high level of scrutiny is required when it comes to the development of new technologies, since the failure to do so can impact the safety of the workforce in the respective occupation. We therefore argue that VR research should treat these occupations as separate fields of inquiry when assessing the impact on human factors.

4 Search Strategy

This section describes the details of the publication search and selection strategy and explains the reasons for their application in this systematic review.

4.1 Search-Terms and Databases

Firefighter research within human–computer interaction (HCI) is a multidisciplinary field; hence, this review aims to capture work published in engineering and computer science, as well as in all life-, health-, physical-, and social-sciences fields. While this has resulted in only a few unique additions to the search results, this inclusive approach was chosen to prevent the omission of potentially relevant work. The following databases were used for the systematic search:

• Scopus (Elsevier Publishers, Amsterdam, Netherlands)

• Ei Compendex (Elsevier Publishers, Amsterdam, Netherlands)1

• IEEE Xplore (IEEE, Piscataway, New Jersey, United States)

• PsycINFO (APA, Washington, Washington DC, United States)

For the purpose of this review, we aimed to purposefully narrow the scope of the assessed literature to human factors evaluation of training systems for fire service employees using immersive virtual reality technology. As such, the search terms had to be specified and justified with regard to that goal.

4.1.1 Technology

The value of immersive VR for training simulations lies in the match of immersive properties with the threats faced by the target population. With a large part of the most dangerous threats encountered by firefighters being environmental in nature, there is an opportunity for immersive VR to make a unique contribution to training routines. While mixed reality systems might arguably be able to present threats to trainees with similarly high physical fidelity, results obtained from evaluations deploying these technologies in the firefighting domain might not be transferable to immersive VR training and further increase noise for establishing a clear baseline for the utility of this technology.

For this review, the following terms were used as part of the systematic search:

virtual reality; VR

4.1.2 Target Population

As discussed previously, the population of firefighters occupies a unique position within the emergency response domain with regard to threats faced and skills needed. To capture the entirety of the target population, the terms used in the search were kept broad and only included a few specialized terms, such as land search and rescue (LandSAR), which revealed additional citations that were not covered by the other, more general, search terms. The broadness of the terms used means that more additional manual processing and filtering of the resulting citations will be needed, but this was deemed necessary to prevent any possible omission of work in this domain.

For this review, the following terms were used as part of the systematic search:

firefight∗; fire service∗; fire fight∗; fire department; landsar; usar

4.1.3 Aim

The aim of this article was to capture any possible application of immersive VR systems for training purposes. Training in this case is defined as any form of process applied with the aim of improving skills (mental and physical) or knowledge before they are needed. During preliminary searches, we found that several terms overlapped with the terms already being used, resulting in no new unique citations, and were therefore excluded from the systematic search, namely, teach∗coach∗, and instruct∗.

For this article, the following terms were used as part of the systematic search:

train∗; educat∗; learn∗; habituat∗; condition∗; expos∗; treat∗

4.2 Selection Criteria

4.2.1 Target Population

The target population of the citation needs to be concerned with fire service employees. This does include any kind of specialization that can be obtained within the fire service and extends throughout ranks. We excluded articles that exclusively investigated other emergency response personnel or unrelated occupations.

4.2.2 Technology Used

Immersive virtual reality, i.e., a CAVE system or head-mounted display, needs to be used as the main technology in the article. Augmented- or mixed-reality, as well as monitor-bound simulations, are not within the scope of this review.

4.2.3 Practical Application

The aim of this investigation is to evaluate the scope of research done in the domain of human factors research. For an article to be included in this review, it needs to be aimed towards a practical application of technology for the fire service. Pure system articles, e.g., development of algorithms, will be excluded.

4.2.4 Sample

The sample used during evaluation needs to represent the population of firefighters. This does include the approximation of the target population by using civilian participants to act as firefighters. When proxies were used instead of firefighters, this limitation needed to be clearly acknowledged as a potential limitation.

4.2.5 Aim

The research needs to be on a training system that is concerned with the acquisition or maintenance of skills or knowledge before an event demands them during real deployment. Systems intended for use during deployment, e.g., technology to improve operations in real life, or post deployment, e.g., for the treatment of conditions such as PTSD, will be excluded.

4.2.6 Measures

The research needs to evaluate the impact of the system with relevant outcome measures for the human factors domain. Articles with a sole focus on system measures with no, or vastly inadequate, user studies will be excluded from the review.

4.3 Process and Results

The process of the systematic search can be seen in Figure 1.

FIGURE 1. Process overview for systematic search.

First, the search terms were defined to specify the scope of the review, while retaining a broad enough search to obtain all relevant literature. Databases were selected based on their coverage of relevant fields with expected redundancy among the results. The search procedure for all databases was kept as similar as possible. The search terms were used to look for matches in the title, abstract or associated keywords of the articles. Only English language documents were included in the review, and appropriate filters were set for all database search engines. While the exact settings differed slightly depending on the database, as certain document types were grouped together, only journal articles, conference articles and review articles published up to the writing of this article2 were included as part of the review. The total amount of citations identified was 300. After the removal of duplicates, the citation pool was reduced to 168 articles.

Next, for the first round of applying the exclusion criteria, as specified above, the abstracts and conclusions were evaluated and articles were removed accordingly. Afterward, the remaining 110 articles were evaluated based on the full text. Any deviation from the above mentioned criteria resulted in the exclusion of the publication. This was also applicable to work that, for example, failed to describe the demographics of participants entirely (i.e., it is unclear whether members of the target population were sampled) or did not describe what hardware was used for training. The latter becomes especially troubling with the term virtual reality having been used interchangeably with monitor-bound simulations in many bodies of work. In these cases, some articles needed to be excluded, because no further information was given as to whether or not immersive or non-immersive virtual reality was utilized. The number of citation left after this was six. For all six publications, an additional forward and backward search was carried out to ensure that no additional literature was missed.

The following literature review is based on a total of six publications (see Table 1). The relatively low number of selected publications in this specialized domain allowed us, in addition to just provide summaries and interpretation of study results, to make suggestions about what can be learned from the systems, the methodologies applied, and the results obtained.TABLE 1

This image has an empty alt attribute; its file name is frvir-02-671664-t001.jpg

TABLE 1. Selected Literature for Review. For more detail, please refer to the Supplementary Material

5 Literature Review

5.1 Overview and Type Description

The six studies selected are all investigating the effect of VR training with regard to human factors considerations (see Table 1). Four of the studies include a search and rescue task in an urban environment (i.e., an indoor space), and two studies investigate aerial firefighting. Three of the studies are concerned with the training of direct firefighting tasks. The two studies by Clifford et al. (2018b,a) are dealing with the training of aerial attack supervisors who coordinate attack aircrafts for aerial firefighting, and the study by Cohen-Hatton and Honey (2015) deals with the training of commanders for urban scenarios.

5.2 Results

5.2.1 Search and Rescue

The studies by Bliss et al. (1997)Backlund et al. (2007), and Tate et al. (1997) were grouped together as they all investigate urban/indoor search and rescue scenarios. Bliss et al. (1997) focused on navigational training in a building with a preset route within a VR environment, using an HMD and a mouse for movement input, and contrasted this with either no training at all or with training the memorization of the route using a blueprint of the building. All three groups were subsequently assessed in a real building with the same layout as the training materials. The participants were told to execute a search and rescue in this building, with the two trained groups being advised to take the route that was trained prior. As expected, both the VR and blueprint training groups outperformed the group that received no prior training, as measured by completion time and navigation errors made. No difference between the blueprint and VR training groups was observed. Also of note is the correlation obtained between frequency of computer use and the test performance, indicating that familiarity and enjoyment of computer use do have an effect on training outcomes in VR. The researchers further note that the familiarity that firefighters have with accessing blueprints prior to entering a search and rescue scenario might have also led to the results obtained. Interesting to note is that the cost, difficulty in implementation, and interaction fidelity are constraints that might have influenced the outcomes.

While Bliss et al. (1997) were more concerned with the fidelity of simulating a real scenario (without augmenting the content in any way), Backlund et al. (2007) specifically aimed to create a motivating and entertaining experience to increased training adherence, while eliciting physical and psychological stress factors related to a search and rescue task; they made use of game elements, such as score and feedback. Participants were divided into two groups, with one group receiving two training sessions using the VR simulation (called Sidh) before executing the training task in a real-world training area. The second group first performed the task in the training area and then did a single training session in the VR simulation. The VR environment was constructed by projecting the environment on four screens surrounding the participant. The direction of the participant was tracked, and movement enabled by accelerators attached to the boots (enabling walking in place as a locomotion input). The participants were tasked with carrying out a search and asked to evacuate any victims they came across. A score was displayed to participants as feedback after completion of the task, which factors in the total area searched, remaining time, and number of attempts. Physical skills, such as body position and environment scanning, were tracked to allow for feedback mechanisms. The researchers found the simulation greatly increased learning outcomes, stating that performance in the simulation was significantly better in the second session compared to the first. They highlight that the repeated feedback obtained during the first sessions resulted in a clear learning effect, which made participants more thorough in their second search a week later. Additionally, the tracking of the body position of participants, and relating appropriate feedback, resulted in the majority keeping a low position during the task, i.e., applying a vital safety skill. According to qualitative data, physical stress was elicited successfully. In addition, more than two thirds of the participants stated that they learned task relevant knowledge or skills. Participants generally stated that the simulation was fun.

The third study investigated the training of a search and rescue task in a novel environment, namely that of a Navy vessel (Tate et al., 1997). While not a traditional search and rescue task, i.e., the task was concerned with locating and extinguishing the fire while navigating the interior correctly, the general nature of the task, traversing an indoor environment for firefighting tasks under limited visibility, does align with the other two studies discussed in this section. The participants were split into two groups. For phase one of the experiment, all participants received a briefing that included the tasks to be performed and diagrams of the route to follow. The experimental group received additional training using a VR simulation that recreated the ships interior, while the control group received no additional training. For the evaluation, all participants were tasked with traversing the ship to a predefined location, and the time of completion was measured. The second phase of the experiment mirrors the same procedure as phase 1 with the experimental group receiving additional VR training before the actual test was conducted. The task itself was altered to include the location of the gear needed for a manual fire attack and the subsequent location and extinguishing of the fire. For both phases, the participants training in VR outperformed the control groups with faster completion times and less navigation errors. The researchers conclude that the VR training provides a viable training tool for practicing procedures and tactics without safety risks.

5.2.2 Commander Training

Rather than assessing the execution of physical skills in VR, Cohen-Hatton and Honey (2015) evaluated the training of cognitive skills of commanders in a series of experiments. In their three-part study, the aim was to evaluate whether goal-oriented training, i.e., the evaluation of goals, the anticipation of consequences, and the analysis of potential risks and benefits for a planned action, would lead to better explicit formulation of plans and the development of anticipatory situational awareness. This was compared to groups given standard training procedures for the same scenarios. The researchers used three different scenarios as follows: a house fire, a traffic accident, and a fire threatening to spread across different buildings in an urban area. Participants encountered all three scenarios: first in a VR environment (experiment 1) and then on the fireground (experiment 2). Lastly, the house fire was recreated in a live-burn setting for the third experiment. Participants were compared based on whether they had received standard training or goal-oriented training procedures. The scenarios presented the participants with situations that demanded decisions to be taken dynamically based on new information that would be presented during the trial (e.g., an update of the location of a missing person, the arrival of a new fire crew, or sudden equipment failure). Their behavior was coded to obtain the frequency and chronology of occurrence of information gathering (situation assessment (SA)), plan development (plan formulation (PF)), executing a plan by communicating actions (plan execution (PE)), and anticipatory situational awareness. The researchers concluded that the VR environment accurately mirrors the commander activities as executed in real-life scenarios, because the chronology of SA, PF, and PE follows the same pattern for the group that received standard training. The patterns obtained during experiment two and three further support the notion of VR as a viable analog to real-life training. The behavior for the participants receiving goal-oriented training was further consistent across all degrees of realism, which supports the viability of VR for commander training.

The viability of training commanders utilizing immersive VR technology was also demonstrated by Clifford et al. in two studies (Clifford et al., 2018aClifford et al., 2018b). These studies complement the work carried out by Cohen-Hatton and Honey (2015), since the work environment and the nature of the measures were different while the overall question of the viability of a virtual environment for firefighter training remained the same. The first study (Clifford et al., 2018b) was investigating the effect of different types of immersion, by varying the display technology used, and their impact on the situational awareness of aerial attack supervisors (AASs). AAS units deployed in wildfire scenarios are tasked with coordinating attack aircraft that aim to extinguish and control the fire. These commanders are flying above the incident scene in a helicopter and need to assess the situation on the ground to coordinate fire attacks. The researchers put commanders in a simulated environment, showing a local wildfire scenario, using either a high-definition TV, an HMD (Oculus Rift CV1), or a CAVE setup (270° cylindrical projection). While there were no differences in the abilities to accurately ascertain where the location of the fire is between display types, the location of secondary targets, such as people and buildings, was easier to determine with the HMD and CAVE setup which was attributed to the wider field of view (FOV) of these two display devices. The comprehension of the situation and the prediction of future outcomes, as part of the situational awareness scales, were also significantly better with the immersive VR options. The researchers found no significant differences between the two immersive display types for any of the subscales of the situational awareness measure. The researchers conclude that the immersive displays offer better spatial awareness for training firefighters in VR and are overall preferred by trainees compared to the non-immersive training.

The second study by Clifford et al. (2018a) investigated the elicitation of stress by manipulating interference in communication between the air attack supervisor and the pilots of the air attack aircraft. The AASs were put into a simulator that visualized a local wildfire using a CAVE setup (Figure 2). The AAS could communicate with the pilot of the helicopter sitting in and using the internal communication device hand signals, while using a foot pedal to activate outgoing radio communication with attack pilots and operations management. Communication disruptions were varied, first only using vibration of the seat (simulated in the CAVE) and the sound of the helicopter, then introducing background radio chatter from other pilots, and lastly, interrupting the radio transmissions to simulate a signal failure. Heart-rate variability the and breathing rate were used as physiological measures of stress as well as self-report questionnaires for stress and presence were applied. The researchers conclude that the system was successful in simulating the exercise as all participants completed the task successfully. The trainees felt present in the virtual space, although the realism and involvement measured did not significantly differ from the observable midpoint. While the signal failure did not show a significant increase in physiological stress compared to the radio chatter condition, overall the physiological stress measures showed an increase in stress responses. It has to be noted that the researchers do associate the increase in breathing rate to the overall increase in communication between conditions and therefor discount this as a viable stress measure. Qualitative data, together with the self-report data, suggest that the communication disruption successfully induced stress in participants. The participants additionally reported enjoyment in using the system.FIGURE 2

FIGURE 2. CAVE system simulating helicopter cockpit for Air Attack Supervisor Training Clifford et al. (2018b).

6 Discussion

The studies reviewed for this article, despite limited numbers, do offer valuable insights into the viability of VR as a tool for firefighter training. Immersive VR technology provides an ecologically valid environment that mimics that of real-life exercises adequately. As shown by Clifford et al. (2018b), the use of monitor-bound simulations has limitations that negatively impact situational awareness. Being able to train spatial and situational awareness with a FOV that more closely resembles that of normal human vision, using an HMD or CAVE setup enables the creation of training environments in which trainees feel present. The studies conducted by Cohen-Hatton and Honey (2015) provide even stronger evidence for this, by showing that the behavior of their participants was consistent across levels of fidelity:

“In Experiments 1–3, the same scenarios were used across a range of simulated environments, with differing degrees of realism (VR, fireground, and live burns). The patterns of decision making were remarkably similar across the three environments, and participants who received standard training behaved in a manner that was very similar to that observed at live incidents […].”

While only applicable to two of the studies, the training of physical skills could successfully be done in the studies using natural input methods, by either tracking body posture or using firefighting gear as input devices. Trainees, when being provided with feedback in the virtual reality environment, do learn from their mistakes and improve the execution of physical skills in successive trials. This underscores the value of experiential learning enabled by VR. Natural input methods are becoming more and more prevalent for VR applications, due to the improvements in tracking. Two of the studies reviewed were conducted in the late 90s (Bliss et al., 1997Tate et al., 1997), which resulted in constraints for the possibilities of more natural input. With both studies having been conducted more than 20 years ago as of the writing of this article, the outlook for future work by Bliss et al. (1997) was already anticipating the reappraisal of VR capabilities for training:

“The benefits of VR need to be assessed as the type of firefighting situation changes and as the capabilities and most of VR changes.”

On the other hand, the study conducted by Cohen-Hatton and Honey (2015) was concerned with commander training and therefore relied more heavily on decision-making tasks rather than physical skills, which are more easily simulated since the execution is mainly verbal.

Many of the studies observed, both old and new, make an effort to provide an ecologically valid environment for their simulation that is as analogous to the real-life activity as possible; even if, as previously stated, they are limited by technology. For example, both Backlund et al. (2007) and Cohen-Hatton and Honey (2015) required their participants to wear firefighting uniforms during their tasks (Figure 3). Bliss et al. (1997) did not require the participants to equip any firefighting gear but did give them industrial goggles sprayed with white paint to inhibit their vision in a similar manner how smoke would in a real scenario. Likewise, Backlund et al. (2007) use a real fire hose (Figure 4) to give a more apt input method than the joysticks and VR controllers used in the other studies observed.FIGURE 3

FIGURE 3. Example of a firefighter interacting with the Sidh system by Backlund et al. (2007) (used with permission).FIGURE 4

FIGURE 4. Breathing apparatus worn and input device used in Sidh by Backlund et al. (2007) (used with permission).

However, there still remains much room for future research into furthering the ecological validity of the virtual environment within the context of firefighting training. Of all the studies observed, very few attempt to involve the senses outside of the auditory and visual systems. The inclusion of extra senses into the virtual environment—for example, haptic feedback (Hoffman, 1998Insko et al., 2001Hinckley et al., 1994) or smell (Tortell et al., 2007)—has been shown to improve learning outcomes, aid user adaptation to the virtual environment, and increase presence. Many studies already exist that can be incorporated into firefighting training to provide a richer and more realistic environment for the trainee. For example, Shaw et al. (2019) presented a system to replicate the sensation of heat (via heat panels) and the smell of smoke into their virtual environment [for smell, see also the FiVe FiRe system (Zybura and Eskeland, 1999)]; although in the context of a fire evacuation instead of firefighting training, the authors note that their participants demonstrated a more realistic reaction when presented with a fire. Likewise, Yu et al. (2016) present a purpose-built firefighting uniform that changes its interior temperature in reaction to the simulation. For haptic feedback, there is promising research into haptic fire extinguisher technology that could also be incorporated (Seo et al., 2019). Looking to commercial systems, the evaluation of other input methods could be promising for increasing ecological validity and improving the possible transfer of physical skills; see, for example, Flaim Trainer3 or Ludus VR4. While current studies (Jeon et al., 2019) are already showing promise in improving the ecological validity of firefighting training by partially incorporating these suggestions, additional research and study would be very beneficial to the field.

Regarding the training of mental skills, the review obtained ample evidence for the viability of skill transfer from VR to real deployment. Especially navigation tasks, requiring trainees to apply spatial thinking, were successfully trained in three of the reviewed articles. Training with VR was on par with memorizing the building layout utilizing blueprints and improves performance in subsequent real navigation tasks. As was highlighted by participants in the study by Tate et al. (1997), the VR training enabled spatial planning and subsequently improved performance:

“Most members of the VE training group used the VE to actively investigate the fire scene. They located landmarks, obstructions, and possible ingress and egress routes, and planned their firefighting strategies. Doing so enabled them to use their firefighting skills more effectively.”

Another important finding is the heightened engagement of trainees during VR training. The majority of studies reviewed found evidence for trainees preferring, enjoying, and being engaged with the training. The study by Backlund et al. (2007) went one step further by utilizing score and feedback systems to enhance engagement, which they deem to be important for voluntary off-hour usage of such a system. VR, as opposed to traditional training, provides the possibility of logging performance, analyzing behaviors, and providing real-time feedback to trainees without the involvement of trainers. Just as important as the frequency of training for the upkeep of skills, which is made possible with the relative ease of administration and the heightened engagement during VR training, the mental preparation of firefighters also plays an important role in counteracting possible adverse mental effects brought upon by threatening conditions during deployment. Physiological measures used by Clifford et al. (2018a) show that stress can be elicited successfully in a VR training scenario. Multi-sensory stimulation seems to further add to the realism and stress experienced as was stated in their study:

“With distorted communications and background radio chatter, you’re fearing to miss teammates communications and wing craft interventions. But the engine sound and the vibrations make the simulation much more immersive.”

Unlike many other studies in this inquiry, Bliss et al. (1997) concluded that the results from the group that used VR training were not significantly better than their peers who used the traditional training solution—in this case, the use of blueprints. While the VR group performed on par with those who used blueprints, the results are underwhelming in comparison to other studies observed in this inquiry. In line with Engelbrecht et al. (2019), who deemed technology acceptance a weakness of VR technology in their analysis, the authors point to their participants’ low acceptance of technology and their familiarity with using the traditional training method as an explanation. However, while it is true that the study was conducted in times with arguably less prevalent acceptance of technology in general, this factor of familiarity, acceptance, and embrace of technology as a viable training tool should be considered in future work.

In addition to technological acceptance of potentially impacting learning outcomes, it is important to note the limitations of the technology used in all articles observed, especially earlier examples, and what effect this could have had on their results. Resolution of screens, their refresh rate, and the FOV of the headset have all improved significantly since the late 90s when two of the studies of this inquiry took place (see Table 2). Likewise, as Table 2 shows, earlier modern examples of HMDs, such as the Oculus Rift DK1, are considerably more under-powered than their more modern iterations.TABLE 2

TABLE 2. A comparison of VR headsets.

As can be seen, the FOV of the headsets used in the older studies was significantly more constrained than any used in more recent research. The I-Glasses by virtual I/O used by Bliss et al. (1997) had only a 30-degree field of view in each eye while VR4 by virtual research systems used by Tate et al. (1997), a similarly aged study, had a FOV of 60°. For comparison, the more modern Oculus DK1 and CV1, used by Clifford et al. (2018a) and Cohen-Hatton and Honey (2015), have a FOV of 110°. This is potentially significant as, in the context of “visually scanning an urban environment for threats”, Ragan et al. (2015) found the participants performed substantially better with a higher FOV. Toet et al. (2007) found that limiting the FOV significantly hindered the ability of their participants to traverse a real-life obstacle course—a setting closer, albeit not virtual, to the task set by Bliss et al. (1997) and Tate et al. (1997). This limitation could potentially give further explanation as to why the VR training group did not outperform the blueprint group in the study of Bliss et al. (1997). However, in the study of Tate et al. (1997), the VR group outperformed traditional methods despite sharing the same limitation of FOV; although it is possible, as Ragan et al. (2015) suggest, that the limited FOV had other negative consequences, such as causing the user to adopt an unnatural method of moving their head to observe the virtual environment.

In addition, the lower refresh rates of some HMDs could be cause for consideration. Low refresh rates of headsets have been directly correlated to the sensation of cybersickness in VR LaViola (2000) which in turn has been shown in previous studies to significantly negatively affect the performance of participants in VR simulators (Kolasinski, 1995). For comparison, the Oculus CV1, as used by Clifford et al. (2018b), has a refresh rate of 90hz whereas the I-Glasses, VR4, and Oculus Rift DK1 as used by Bliss et al. (1997)Tate et al. (1997), and Cohen-Hatton and Honey (2015) can only produce a maximum of 60hz (Ave and Clara, 1994Herveille, 2001) with Tate et al. (1997) specifying that their simulation ran at approximately 30 frames per second. As a baseline, LaViola (2000) note that:

“A refresh rate of 30 Hz is usually good enough to remove perceived flicker from the fovea. However, for the periphery, refresh rates must be higher.”

Therefore, all HMDs used in this inquiry, despite their age, should be within these limits. Bliss et al. (1997), with the lowest refresh rate of all studies observed, support this by stating that, unlike previous research, there was no sign of performance decrements in their study due to cybersickness with only two of their participants reported having experienced it. Likewise, Tate et al. (1997) used 1 minute rest breaks to avoid any simulation sickness which therefore mitigates any potential impact this would have had on their results. In addition, Cohen-Hatton and Honey (2015) report that only two of 46 of their participants experienced cybersickness despite the comparatively low refresh rate of the Oculus Rift DK1. However, it is important to note that various studies have shown that cybersickness affects female users more acutely than males (LaViola, 2000Munafo et al., 2017), and in each of the aforementioned studies, the majority of the participants were male (Bliss et al. (1997) and Cohen-Hatton and Honey (2015): all participants were male, Tate et al. (1997): 8/12 participants were male. Therefore, any effect of negative impact on performance that could have been caused by the lower refresh rates of the HMD may have been avoided—or, at least, mitigated—due to the gender distribution heavily leaning towards males in the firefighting profession (Hulett et al., 20072008) which was reflected in the participant selection of the studies observed. Regardless, we can note that the refresh rates of all HMDs observed would not seem to detract from their findings, although future studies should attempt to use HMDs with a high refresh rate to avoid any such complications.

Both Tate et al. (1997) and Bliss et al. (1997) used a Silicon Graphics Onyx computer using the Reality Engine II to create the virtual environments. Likewise, Backlund et al. (2007) used the half-life 2 engine (released in 20045). While both engines were powerful for the time, computer hardware has increased exponentially since either of their releases (Danowitz et al., 2012). As such, these simulations have a much lower level of detail, both of the environment and the virtual avatar, than the more modern examples examined which use modern engines (such as Unity3D). This could potentially have an effect on the results from these studies and is important to investigate.

Regarding model fidelity’s effect on presence, Lugrin et al. (2015) found that no significant differences could be found between realistic and non-realistic environments or virtual avatars. Ragan et al. (2015), in the context of a visual scanning task, noted that visual complexity—which could include model/texture detail, fog, or number of objects in the environment—had a direct effect on task performance. Principally, they noted that the participants performed better in environments with fewer virtual objects. Due to this, Ragan et al. (2015) recommended that designers should attempt to match the visual complexity of the virtual environment to that of its real-life counterpart. However, the authors concede that different factors of visual complexity could affect the task performance in varying levels of severity and that future work would be required to gauge the impact of each factor. Lukosch et al. (2019) stated that the low physical fidelity of environments does not significantly impact learning outcomes or the ability to create an effective learning tool. Therefore, while there could be certain factors that are impacted by lower graphical quality, we cannot find sufficient grounds to discount or significantly question the results of the aforementioned studies.

7 Conclusion

While this review can only draw limited conclusions with regard to the viability of VR technology for general firefighter training, the scrutiny applied to the sourcing of publications provides an important step forward. The findings from previous work highlight the potential of VR technology to be an ecologically valid analog to real-life training in the acquisition of physical and mental skills. It can be applied to the training of commanders as well as to support the training of navigation tasks for unknown indoor spaces. The limitations of the technology used in the summarized studied, such as not being able to create and display high-fidelity immersive environments and the lack of using natural input methods, can be overcome with the developments that have been made in the immersive VR space over the past years. This opens up new opportunities for researchers to investigate the effectiveness of VR training for the target population. VR research for firefighters is wide open and promising, as Engelbrecht et al. (2019) stated in their SWOT analysis of the field: “Without adequate user studies, using natural input methods and VR simulations highly adapted to the field, there is little knowledge in the field concerning the actual effectiveness of VR training.”

While there is room to transfer findings from other domains to inform designs, evidence for the effectiveness of training itself should be approached with caution when drawing conclusions for the entirety of the emergency response domain. The work presented in this article can serve as a helpful baseline to inform subsequent research in this domain and might also be useful to inform the design of systems in adjacent domains; however, evidence of the effectiveness of training itself should not be generalized to other emergency response domains.

The full report, references and links can be found at https://www.frontiersin.org/articles/10.3389/frvir.2021.671664/full

Our Guide To Filming, Recording & Enabling 360 Media, Communications & Editing

360-degree video is a mostly new form of filmmaking. With specialist equipment now accessible to consumers, it's easier to make great 360 videos than ever before. If you want to become an immersive film director yourself, nothing is holding you back...except any uncertainty as to how to get it right.

In many respects, 360-degree film production is a lot like making a traditional film – you have a location, a script, actors and props...but you shoot from multiple angles at the same time. There are also many quirks owing to the immersive nature of the medium that mean you have to make some extra considerations. With all this in mind, you may be at a loss as to where to turn. This is why we want to share our top 10 dos and don’ts when it comes to making your own mesmerising panoramic motion pictures.

immersive portal ticker

DO: Decide on a good height and angle at which to film

As a general rule of thumb, the camera setup should be positioned no lower than eye level. Placing the camera too low might make your audience feel uncomfortable and intimidated by their virtual surroundings.

immersive portal cross

DON'T: Place objects too close to the stitching area

Before you start your cameras rolling, you have to have a good idea of where the stitching areas are going to be in your film. These are the points at which the footage from your camera lenses will be ‘stitched’ together during post-production to make one panoramic film.

If an object of focus straddles across the stitch line in the final film, you’ll know it. It could end up looking bodged together, wedge-shaped, or even invisible at one side. The closer the object is to the camera lens, the worse the effect looks, so be mindful of where you position everything!

immersive portal ticker

DO: Get the plate!

Plate shots are still photos or video recordings taken of the scene albeit with no action. They themselves might not make it into your final film, but trust us: having them to hand will save you a lot of hassle during post-production!

Once you’ve done all your filming, you’ll have to edit out the tripod in post-production; if your eagle-eyed viewer looks down where their feet would be to find three metallic legs straddling across the floor, it’ll break all immersion (or make them think they’re supposed to be a telescope!). If you’re filming on a flat surface of a single colour, you might just get away with airbrushing it out in post-production. You couldn’t get away with the same trick if the ground surface is visually more complex, however (e.g. if you were filming on a bridge or a patterned rug).

This is an example of a scenario where plate shots of the ground come to the rescue. These should be as close to the original setting as possible, snapped by a camera at approximately the same position and angle as your filming rig.

immersive portal cross

DON'T: Move too much

This is one of the biggest mistakes you could make when filming spatial reality content.

That’s not to say there can’t be any sense of movement at all in a 360-degree film; you just have to be mindful of how you convey it. Perhaps the best way to create a comfortable illusion of motion is to use what’s sometimes known as the cockpit effect to your advantage: this is where there is a stationary frame of reference from which the audience views everything else moving (e.g. imagine you’re in a car or a plane). Otherwise, make sure you move the camera at a gentle, natural pace.

Don’t rotate the camera either: remember that one of the biggest pulls of 360 video is that it grants viewers the ability to look around the world themselves!

immersive portal ticker

DO: Keep your camera stable

Unsteady camera footage is nauseating in any video, but just imagine watching a shaky 360-degree video; we bet it’d probably feel like the whole world is shaking around you, or that you’re on a horrible theme park ride you just can’t wait to get off!

You don’t want your audience to feel like that when they view your content. Fortunately, virtual worlds of wobbling can be easily prevented: just make sure your camera rig is set up on a tripod and/or a stable surface where it won’t rock or topple over.

immersive portal cross

DON’T: Hesitate to overshoot

It’s understandable if you don’t want to spend any more time filming than you have to, whether your SD card is filling up fast, or you just want to pack up and go home as soon as you can. But that doesn’t mean you shouldn’t.

It’s always better to come away with more material than you need, rather than realise you forgot to include a particular shot or there’s a glaring error in some of your vital footage.

immersive portal ticker

DO: Pay attention to the lighting and weather

Most cameras just don’t work as well in the dark as they do in well-lit environments, and unfortunately, 360-degree camera setups are no exception. That’s not to say every immersive film should take place in broad daylight, but if you want your audience to be able to discern their surroundings as they look around, it’s best to set up at least some sort of lighting in a dark setting.

If you are using a panoramic camera, keep in mind that their lenses are even more prone to rain droplets than fixed frame cameras as you can’t cover them up as easily.

immersive portal cross

DON’T: Switch between scenes too quickly

If you are going to transition to other environments within your film, make sure you do it gradually and give your audience a chance to peruse each one.

Because the primary purpose of 360-degree video is to let your audience explore the world around them, changing between settings quickly and frequently can break the immersion and cause confusion.

immersive portal ticker

DO: Be prepared for intensive post-production

The post-production stage of creating a VR film is just as important a process as the filming itself. Not only will you have to stitch all the footage together, but you’ll also have your work cut out colour-correcting, sound-editing and everything else!

It may take longer than it would for a typical film, but it’s all worth it if you want to deliver a  truly seamless experience to your audience!

immersive portal cross

DON’T: Expect post-production to fix everything

The “we’ll fix it in post” attitude is a harmful one when it comes to any type of film production. Post-production is a crucial stage that can either make or break your film, but you still have to be careful not to think of it as a crutch. Video editing software is constantly getting better at ironing out the kinks and little imperfections you notice in your recordings, but it still can’t make a cinematic masterpiece from inherently bad footage.

Instead, think of post-production as the stage where you can filter out the things you notice later on down the line. When you’re in the process of producing your own 360-degree film, try to film your raw footage in the right ballpark as much as possible: if you notice something wrong during filming, rectify it yourself if you can and capture another take. You’ll also save yourself a heck of a lot of hassle if you took all of these pointers beforehand!


Holospatial delivers an alternate method of delivery for any VR or non-VR application through advanced interactive (up-to 360-degree) projection displays. Our innovative Portal range include immersive development environments ready to integrate with any organisational, experiential or experimental requirement. The Holospatial platform is the first ready-to-go projection platform of it's type and is designed specifically to enable mass adoption for users to access, benefit and evolve from immersive projection technologies & shared immersive rooms.


Volumetric Displays: Will Voxels Immersive Display Technology Ever Become The New pixels?

There’s a whole new type of virtual display that we don’t really see discussed that much.

A volumetric display has three physical dimensions, so you can view projected images from multiple angles as if they were real-life objects. It may sound like something from a science fiction film, but it excites us to think about how we can use them to enrich an immersive experience - from displaying data to mesmerising models.

Wait, aren’t they just holograms?

Not quite. Volumetric displays may look and sound similar to holograms, but they’re actually much more advanced. Holograms may look 3D at times, but technically they aren’t: the image is actually projected onto a 2D surface.

There’s nothing 2D about a volumetric display, however. When it displays an object, it processes information about the area said object takes up in all three spaces, and then scatters light across a wider space accordingly.

In other words, if a holographic display paints pictures, a volumetric display crafts sculptures.

Display Types

Swept volume displays

Swept volume displays automatically divide a 3D image into multiple slices. The display then projects the slices onto a rapidly-rotating screen in succession, one at a time, giving us the illusion of a static three-dimensional object.  

This system works by taking advantage of our persistence of vision.  Even when an image physically leaves our sight, it lingers for a flash in our mind's eye. Our eyes can only process around 10-12 frames per second individually, while films typically have 24 or 25 frames.  You could say a swept volume display works just like a flipbook animation - what our eyes and mind interpret as a single moving scene is actually a sequence of multiple drawings (a.k.a. frames) played at a rapid rate.

See the video below for an example of an advanced swept volume display in action, developed by Voxon Photonics.

Static volume displays

Unlike swept volume displays, static volume displays don't have any moving surfaces. Just like how traditional displays are made up of pixels, static volume displays are composed of multiple tiny elements of light known as voxels. The voxels can have one of two states: they are either off (in which they’re unlit and transparent) or on (lit up and opaque/translucent).

The image below illustrates a hypothetical static volume display from 2003, developed by a group of students and teachers from a high school in Germany.

  Sounds cool, but why don't we hear of these more?

As exciting as volumetric displays sound, especially in terms of the possibilities they could unlock, the future of this technology is up in the air. It simply isn't feasible for mainstream use at the moment, for multiple reasons:

  1. They’re expensive

Voxels and slices of 3D images require much more computing power to store and process than regular pixels and visual information. Swept volume displays may require very high refresh rates to maintain persistence of vision - combine that with the complexity of the voxels and the display could end up requiring a bandwidth of several gigabytes per second to run!

This obviously means that wide-scale usage of proper volumetric displays would be extremely costly: to produce, purchase, and to use!!

  2. They’re inefficient

The sheer amount of power required to make and use them wouldn’t just burn a hole in your wallet, but perhaps also the ozone layer!  With climate change concerns growing, our society becomes more and more conscious of our carbon footprint by the day. The environmental resources required to generate electricity to power them would probably be more than they’re worth, so we’d have to work hard to develop a greener, less power-hungry solution.

  3. The graphics quality is currently limited

Even in a scenario where financial and environmental concerns  weren’t an issue, volumetric displays still pose challenges of their own within the visual aspect of them.

With a lot of research yet to be done and obstacles to be cleared, volumetric displays have a long way to go and we probably won’t see them emerge on the market anytime soon.

But who knows? Maybe in a few years’ time, we could all be using monitors that display our images in true 3D.

  Bibliography

Knut Langhans, Christian Guill, Elisabeth Rieper, Klaas Oltmann, & Detlef Bahr (2003). SOLID FELIX: a static volume 3D-laser display. In: [Online]. 30 May 2003. Available from: https://doi.org/10.1117/12.474155Smalley, D., Poon, T.-C., Gao, H., Kvavle, J. & Qaderi, K. (2018). Volumetric Displays: Turning 3-D Inside-Out. Optics and Photonics News. 29 (6). p.p. 26.

Delicious Data: Will We Ever Taste Our Computer Applications?

Just imagine you are in a restaurant. A virtual restaurant. You feel as if you are really sitting at a table on a chair. You hear the chatter and plate clattering of your nearby diners. You see the posh lighting, finely decorated tables and velvety restaurant wallpaper. You pick up virtual menus and look through what you fancy eating. Maybe you can even smell the ambience of all that delicious food being prepared from in the chefs’ kitchen. But then, your meal arrives. It looks convincing and appetising, and you pick up your digital knife and fork to cut it up and try it...but it’s bland. There’s no mouth feel, no tingling taste buds...no eating experience.

The sense of taste is one which has often been overlooked in the world of technology. As graphic displays, audio speakers and even haptic devices have all advanced over the years, the idea of gustatory devices has been left in the dust. Not because people aren’t interested in making digital taste happen, but rather that a lot more research needs to be done to understand how we can make it a reality. The sense of taste is not quite as straightforward a sense as you may expect...

How does taste work?

Of course, one of the secrets to how we can appreciate the different flavours of what we eat is in the tens of thousands of taste buds in our tongue. Within each taste bud is a cluster of taste receptor cells.  There are actually only five taste sensations the tongue can experience: sweetness, saltiness, sourness, bitterness and umami/savouriness, which are triggered by the detection of sugar, sodium chloride, acids, alkaloids and glutamates respectively.

But how is it that can we taste the difference between lemon juice and vinegar if both are sour? That’s where your nose comes into play, as the sense of smell and taste are actually very closely linked. As you eat or drink, the chemicals in what you consume trigger the olfactory receptors in your nasal cavity as well as your taste buds, and it is the combination of these responses that signal flavours to your brain. Next time you eat something tasty, hold your nose while you put it in your mouth and chew...and it’s all but guaranteed you won't have the gustatory explosion you would have felt in your mouth otherwise.

What will taste tech look like?

The world of taste-inducing technology is still in the conception stage, let alone its infancy. Due to the complex, chemical nature of our sense of taste, thinking of ways to simulate human taste sensations artificially and safely have proven difficult. No product or standard for taste-simulating technology currently exists in the market, and it’s currently impossible to break down smells into categories or elements. However, a few researchers have come up with some potential ideas as to how taste can work...

Tokyo University’s Takuji Narumi drew attention to the close link between the senses of smell and taste, and explored how exposure to different visual and olfactory stimuli could affect how we taste. At a computing conference in Canada in 2011, he demonstrated the Meta Cookie system, a head-mounted visual and olfactory display which was worn while eating an unflavoured biscuit. The headset’s display laid an image of a different-coloured biscuit over the original using augmented reality, as a perfume scent travelled through tubes attached to the nose.

In 2012, another team of researchers at the National University of Singapore, led by Nimesha Ranasinghe, explored how technology could tantalise the taste buds directly. They developed an experimental tongue-mounted device which aimed to replicate rudimentary taste sensations via controlled electrical stimulation. The results of this experiment found that the device was most effective at replicating sour taste sensations, and was capable of doing so in three degrees of intensity.

Six years later, another team of researchers proposed a similar taste-actuating interface of their own, this time activating the taste buds by changing temperature. This method could stimulate sweetness much better than any other taste, indicating that different taste sensations may require different strategies.

Where will we see taste tech used?

Imagine a culinary arts training simulation where you can actually learn what flavours to look out for when sampling your cooking. Or a virtual marketing campaign where you can sample beverages without physically having to drink them (want to taste a wine but need to drive back home?). Or maybe a virtual travel experience where you can actually get a taster for another country’s cuisine, all within one of our Portals perhaps!

Integrating the sense of taste into technology is still ages away from being a possibility, but it’s nonetheless exciting to think about the doors it’ll open up for what kind of immersive applications we can create.

Citations

Karunanayaka, K., Johari, N., Hariri, S., Camelia, H., Bielawski, K.S. & Cheok, A.D. (2018). New Thermal Taste Actuation Technology for Future Multisensory Virtual Reality and Internet. IEEE Transactions on Visualization and Computer Graphics. 24 (4). pp. 1496–1505.

Narumi, T., Nishizaka, S., Kajinami, T., Tanikawa, T. & Hirose, M. (2011). Augmented reality flavors: Gustatory display based on Edible Marker and cross-modal interaction. Conference on Human Factors in Computing Systems - Proceedings.

Ranasinghe, N., Nakatsu, R., Nii, H. & Gopalakrishnakone, P. (2012). Tongue Mounted Interface for Digitally Actuating the Sense of Taste. In: 2012 16th International Symposium on Wearable Computers. [Online]. June 2012, Newcastle, United Kingdom: IEEE, pp. 80–87. Available from: http://ieeexplore.ieee.org/document/6246147/. [Accessed: 1 November 2020].

Accessibility Guidelines for VR Games & Immersive Projection - A Comparison and Synthesis of a Comprehensive Set

Below is a featured report enabling deep understanding of how accessibility can be achieved in gamified content, but the report also considers wider factors for various user levels. Accessibility and inclusion is a critical part of what we do at Portalco as our environments are all designed physically and in their interfaces to enable all users to interact with immersive technology.

This requires multiple aspects to be considered and if you are looking to create or develop content, or start a project for your people then reports like this are a great place to start, to understand pre-existing features that can make a huge difference to your deliverables.

Increasing numbers of gamers worldwide have led to more attention being paid to accessibility in games. Virtual Reality offers new possibilities to enable people to play games but also comes with new challenges for accessibility. Guidelines provide help for developers to avoid barriers and include persons with disabilities in their games. As of today, there are only a few extensive collections of accessibility rules on video games. Especially new technologies like virtual reality are sparsely represented in current guidelines. In this work, we provide an overview of existing guidelines for games and VR applications. We examine the most relevant resources, and form a union set. From this, we derive a comprehensive set of guidelines. This set summarizes the rules that are relevant for accessible VR games. We discuss the state of guidelines and their implication on the development of educational games, and provide suggestions on how to improve the situation.

1 Introduction

In 2020 the number of people who play video games was estimated to 3.1 billion worldwide, which is 40% of the world population (Bankhurst 2020). This shows that video games are not a niche hobby anymore. The game industry has picked up new technologies like Virtual Reality (VR). Thus, VR is thriving recently, with more and more (standalone) headsets being developed for the consumer market. The current state of the art in VR headsets is dominated by Sony and Facebook’s Oculus, and the market is expected to grow rapidly in the following years (T4, 2021).

1.1 Games and Disability

The rising numbers of gamers worldwide and the technological advances come with new challenges for accessibility. According to an estimate of the World Health Organization (WHO) from 2010, around 15% of the world population has some form of disability (World Health Organization, 2011). This means over a billion people live with physical, mental, or sensory impairments. It is not surprising that an increasing number of these people play or want to play video games but are excluded from it because of barriers they cannot overcome (Yuan et al., 2011). Furthermore, not only people with impairments can profit from accessible games. Situational disabilities like a damaged speaker, loud environment or a broken arm can affect any gamer (Sears et al., 2003Grammenos et al., 2009Ellis et al., 2020).

VR comes with new chances to include people with disabilities and make games more accessible. However, it also adds to the accessibility problems that can occur in games. As it is a relatively new technology, new rules and interaction forms still need to be developed.

1.2 Scope and Methodology of the Review

The matter we illuminate in this work is the importance and the need for accessible games in general and VR games in particular. Like others we come to the conclusion that what is needed is more awareness and a well-formulated set of rules developers can follow. By showing how relevant it is to make accessible games, we want to draw attention to and emphasize what the problem with the current state of accessibility guidelines is. The few accessibility guidelines for games that exist, do not or little deal with special requirements for VR.

Besides the general importance of accessibility due to increasing demand, in most countries educational games including VR games are legally required to be accessible for persons with disabilities. To achieve this designers and developers need a guide they can understand and follow. However the existing guidelines make it hard for game developers to apply and follow them when developing a VR game. This work shows what already exists in this field and explores whether it is sufficient.

We evaluate all noteworthy guidelines for games and VR applications. The result shows how small the number of applicable guidelines is. We then combine the found guidelines to a union set. The challenge is, that the different sources often contain the same rules but in different formulations and levels of detail. We also evaluate which of the rules are relevant for VR games in particular and therefore reduce the need for developers to filter relevant guidelines themselves. The union set reveals what rules are missing in the evaluated relevant works and where there is room for improvement. The comparison can help developers to read about guidelines from different sources and give a broader understanding of how to increase accessibility in their games.

2 Related Works

In this section, we look at 1) the state of accessibility in games in general, 2) the state of accessibility of VR games, and 3) the role of guidelines for making games accessible.

2.1 Accessibility in Games

The accessibility of games is a more complex problem than software or web accessibility in general because they often require a lot of different skills to play (Grammenos et al., 2009). Accessibility problems in video games can affect different parts of a game. The reasons are typically divided into motor disability, sensory disability and cognitive disability (Aguado-Delgado et al., 2018).

Video games are not only a pastime for disabled players, although this is an essential part of being able to play. The benefits of making accessible games are presented by Bierre et al. (2004)Harada et al. (2011)Beeston et al. (2018)Cairns et al. (2019a), and Cairns et al. (2019b), and These sources can be summarized into the following list:

• Entertainment and satisfaction: Games are made to be a source of entertainment and distraction.

• Connection and socializing: Playing games gives the chance to participate and feel included.

• Enabling: Playing games can enable impaired people to do things they otherwise cannot do.

• Range: For companies it is important to realize that many people benefit from accessible games.

• Legal issues: Legal requirements for accessibility are becoming more, including games.

• Universal: Everyone can benefit from accessible games.

Developing accessible games has long been seen as a subordinate topic mostly looked at in special interest groups or academics. The majority of accessible games are not mainstream games and/or never leave the state of research. Often accessible games are developed specifically for one particular disability. Making special accessible games can lead to multiple point-solutions that can only be played by a small group of people. (Cairns et al., 2019a)

Additionally, many studies concentrate on easy-to-use games with simple gameplay. Most games rely on hand usage to control them and visuals and sound as output. Problems mainly arise when people are not able to receive the output feedback (visual, auditory, haptic) or use the required devices to give input (Yuan et al., 2011Hamilton 2018). People with severe visual impairment can not use assistive features and accessible input devices often offer only limited possibilities for interaction. This is why games that can be played without visuals or with alternative input devices are often simple and require minimal input or interaction. (Yuan et al., 2011)

A reason for poor accessibility could be lacking information in schools for developers or the false assumption that making accessible games is not worth it because the number of people who benefit from it is too small. Complications in developing accessible games can be the individuality of impairments or the necessity to change the game fundamentally to make it accessible. It is difficult to find a good compromise between challenge and accessibility. (Yuan et al., 2011)

These difficulties lead to the most general problem with game accessibility: There are not many accessible games on the market. Examples of accessible games for each type of impairment are surveyed by Yuan et al. (2011), which also demonstrates the mentioned problem of point-solutions. A noteworthy mainstream game that is said to be the most accessible game to date is The Last of Us: Part 2. It has over 60 accessibility features that tend to visual, hearing and mobility impairments (PlayStation, 2020).

Many websites, organizations, and communities support accessible gamers and raise awareness. Well-known contact points are the International Game Developers Association (IGDA) Game Accessibility Special Interest Group (GASIG) (IGDA GASIG, 2020), the Able Gamers Charity (AbleGamers, 2018b) and the charity SpecialEffects (SpecialEffect, 2021). An organization that specialized in Extended Reality (XR), including VR, is the XR Access Initiative (XR Access, 2020).

2.2 Accessibility in VR and VR Games

The accessibility problems that occur in regular games overlap with the accessibility problems in VR games. VR applications and VR games come with both: ways to bypass the accessibility problems in games and new challenges and barriers that add to them. In VR, there is still little experience on a best practice compared to other domains. There is no conclusion on what approaches are good or not so far (Hamilton, 2018). This also influences already lacking methods for game accessibility (Mott et al., 2019).

Interaction in VR relies mainly on the head, hands and arms, which can be a huge barrier for people with motor impairment. Hamilton (2018), a better-known activist in the accessible games scene, did a thorough research of accessibility for all kinds of impairments in VR. Besides simulation sickness, Photosensitive Epilepsy and situational disabilities like not seeing one’s hands, he emphasized the problems with VR controllers. He summarizes issues that occur for people with motor impairment in VR games such as the presence, strength or range of limbs or the user’s height. VR controllers have developed into using one controller in each hand. They often have an emphasis on motion controls, like mid-air motions, requiring more physical ability than normal controllers or keyboards (W3C, 2020). In many games and applications there are no alternative input methods to using the controller (Mott et al., 2019). Additionally, at the moment, each manufacturer uses their own controllers, each model being different in terms of accessibility (W3C, 2020). Besides hand usage, most VR headsets are heavy and bulky, which requires strength of the neck and shoulders to use. Many games dictate the position in which the player must be. They require upper-body movements or even have to be played standing.

A more obvious barrier is the visual aspect. Apart from full blindness, barriers in VR can also occur for people with low vision, unequal vision in one eye or stereo-blindness (Hamilton, 2018). An issue that occurs only in VR is problems with wearing glasses under the HMD. Another problem is traditional zooming which can cause simulation sickness and disorientation in VR environments (Chang and Cohen, 2017). Similar problems occur for hearing impairments, such as stereo sound. Subtitles or captions are a special challenge in VR as they can not simply be put at the bottom of the screen. (Hamilton, 2018)

Despite the additional accessibility problems, VR can also help people with disabilities experience things they could not do otherwise, such as horseback riding or driving a car. Contrary to the exclusion people with disabilities might experience in games and the real world, Mott et al. (2019) see VR as a chance for all users to be equal in a game. VR offers new ways for input and output that are not possible with standard devices. Many of these can be realized with the sensors that are already included in current Head-Mounted Displays (HMD).

Most studies on accessible VR concentrate on removing barriers of VR headsets with special applications rather than introducing full games. Therefore there are not many specially made accessible VR games yet. Some games provide accessibility options, but often they only tend to one specific issue which is demonstrated by Bertiz (2019) presenting a list of some of these games. However, tools like SeeingVR (Zhao et al., 2019) and WalkinVR (2020) make VR applications more accessible in general.

2.3 The Role of Guidelines for Accessible Gaming

Software, in general, is becoming more accessible due to better awareness and legal requirements (Miesenberger et al., 2008). Guidelines are an important tool to support this. In Human-Computer-Interaction (HCI) they help designers and developers to realize their projects while also ensuring a consistent system. As for accessibility guidelines especially in the web environment they are well represented.

Different aspects of accessibility are considered in this work: games in general and VR games. The limited range of accessible games and VR games is attributed to a lack of awareness. Grammenos et al. (2009) brings this into relation with the problem of missing guidelines.

Although accessible games gained more awareness in the last few years, there is still a big gap between the regulations for web accessibility and games, which was researched by Westin et al. (2018). They compared the Web Content Accessibility Guidelines (WCAG) (Kirkpatrick et al., 2018) with the Game Accessibility Guidelines (GAG) (Ellis et al., 2020) in the context of web-based games and found that there are many differences. As a conclusion they state that game guidelines would have to be used in conjunction with WCAG for web-based games. Different references, for example Yuan et al. (2011) and Cairns et al. (2019a), draw attention to the lack of good literature, universal guidelines and awareness for accessibility in games. There is no official standard that can be used for games like the WCAG for web applications.

Zhao et al. (2019) found this to be especially true for VR games. It was also noticed by (Westin et al., 2018) who emphasize the importance to pay attention to XR in future guideline development. So far, guidelines for games rarely consider VR accessibility and few guidelines are exclusively made for VR applications. Many of them are specialized in one specific impairment or device. The way users interact with VR is hardly comparable with other software, so generalized guidelines can not be applied (Cairns et al., 2019a).

The success of using guidelines to make a game accessible depends on how good the guidelines are. Some guidelines are very extensive and aim to be a complete guide, while others are summaries or short lists of the most important rules. Many sets of rules try to be as broadly applicable as possible and match a wide variety of use-cases. However, in practice, this makes them harder to understand. It is not easy to make guidelines that are general enough, but at the same time developers can transfer them to their scenario (Cairns et al., 2019a). It can also be hard to decide what guidelines are relevant in a specific context and extract them from a big set. Yuan et al. (2011) see this as a problem when guidelines do not explain each rule’s purpose and when they should be applied.

3 Guidelines

In this section, we introduce existing guidelines that are noteworthy for this work. For each set of guidelines a summarized description is provided. They are the most relevant resources we were able to find in the English language. The guidelines were chosen by relevance in the areas of accessible games and accessible VR applications. Most of them contain explanatory texts for each rule, stating what they are for and providing good examples and tools. The relatively small number of found guidelines confirms the concerns of Yuan et al. (2011) and Cairns et al. (2019a).

The EN 301 549 standard is a collection of requirements from the European Telecommunications Standards Institute (2019). It was included in this comparison as it is the relevant work for legal requirements on accessibility. Its goal is to make Information and Communication Technology (ICT) accessible. This includes all kinds of software such as apps, websites and other electronic devices. As a European standard, these rules have to be followed by the public sector, including schools and universities (Essential Accessibility, 2019). Where applicable, the standard reflects the content of WCAG 2.1, which is why we do not look at WCAG separately in this work. The guidelines were updated several times since 2015. We use version V3.1.1 from 2019 for our comparison. Because the EN 301 549 is a very extensive document that considers any form of information technology, not all chapters are suitable for accessible games or VR. Therefore, the less relevant chapters were omitted, integrated into other guidelines or summarized into one more general rule.

3.1 Guidelines for Games

Many game guidelines build on each other or are mixtures of different sources. The most extensive game guidelines mentioned frequently in relevant literature are Includification and the GAG.

3.1.1 IGDA GASIG White Paper and Top Ten

In 2004 the IGDA GASIG published a white paper (Bierre et al., 2004) that lists 19 game accessibility rules found out from a survey. Later, they summarized these to a top ten list (IGDA GASIG, 2015) that is constantly updated. It boils down to the most important and easy to realize rules a developer should follow, providing quick help.

3.1.2 MediaLT

Based on the rules from the IGDA GASIG white paper MediaLT, a Norwegian company developed their own guidelines (Yuan et al., 2011). They presented 34 guidelines for “the development of entertaining software for people with multiple learning disabilities” (MediaLT, 2004).

3.1.3 Includification

Includification from the AbleGamers Charity (Barlet and Spohn, 2012) came up in 2012. It is a long paper including an accessibility checklist for PC and console games. Each rule is additionally explained in detail in plain text.

3.1.4 Accessible Player Experience

As a successor to Includification AbleGamers published a set of patterns on their website called the Accessible Player Experience (APX) in 2018 (AbleGamers, 2018a). They are, in fact, more patterns than guidelines, providing a game example for each accessibility problem.

3.1.5 Game Accessibility Guidelines

The Game Accessibility Guidelines (GAG) (Ellis et al., 2020) were also developed in 2012 and are the most known and extensive guidelines for games. They are based on WCAG 2.0 and the internal standards of the British Broadcasting Corporation (BBC) (Westin et al., 2018). The rules are constantly updated. For each guideline the GAG offer game examples that implemented the rule and list useful tools to do so.

We used the GAG as the basis for this work because they are the most extensive game guidelines of all considered. At the same time they also fit the game context best and provide easy-to-follow wording.

3.1.6 Xbox

Like many other companies, Microsoft has its own guidelines for products. For games on the Xbox console the Xbox Accessibility Guidelines (XAG) provide best practices (Microsoft, 2019). These guidelines are based on the GAG and also include some references to the APX.

3.2 Guidelines for VR

As before, we make no distinction between VR games and other VR applications. Only two sources that list measures for better accessibility in VR in the form of guidelines were found.

3.2.1 XR Accessibility User Requirements

The XR Accessibility User Requirements (XAUR) are a set of guidelines published by the Accessible Platform Architectures (APA) Working Group of the World Wide Web Consortium (W3C) in 2020. They contain definitions and challenges as well as a list of 18 user needs and requirements for accessible XR applications (including VR). The current version is a Working Draft as of September 16, 2020. (W3C, 2020).

3.2.2 Oculus VRCs: Accessibility Requirements

The Virtual Reality Checks (VRC) from Oculus developer portal are requirements a developer must or should follow to publish his/her app on Oculus devices. These VRCs have recently (in 2021) been extended by the section “Accessibility Requirements”, providing recommendations to make VR apps more accessible. (Oculus, 2020).

3.2.3 The University of Melbourne

On their website the University of Melbourne provides an overview of “Accessibility of Virtual Reality Environments” (Normand, 2019). The main content are the pros and cons of VR for people with different types of disabilities. For each type they provide a list which also includes use cases that can be seen as guidelines.

4 Synthesis of Guidelines

We used the previously introduced sources to derive a comprehensive set of guidelines that includes all rules that are relevant for accessible VR games. Inspired by the proposed procedure of the GAG we took the following steps to achieve this.

1) All guidelines mentioned above were evaluated and filtered by what is directly relevant for VR environments and games.

2) The remaining rules were compared to each other and the union set was formed. Similar guidelines were summarized and the formulations slightly changed or enhanced.

3) The result is a set of guidelines that combine and summarize all rules for accessible VR games found in the existing sources.

All found guidelines are shown as a list below. To avoid duplicate entries, this set is sorted by topic not by impairment or importance. This classification does not imply that some rules can not be relevant for other categories. The main source of the wording is given in parenthesis. Because the GAG was used as a basis, the most formulations were overtaken from them. This does not mean that those rules are not included in other guidelines. To provide good readability and the source of the text at the same time, the guidelines are color coded as follows:

• Black text in normal font type: Text written in black was taken as is from the original source which is written behind each rule in parenthesis. This does not mean that this rule does only appear in this particular set. It merely marks where the formulation was used from.

• Orange text in italic font type: Text written in orange marks where the original formulation from the source in parenthesis was changed or extended. This could be because the wording from another source was added or if the wording was adapted to be more clear.

The full comparison table is available as supplementary material on this paper.

Input and Controls

• Allow controls to be remapped/reconfigured; avoid pinching, twisting or tight grasp to be required (GAG)

• Provide very simple control schemes. Ensure compatibility with assistive technology devices, such as switch or eye tracking (GAG)

• Ensure that all areas of the user interface can be accessed using the same input method […] (GAG)

• Include an option to adjust the sensitivity of controls (GAG)

• Support more than one input device simultaneously, include special devices (GAG)

• Ensure that multiple simultaneous actions (eg. click/drag or swipe) and holding down buttons are not required […] (GAG)

• Ensure that all key actions can be carried out with a keyboard and/or by digital controls (pad/keys/presses) […] (GAG)

• Avoid repeated inputs (button-mashing/quick time events) (GAG)

• Include a cool-down period (post acceptance delay) of 0.5 s between inputs (GAG)

• Include toggle/slider for any haptics (e.g., controller rumble) (GAG)

• Provide a macro system. If shortcuts are used they can be turned off or remapped (GAG)

• Make interactive elements that require accuracy […] stationary or prevent using them (GAG)

• Make sure on-screen keyboard functions properly (Includification)

Audio and Speech

• Provide separate volume controls and stop/pause or mutes for effects, speech and background sound/music (independent from the overall system) (GAG)

• Ensure no essential information is conveyed by sounds alone (GAG)

• Use distinct sound/music design for all objects and events (GAG)

• Use surround sound (GAG)

• Provide a stereo/mono toggle and adjustment of balance of audio channels (GAG)

• Avoid or keep background noise to minimum during speech (GAG)

• Provide a pingable sonar-style audio map (GAG)

• Provide a voiced GPS (GAG)

• Simulate binaural recording (GAG)

• Provide an audio description track (GAG)

• Allow for alternative Sound Files (IGDA White Paper)

• Provide pre-recorded voiceovers and screenreader support for all text, including menus and installers (GAG)

• Masked characters or private data are not read aloud without the users allowance (EN 301 549)

• The purpose of each input field collecting information about the user is presented in an audio form (EN 301 549)

• […] Speech output shall be in the same human language as the displayed content […] (EN 301 549)

• Ensure that speech input is not required […] (GAG)

• Base speech recognition on individual words from a small vocabulary (eg. “yes” “no” “open”) instead of long phrases or multi-syllable words (GAG)

• Base speech recognition on hitting a volume threshold (eg. 50%) instead of words (GAG)

Look and Design

• Ensure interactive elements/virtual controls are large and well spaced […] (GAG)

• Use an easily readable default font size and/or allow the text to be adjusted. Use simple clear text formatting. (GAG)

• Ensure no essential information is conveyed by text (or visuals) alone, reinforce with symbols, speech/audio or tactile (GAG)

• Ensure no essential information is conveyed by a colour alone (GAG)

• Provide high contrast between text/UI and background (at least 4.5:1) (GAG)

• UI Components and Graphical Objects have a contrast ratio of at least 3:1 or provide an option to adjust contrast (GAG)

• Provide a choice of […] colour […] (GAG)

• Allow interfaces to be rearranged (GAG)

• Allow interfaces to be resized (GAG)

• Provide a choice of cursor/crosshair colours/designs and adjustable speed and size (GAG)

• Instructions provided for understanding and operating content do not rely solely on sensory characteristics of components such as shape, color, size, visual location, orientation, or sound (original from WCAG 1.3.3) (EN 301 549)

• No 3D Graphics Mode (IGDA White Paper)

• Indicate focus on (UI) elements (XAG)

• Enable people to edit their display settings such as brightness, include screen magnification (VRC)

• Provide an option to turn off/hide background movement or animation. Moving, blinking or auto-update can be turned off or paused (GAG)

• Headings, Labels and Links describe their topic or purpose in their text. If they are labeled, the label contains their text (EN 301 549)

Subtitles/Captions

• Provide subtitles for all important speech and supplementary speech. (Provide a spoken output of the available captions) (GAG)

• If any subtitles/captions are used, present them in a clear, easy to read way and/or allow their presentation to be customised (GAG)

• Ensure that subtitles/captions are cut down to and presented at an appropriate words-per-minute for the target age-group (GAG)

• Ensure subtitles/captions are or can be turned on with standard controls before any sound is played (GAG)

• Provide captions or visuals for significant background sounds. Ensure that all important supplementary information conveyed by audio is replicated in text/visuals (GAG)

• Provide a visual indication of who is currently speaking (GAG)

• Captions and Audio Description have to be synchron to the audio (EN 301 549)

Simplicity

• Use simple clear language. Employ a simple, clear narrative structure. (GAG)

• Include tutorials (GAG)

• Include a means of practicing without failure […] (GAG)

• Include contextual in-game help/guidance/tips (GAG)

• Include assist modes such as auto-aim and assisted steering (GAG)

• Indicate/allow reminder of current objectives during gameplay (GAG)

• Indicate/allow reminder of controls during gameplay (GAG)

• Offer a means to bypass gameplay elements […] and/or give direct access to individual activities/challenges and secret areas (GAG)

• Allow the game to be started without the need to navigate through multiple levels of menus (GAG)

• Offer a wide choice of difficulty levels. Allow them to be altered during gameplay, either through settings or adaptive difficulty (GAG)

• Include an option to adjust the game speed and/or change or extend time limits (GAG)

• Allow players to progress through text prompts at their own pace (GAG)

• Allow all narrative and instructions to be paused and replayed, care for automatic interruption. (GAG)

• Give a clear indication on important or interactive elements and words (GAG)

• Provide an option to turn off/hide all non interactive elements (GAG)

• Players can confirm or reverse choices they have made [] (APX)

VR

• Avoid (or provide an option to disable) VR simulation sickness triggers (GAG)

• Allow for varied body types in VR, all input must be within reach of all users (GAG)

• Do not rely on motion tracking and the rotation of the head or specific body types (GAG)

• If the game uses field of view, set an appropriate default or allow a means for it to be adjusted (GAG)

• Avoid placing essential temporary information outside the player’s eye-line (GAG)

• Ensure the user can reset and calibrate their focus, zoom and orientation/view in a device independent way (XAUR)

• Applications should support multiple locomotion styles (VRC)

• Provide an option to select a dominant hand (VRC)

Multiplayer

• Support voice chat as well as text chat for multiplayer games (GAG)

• Provide visual means of communicating in multiplayer (GAG)

• Allow a preference to be set for playing online multiplayer with players who will only play with/are willing to play without voice chat (GAG)

• Allow a preference to be set for playing online multiplayer with/without others who are using accessibility features that could give a competitive advantage (GAG)

• Use symbol-based chat (smileys etc) (GAG)

• Realtime text - speech transcription (GAG)

Others

• Allow gameplay to be fine-tuned by exposing as many variables as possible (GAG)

• Avoid flickering images and repetitive patterns to prevent seizures and physical reactions (GAG)

• Provide an option to disable blood and gore, strong emotional content or surprises (GAG)

• Avoid any sudden unexpected movement or events as well as a change of context (GAG)

• Provide signing (GAG)

• Include some people with impairments amongst play-testing participants and solicit feedback. Include every relevant category of impairment [], in representative numbers based on age/demographic of target audience (GAG)

• Provide accessible customer support (XAG)

• If a software can be navigated sequentially, the order is logical (EN 301 549)

• Provide details of accessibility features in-game and/or as accessible documentation, on packaging or website. Activating accessibility features has to be accessible (GAG)

• Ensure that all settings are saved/remembered (manual and autosave). Provide thumbnails and different profiles (GAG)

• Do not make precise timing essential to gameplay [] (GAG)

• Allow easy orientation to/movement along compass points (GAG)

• Where possible software shall use the settings (color, contrast, font) of the platform and native screen readers or voice assistance (XAUR)

• Ensure that critical messaging, or alerts have priority roles that can be understood and flagged to assistive technologies, without moving focus (XAUR)

• Allow the user to set a “safe place” - quick key, shortcut or macro and a time limit with a clear start and stop (XAUR)

• Locking or toggle control status can be determined without visual, sound or haptic only (EN 301 549)

• Using closed functionality shall not require to attach, connect or install assistive technology (EN 301 549)

5 Discussion and Final Remarks

The rapidly growing market of video games and VR headsets indicates an increase in the number of people who play games.

In this work, we address the chances and problems for people with disabilities regarding games and VR applications. Our comparison of existing game and VR guidelines provides a broader understanding on existing guidelines from various sources. It can also help the authors of the guidelines to improve them in the future as they see what might be missing. Furthermore, we hope this work can help raise awareness, especially for accessible VR games.

The comparison showed that none of the presented guidelines is an exhaustive list. We found that there are some important rules missing in the relevant works that are included in other guidelines. However, most rules are covered by either the Game Accessibility Guidelines or the EN 301 549 standard. Among game guidelines, only the GAG and Xbox Accessibility Guidelines include rules that are specific to VR. As can be seen, the guidelines from MediaLT (2004) and the Top Ten from IGDA GASIG (2015) do not add any rules to the set that are not included in other guidelines.

It should be noted that our resulting set of guidelines is based on literature research only, and that we have not conducted empirical research with users to identify possible omissions of accessibility requirements in existing guidelines. Therefore, the “comprehensive” set of guidelines that we present in this paper may need to be further extended in the future to address accessibility needs that have yet to be identified in the field.

We noticed that there are a few guidelines in the EN 301 549 standard that do not occur in the GAG. On the other hand, there are some rules that are missing in the European standard or are not stated with sufficient specificity. We conclude that the legal requirements are currently not sufficient to cover the full range of accessibility needs of the users. Therefore, we suggest that missing guidelines should be added to the European standard.

Another problem with the European standard is its structure and wording. During the evaluation it became apparent that the standard is very hard to read and understand. Rules that are not linked to WCAG can be interpreted in different ways and no examples are given. We fear that the EN 301 549 may not be suitable as a guide to be used by developers directly. A possible approach would be to translate the standard into an alternative version of more applicable rules with practical examples. Also, a tutorial should be provided that shows in detail how each criterion is applied to VR applications.

A last remark on the European standard relates to the fact that it does not include a table that lists all criteria that are legally required for VR applications. Such tables are given for Web and mobile applications. Therefore, it is currently unclear which criteria are enforced by the European Commission for VR applications in public institutions, as opposed to criteria that are “nice to have”.

The overall conclusion from working with the available guidelines was that there is room for improvement in all existing guidelines and including rules that are specific for VR should be considered by the most relevant guidelines.

A comprehensive and widely acknowledged set of accessibility guidelines for VR games is needed in the future, just as the Web Content Accessibility Guidelines for Web applications. The guidelines we presented in this paper can be a starting point for this. However, we use the wording of the original sources and there are no explanations or examples included. To make for a good checklist for developers to follow, a much more detailed description of each guideline would be necessary. Also, a companion tutorial would be useful to provide support for VR game developers who are new to the field of accessibility.

As mentioned, not only guidelines are underrepresented for accessibility, but there is also a lack of available tools for developers. Many of the approaches to avoid accessibility problems in games could be supported by suitable libraries and automatic checking tools. This takes some of the burden away from developers and makes development much easier and faster while ensuring a consistently high level of accessibility. Eventually, the employment of suitable platforms and libraries should ensure a decent level of accessibility, and the majority of guidelines could be automatically checked and hints for fixes provided by development tools.

Author Contributions

This manuscript was written by FH with corrections and minor adaptions made by GZ. The research work was conducted by FH and supervised by GZ and PM. All authors have read, and approved the manuscript before submission.

Time Compression Affect In Virtual Reality vs Conventional Monitor

The following report is available to download from here and was authored by Grayson Mullen & Nicolas Davidenko

Time & Time Perception: Abstract

Virtual-reality (VR) users and developers have informally reported that time seems to pass more quickly while playing games in VR. We refer to this phenomenon as time compression: a longer real duration is compressed into a shorter perceived experience. To investigate this effect, we created two versions of a labyrinth-like game. The versions are identical in their content and mode of control but differ in their display type: one was designed to be played in VR, and the other on a conventional monitor (CM). Participants were asked to estimate time prospectively using an interval production method. Participants played each version of the game for a perceived five-minute interval, and the actual durations of the intervals they produced were compared between display conditions. We found that in the first block, participants in the VR condition played for an average of 72.6 more seconds than participants in the CM condition before feeling that five minutes had passed. This amounts to perceived five-minute intervals in VR containing 28.5% more actual time than perceived five-minute intervals in CM. However, the effect appeared to be reversed in the second block when participants switched display conditions, suggesting large novelty and anchoring effects, and demonstrating the importance of using between-subjects designs in interval production experiments. Overall, our results suggest that VR displays do produce a significant time compression effect. We discuss a VR-induced reduction in bodily awareness as a potential explanation for how this effect is mediated and outline some implications and suggestions for follow-up experiments.

Keywords: Virtual realitybodily awarenessinteroceptiontime compressionprospective time estimationpresenceimmersion

1. Introduction

Virtual-reality (VR) head-mounted displays (HMDs) take up the user’s entire field of view, replacing all of their real-world visual cues with a contrived virtual world. This imposes unique conditions on human vision and on all other brain functions that make use of visual information. The consequences have mostly been studied in terms of presence, or the feeling of being inside the virtual scene presented on the HMD rather than in the real world (see Heeter, 1992 for a more encompassing and widely used definition of presence). Because the virtual scene can be designed to look like anything, VR can produce unique psychological effects by placing users in situations that rarely (or never) occur naturally. For example, it can present visual stimuli that conflict with the users’ vestibular cues, causing cybersickness (Davis et al., 2014). VR experiences have also been intentionally used to reduce pain in burn patients (Hoffman et al., 2011), to elicit anxiety or relaxation (Riva et al., 2007), and even to affect self-esteem and paranoia by manipulating the height of the user’s perspective relative to the virtual scene (Freeman et al., 2014).

One unintentional effect, which has been anecdotally reported by VR users and developers, is a time compression phenomenon wherein a larger real duration is compressed into a shorter perceived experience. At a 2016 gaming conference, Hilmar Veigar (of CCG Games) said, “You think you’ve played for 15 minutes and then you go out and it’s like, ‘Wait, I spent an hour in there?’ There’s a concept of, I don’t know, VR time” (Miller, 2016). Palmer Luckey (founder of Oculus) suggested that the effect could be a result of not having access to real-world environmental cues, like the position of the sun. Distorted time perception has been observed as an effect of conventional gaming (Nuyens et al., 2020), but the influence of VR on time perception has been studied relatively less.

One notable study (Schneider et al., 2011) successfully used VR experiences to shorten perceived durations during chemotherapy and found individual differences in time compression effects related to diagnosis, gender and anxiety. It is not clear, though, whether a non-VR version of the same experience would have resulted in a similar distortion of time perception. Only a few studies have directly compared time estimation processes between a VR experience and a non-VR counterpart, and none so far have found significant differences.

Bansal et al. (2019) examined the influence of a novel modification of a VR game (which coupled the flow of time to the speed of players’ body movements) on participants’ performance on subsequent time estimation tasks. Compared to control groups, participants who played the modified game made shorter estimates of brief (6 s and shorter) intervals, but only on estimation tasks that involved continuous movement. No significant difference in time perception was found between participants who played an unmodified (normal-time) version of the VR game and those who played a non-VR game. These results indicate that VR alone may not recalibrate temporal perception, but that a specifically tailored VR experience may induce such an effect. Because all the time estimation tasks were performed outside of VR, these results do not provide an answer to the question of whether time perception is distorted during VR use.

Schatzschneider et al. (2016) investigated how time estimation was affected by display type (VR/non-VR) and cognitive load. The researchers found no significant difference in time estimation between the display conditions, but the study used a within-subjects design and all participants experienced the non-VR condition first and the VR condition second. Completing the non-VR condition first may have anchored participants’ time estimates in the subsequent VR condition. Thus, it is possible that the lack of counterbalancing in Schatzschneider et al. (2016) may have obscured an effect of display type. Another study (van der Ham et al., 2019) also found no difference in time estimates between VR and non-VR displays, but used a retrospective time estimation paradigm.

According to Block and Zakay (1997), retrospective and prospective time estimates depend on different processes. Retrospective estimates are made when participants are unaware that they will be asked to estimate a duration until after the interval has ended. These estimates are based only on information that is stored in memory. Factors that have been found to affect retrospective time estimates are mostly related to stimulus complexity and contextual changes (more complex information and more changes are associated with longer retrospective estimates). Because they rely on memory, retrospective time estimates are affected by cognitive load only indirectly, when information relevant to cognitive load is stored in memory.

In contrast, prospective estimates are made by participants who are aware during the interval that they will be asked to estimate its duration. The most prominent model to illustrate the processes underlying prospective time estimation is Zakay and Block’s (1995) attentional-gate model of prospective time estimation (but see also Grondin, 2010Ivry & Schlerf, 2008; and Wittmann, 2009for reviews of alternate models of time perception). The first component of this abstract model is a pacemaker (which can be thought of as an internal metronome) that generates pulses at a rate that scales with the estimator’s arousal. Before the pulses can be counted, they are modulated by an attentional gate, which is open to a variable degree depending on the amount of attentional resources allocated to tracking time. When attentional resources are consumed by a demanding task, the gate becomes narrower (i.e., fewer resources are available to attend to time), and fewer pulses are able to pass.

The pulses that pass the attentional gate are counted by an accumulator, and the resulting sum is used as the basis for an estimate of the interval’s duration. The larger the count, the more time the estimator reports has passed. This means that time seems to pass more quickly (i.e., it becomes compressed) when attentional demands are high, and it seems to pass more slowly (i.e., it dilates) when attentional demands are low. The attentional-gate model is supported by the preponderance of attention-related manipulations that have been found to significantly affect prospective estimates, but not retrospective estimates (Block & Zakay, 1997). Thus, whereas prospective estimates are affected by cognitive load, retrospective estimates are more affected by contextual changes and other memory-related factors.

The current study is the first to investigate the effect of VR HMDs on time perception using a prospective time estimation paradigm and counterbalanced display conditions. We chose a prospective time estimation paradigm in order to measure the experience of VR rather than the memory of it (Block & Zakay, 1997), and also to obtain results that are relevant to intentional time management while playing VR games. We also used an interval production method of time estimation (Zakay, 1993), in which the research assistant specifies a duration (five minutes, in our case) and the participant starts and ends an interval that they feel matches that duration. This method is less susceptible to rounding biases than methods that ask the participant to report the number of seconds or minutes an interval lasted. In our study, every participant attempts to produce a five-minute interval, and we use the actual durations of the intervals they produce as our main dependent variable.

1.1. Hypotheses

First, we predict that intervals produced while playing a VR game will be longer than those produced while playing an equivalent game displayed on a conventional monitor (CM). This hypothesis is based on the anecdotal reports of a time compression effect in VR, and is motivated by past studies which have probed the relationship between time perception and VR but failed to find evidence of this effect. Based on Block and Zakay’s (1997) comparison of time estimation methods, we expect an interval production method to yield evidence of a compression effect in VR that has not been directly revealed by other methods.

Second, we predict that VR interval durations will be more variable across participants than CM interval durations. Higher variability is naturally expected if VR interval durations are longer, assuming that errors are proportional to the size of the estimate. Additionally, we predict that variability may be further increased by uncertainty in time perception among participants in VR. If VR interferes with normal time perception, participants may be less confident in their ability to track the passage of time, and produce a wider range of interval durations.

2. Methods

2.1 Participants

Forty-one undergraduate students participated for course credit. Two of them produced extreme outlier responses (their intervals in the VR condition were more than three standard deviations above the mean), so our final analysis includes data from 39 participants (24 female and 15 male, ages 18–26, M= 19.5, SD = 1.7). The UC Santa Cruz IRB approved the study and participants provided informed consent.

2.2 Materials

In both conditions, participants played a 3D labyrinth-like game designed in Unity. Each level consisted of a floating maze inside an otherwise empty room with textured walls and floors (see Fig. 1). The lighting and object textures did not change between levels, conditions, or maze sets, and there was no representation of the user’s body. The maze was positioned in front of and below the virtual camera to allow participants to see into the maze from above. Each maze contained a ball and a glowing yellow cube representing a goal, as well as walls and holes in the floor. Participants were directed to guide the ball to the goal by tilting the maze. Each version of the game included one of two maze sets (designed to be equally complex and difficult) so that participants did not repeat any levels between the two conditions. Each version included one practice level followed by up to 13 timed levels, which became increasingly difficult to complete as the mazes became larger and more complex (to simulate the general sense of progression in video games). Letting the ball fall through a hole in the maze would restart the current level, while getting the ball to reach the goal would start the next level. Above the maze in the timed levels, white text reading, “When you think five minutes have passed, press the right bumper and trigger at the same time” continuously faded in and out on an 8-s cycle to remind participants of the interval production task.

Figure 1.
Figure 1.

We decided it was important to include this reminder because when using an interval production method, the interval does not end until the participant chooses to end it. If a participant forgets that they were asked to keep track of time, they could produce exceedingly long intervals that are not accurately descriptive of their perception of time. Although the periodic fading of the reminder may have served as a temporal cue to make time perception more accurate, we do not expect it to have confounded our results because it was presented the same way in the VR and CM conditions of the game.

Participants used an Xbox 360 controller (Microsoft Corporation; Redmond, WA, USA) to manipulate the maze. They could tilt it in eight directions by moving the left joystick and could return it to its original position by holding any of the colored buttons (A, B, X, or Y). The right trigger and bumper (buttons at the back of the controller) were pressed simultaneously to end the practice level, and later to indicate the end the perceived 5-min interval.

In the VR condition, participants wore an Oculus Rift CV1 HMD (Oculus VR; Menlo Park, CA, USA) with head-tracking enabled to show a stable 3D environment. In the CM condition, participants viewed the game on a 20-inch Dell monitor with a 1920 × 1080 pixel resolution and a 60Hz refresh rate. Participants in the CM condition were seated approximately 45 cm away from the monitor. At this distance, the maze subtended approximately 22 degrees by 22 degrees of visual angle. Participants in the VR condition saw the maze from a virtual camera that was positioned similarly with respect to the maze, but the maze subtended a slightly larger visual angle (approximately 30 degrees by 30 degrees). However, participants were allowed to move freely during the game in both conditions, so the visual angle of the maze varied considerably across participants and across maze levels. Other than these differences between displays, the game was played on the same computer hardware between conditions.

After completing both conditions, participants filled out a questionnaire that asked about the difficulty of tracking time and of playing the game, their confidence in their ability to estimate time, previous experience with VR and video games, and included 19 Likert-scale items about immersion (e.g., “I felt detached from the outside world”). The purpose of this immersion scale was to measure whether participants felt significantly more immersed in the VR condition compared to the CM condition, and to show if immersion played a mediating role in any time compression effect we might find.

2.3 Procedure

We used a counterbalanced within-subjects design because we expected time perception accuracy to be highly variable between people. There were two display conditions (virtual reality [VR] and conventional monitor [CM]) as well as two sets of mazes (A and B). Each participant played the game once in VR and once on the CM, one of which used maze set A and the other used set B. Display condition and maze set were both counterbalanced to minimize order and maze difficulty effects.

Participants were asked to keep their phones and watches out of sight for the duration of the experiment, and to sit in front of a computer at a desk in our lab room. No clocks were visible to the participants, and research assistants in adjacent rooms refrained from using time-related language. Figure 2 illustrates the equipment used in each condition. A research assistant read instructions on how to play the game, and the practice level was started while the controls were described. Participants were told they could play the practice level for as long as they wanted to get comfortable with the game, and that it was not timed. Once they were ready to stop practicing, they could start the timed levels, which they were instructed to end once they felt they had been playing for five minutes. The research assistant left the room and shut the door after the instructions to minimize distractions and aural cues from outside the room.

Figure 2.
Figure 2.

We chose not to vary the duration of the intervals that participants were instructed to produce because of our limited sample size. We set the target duration at five minutes because it is a familiar and memorable unit of time, and we expected it would be long enough to discourage deliberate counting of seconds, but short enough to minimize fatigue effects (especially in the second sessions).

When the participant ended the timed levels, the elapsed time in seconds since the end of the practice level was automatically recorded in a text file, along with their practice time and the level that the participant had reached. No feedback about how much time had actually passed was given to the participant. Then, the research assistant briefly reminded the participant of the instructions and started the second game, which used the display condition and maze set that were not used in the first game.

After both versions of the game were completed, the participant was brought to a new room to complete a post-task survey (see Materials above).

3. Results

We conducted a two-way mixed-effects ANOVA with factors of starting display type (VR or CM) and block number (first or second). The results, shown in Fig.3, revealed a main effect of block number (F1,37 = 9.94, p = 0.003, ηp2 = 0.212), indicating that the mean duration of intervals produced in the second block (341.9 s) was significantly longer than that of intervals produced in the first block (290.1 s). Importantly, there was a main effect of starting display type (F1,37 = 6.45, p = 0.015, ηp2 = 0.148). Participants who played the VR game first (and the CM game second) produced longer intervals than participants who played the CM game first (and the VR game second). This means that the effect of display type on interval duration depends on order: in the first block, participants in the VR condition produced longer durations (327.4 s on average) than participants in the CM condition (254.8 s), whereas in the second block, VR durations (299.9 s) were shorter than CM durations (386.2 s). Furthermore, we found a strong correlation between participants’ first and second interval durations (r = 0.62, p < 0.001, n = 39), suggesting individuals’ second intervals were heavily anchored to their first ones. Because of this order effect, we limit our remaining analyses to first-block responses.

Figure 3.
Figure 3.

As shown in Fig. 4, first-block participants in the VR condition let significantly more time pass than first-block participants in the CM condition before indicating that five minutes had passed (t37 = 2.165, p = 0.037, d = 0.693). VR intervals were 327.4 s long (SD = 114.0) on average, and CM intervals were 254.8 s (SD = 95.1) on average. This means that in the VR condition, 72.6 more seconds (95% CI, [4.6, 140.6]) passed on average before participants felt that five minutes had elapsed. This finding supports our first hypothesis, that participants experience time compression in VR compared to playing an identical game on a CM.

Figure 4.
Figure 4.

To rule out an account based on differences in task difficulty, we compared how quickly participants in the two conditions completed the levels of the maze game. Figure 5 shows that the relationship between interval duration and level reached is described by a similar linear relationship in the two conditions. To determine whether these slopes were significantly different, we ran 10,000 bootstrap samples from each condition to compare the resulting best-fit lines and found that the 95% confidence interval for the difference between best-fit slopes in the VR and the CM condition [−0.0021, 0.0072] contained zero. Therefore participants across the VR and CM conditions completed levels at similar rates, suggesting that the time compression effect cannot be attributed to participants spending more time on each level in VR compared to CM and using the number of levels completed as a proxy to decide when five minutes had elapsed. Furthermore we found no significant difference in practice time between conditions (t37 = −0.147 p > 0.5, d = 0.047) suggesting it was not more difficult to learn the game in VR than in CM.

Figure 5.
Figure 5.

We did not find support for the hypothesis that produced interval durations would be more variable in the VR condition. Although intervals produced in the VR condition (SD = 114 s) were slightly more variable than intervals produced in the CM condition (SD = 95 s), Levene’s test showed that there was no significant difference in interval variance between conditions (F1,37 = 0.195, p > 0.5).

The survey responses did not reveal a significant relationship between interval durations and previous experience with video games or with VR, nor was there a significant difference between conditions in rated difficulty (either of the game or of keeping track of time). This result conflicts with our second prediction that time estimation in VR would be more difficult, and that produced intervals would therefore be more variable in VR compared to CM. However, because the survey was administered after participants had completed both tasks, it is possible that participants’ responses pertaining to one condition were confounded by their experience with the other. In fact, we found no significant differences in ratings of immersion between the VR and CM conditions. Only one of the 19 Likert scales about immersion (“I did not feel like I was in the real world but the game world”) appeared to be higher in VR compared to CM (t36 = 2.215, p = 0.033, d = 0.717), but this difference did not reach significance at the Bonferroni-corrected alpha of 0.0026 (see Supplementary Table S1 for the complete immersion scale results). The surprising lack of an immersion difference between conditions suggests that administering the survey after both conditions were completed may have diminished our ability to detect an effect.

4. Review/Discussion

These results constitute the first evidence that VR as a medium produces a unique time compression effect. At least one previous experiment (Schneider et al., 2011) successfully used VR to produce a similar effect, but the present study is the first to observe time compression as a significant difference between VR and non-VR experiences with otherwise identical content. Importantly, our results suggest that there is something inherent about the VR interface (as opposed to a characteristic of its content) that produces a time compression effect.

Most of the previously observed effects on prospective time estimation are related to attention, but the significance of our main finding does not appear to be attributable to a difference in attentional demands. The tasks in both conditions were of identical complexity and difficulty; the two sets of maze levels were counterbalanced across conditions, and participants in both conditions spent about the same amount of time on each level.

The VR condition did present a simpler scene to the participant than the CM condition (it had a narrower field of view, and the physical lab environment was not visible), but this is unlikely to explain our effect either. Visual-stimulus complexity has been found to only affect retrospective estimates (Block & Zakay, 1997). If we were to repeat this experiment using retrospective estimates, we would expect to find shorter perceived intervals in the VR condition, because the VR scene presents a smaller amount of information that could be later recalled from memory. This would also be a kind of time compression effect, but assuming that the participants’ attention remains on the screen during the interval, we would expect a much weaker effect than the one we found. Based on Block and Zakay’s (1997) meta-analysis, though, stimulus complexity should have no significant effect on prospective estimation tasks like the one we used.

Arousal can also influence prospective time estimation in general, but it is highly unlikely to explain our main finding because of the direction of its effect. Images displayed in VR have been found to elicit higher arousal than the same images displayed on conventional monitors (Estupiñán et al., 2014), but higher arousal is associated with time dilation, according to the attentional-gate model (Zakay & Block, 1995). In the context of our study, this would predict that participants in the VR condition would produce shorter intervals than participants in the CM condition. Because produced intervals in the VR condition were in fact longer, we conclude that arousal did not play a role in the main effect we observed, either.

One difference between our two conditions that does seem likely to be responsible for the effect is that participants could not see their own body, or any representation of it, in the VR condition. In pacemaker–accumulator models of time perception, pulse generation is treated as an abstract module of the time estimation process, but it is thought to be a function of bodily rhythms like heart rate, breathing, or neural oscillations (Pollatos et al., 2014Wittmann, 2009). The model’s inclusion of arousal as an influence on the pacemaker is based on this assumption, and there is accumulating evidence that time estimation accuracy is dependent on awareness of bodily rhythms. It has been found that time estimation accuracy is significantly correlated both with ability to estimate one’s own heart rate (Meissner & Wittmann, 2011), and with heart rate variability itself (Cellini et al, 2015). A more recent study found that people with high interoceptive accuracy are less susceptible to emotion-induced distortions of time perception (Özoğlu & Thomaschke, 2020).

Bodily awareness was measured as a participant variable in those studies, but it can also be manipulated. An experiment which used a VR and non-VR version of the same interactive environment found that bodily awareness was reduced in VR (Murray & Gordon, 2001). Specifically, the participants in the VR condition gave significantly lower ratings on scales of cardiovascular, skin, and muscle awareness. This is presumably related to the absence of any visible representation of the users’ body in the VR scene.

The combination of these two findings, (1) that prospective time estimation accuracy is related to awareness of bodily rhythms and (2) that being in VR reduces bodily awareness, suggests a likely explanation for the effect observed in the current study: participants in the VR condition were less aware of the passage of time because they were less aware of the bodily rhythms that form the basis of prospective time perception.

This is notable because the most prominent models of prospective time estimation do not account for interoceptive awareness as an independent influence on perceived interval durations. For example, pacemaker–accumulator models like Zakay and Block’s (1995) attentional gate include arousal, attention, and reference memory ‒ but not interoceptive awareness ‒ as influences on prospective time estimation. Because we suspect that a difference in interoceptive awareness (and not in attention, arousal, or memory) best explains the VR-induced time compression effect, models like these might be modified to account for interoceptive awareness as an independent influence on prospective time estimation. Dedicated timing models (Ivry & Schlerf, 2008) such as the attentional-gate model involve a pacemaker module that produces pulses that depend on bodily rhythms such as heart rate, breathing, or neural oscillations. We propose that such models might be amended to include interoceptive awareness as a factor that mediates the reception of these pulses. Impairing interoceptive awareness would lead to underestimations of time by reducing the number of pulses that ultimately reach the accumulator. Although prominent models so far have not treated interoceptive awareness as its own factor, our results suggest that it may affect time estimation independently from attentional demands, arousal, and reference memory.

The durations of participants’ second intervals were heavily anchored to first interval durations. It could be that the time production task in the first block severely revised each participants’ reference for what a five-minute interval feels like, and caused them to use that new reference to produce the second interval. Second intervals were also longer. This effect was exhibited by participants who played the VR version first and then switched to CM, as well as those who started with CM and switched to VR. The greater durations of second block intervals could be due to a novelty effect which may have dilated time perception more during the first block compared to the second block. Alternatively, participants may have expected to complete more levels in a 5-min period during the second block after having gained experience with the task. If participants expected to complete more levels in the second block, and used the level reached in the first block as a proxy to indicate the passing of five minutes, they may have purposely played additional levels in the second block. In fact, participants did on average play one additional level in the second block, but the rate of completing levels was no faster compared to the first block.

It is well established that order effects in general can confound results when counterbalancing is not used, but in our case the order effect was so overwhelming that the time compression effect becomes completely obscured if we analyze our data without regard for condition order. This suggests that counterbalancing may not be sufficient for experiments which use interval production tasks, and that future studies should use between-subjects designs when possible.

A follow-up experiment could further investigate the role of interoception in VR-induced time compression by having participants complete a bodily awareness scale after they complete the maze game. Using a between-subjects design in such an experiment would allow the questionnaire to be administered immediately after a single playthrough of the maze game, making it more valid than ours (which was administered after participants had completed both conditions).

Including an additional VR condition with a virtual body representation could also help clarify the role of body visibility in time perception (and more broadly, in bodily awareness). It is unclear now if hiding one’s body from view is enough to reduce bodily awareness, or if the effect depends on the VR-induced feeling of presence that makes the user feel as though they are in a place that is remote from their body. If adding a virtual body were found to both increase bodily awareness and mitigate the time compression effect, that would support the idea that reduced body visibility is responsible for the main effect we observed. If that manipulation were found to have no impact on bodily awareness or the time compression effect, it would suggest that the effect depends not on body visibility but on some higher-level feeling of virtual presence.

Another limitation of the present experiment is that we did not vary the duration of the interval that participants were asked to produce. Bisson and Grondin (2013) and Tobin et al. (2010) found that during gaming and internet-surfing tasks, significant time compression effects were only evident during longer sessions (around 30 min or longer). The authors of those studies note that this difference may be due to the time estimation methods they used: participants were asked to verbally estimate durations, and might have rounded their answers to multiples of five minutes. This rounding bias would have a much stronger influence on the results of their shorter-interval trials (12 min) than on their longer-interval trials (35, 42, or 58 min). Our finding of a time compression effect on a five-minute scale suggests that the interval production method we used likely protected our results from such a rounding bias. It is unclear whether or how the VR-induced effect we found might depend on the target duration of the produced interval. Future studies investigating this effect could explore this influence by instructing participants in different conditions to produce intervals shorter and longer than five minutes.

If transient reminders like the one we used are employed during prospective time estimation tasks, we recommend that the durations of the interval be pseudo-randomized. Our reliably periodic reminder may have helped our participants produce more accurate intervals in both conditions. Making the cue unreliable might reveal a larger effect, which could be crucial in experiments that test time perception in more delicate contexts.

4.1 Implications for VR Experience Design

An average of 28.5% more real time passed for participants who played the VR game than for those in the control group ‒ with no difference in perceived duration. If this effect proves to generalize to other contexts at similar magnitudes, it will have significant implications. Keeping track of time accurately is desirable in most situations, and impairing that ability could be harmful.

Time compression might cause VR users to unintentionally spend excessive amounts of time in games, especially as HMDs become more comfortable to wear for long sessions. Even non-immersive games entail some risk of addiction, which has been associated with depression and insomnia (Kuss & Griffiths, 2012). VR games may pose a greater risk of interfering with their players’ sleep schedules, mood, and health by reducing their ability to notice the passage of time. Developers should take care not to create virtual ‘casinos’; a clock should always be easily accessible, and perhaps even appear automatically at regular intervals.

On the other hand, time compression effects can be desirable in situations that are unpleasant but necessary, and there are potential applications that could take advantage of the effect in a beneficial way. VR might be used, for example, to reduce the perceived duration of long-distance travel. More importantly, the value of using VR to make chemotherapy more bearable (Schneider et al., 2011) is supported by the current study. Especially considering that VR has been used successfully as an analgesic (Hoffman et al., 2011), VR experiences could be similarly applied to reduce the negative psychological impact of other painful medical treatments. Our interpretation of the results suggests that other equipment or treatments which reduce bodily awareness, such as sensory deprivation tanks, may also be useful for producing time compression effects.

Supplementary Material

Supplementary material and the full original version of this report is available online at:  https://doi.org/10.6084/m9.figshare.14339438

References

Bansal, A., Weech, S., & Barnett-Cowan, M. (2019). Movement-contingent time flow in virtual reality causes temporal recalibration. Sci. Rep., 9, 4378. doi: 10.1038/s41598-019-40870-6.

Search Google Scholar

Export Citation

Bisson, N., & Grondin, S. (2013). Time estimates of internet surfing and video gaming. Timing Time Percept., 1, 39–64. doi: 10.1163/22134468-00002002.

Search Google Scholar

Export Citation

Block, R.A., & Zakay, D. (1997). Prospective and retrospective duration judgments: A meta-analytic review. Psychon. Bull. Rev., 4, 184–197. doi: 10.3758/BF03209393.

Search Google Scholar

Export Citation

Cellini, N., Mioni, G., Levorato, I., Grondin, S., Stablum, F., & Sarlo, M. (2015). Heart rate variability helps tracking time more accurately. Brain Cogn., 101, 57–63. doi: 10.1016/j.bandc.2015.10.003.

Search Google Scholar

Export Citation

Davis, S., Nesbitt, K., & Nalivaiko, E. (2014). A systematic review of cybersickness. Proc 2014 Conf. Interact. Entertain., 1–9. doi: 10.1145/2677758.2677780.

Search Google Scholar

Export Citation

Estupiñán, S., Rebelo, F., Noriega, P., Ferreira, C., & Duarte, E. (2014). Can virtual reality increase emotional responses (arousal and valence)? A pilot study. In A. Marcus (Ed.), Design, User Experience, and Usability. User Experience Design for Diverse Interaction Platforms and Environments. DUXU 2014. Lecture Notes in Computer Science, Vol. 8518, pp. 541–549. Springer, Cham, Switzerland. doi: 10.1007/978-3-319-07626-3_51.

Search Google Scholar

Export Citation

Freeman, D., Evans, N., Lister, R., Antley, A., Dunn, G., & Slater, M. (2014). Height, social comparison, and paranoia: An immersive virtual reality experimental study. Psychiat. Res., 218, 348–352. doi: 10.1016/j.psychres.2013.12.014.

Search Google Scholar

Export Citation

Grondin, S. (2010). Timing and time perception: a review of recent behavioral and neuroscience findings and theoretical directions. Atten. Percept. Psychophys., 72, 561–582. doi: 10.3758/APP.72.3.561.

Search Google Scholar

Export Citation

Heeter, C. (1992). Being there: the subjective experience of presence. Presence (Camb.), 1, 262–271. doi: 10.1162/pres.1992.1.2.262.

Search Google Scholar

Export Citation

Hoffman, H. G., Chambers, G. T., Meyer, III, W. J., Arceneaux, L. L., Russell, W. J., Seibel, E. J., Richards, T. L., Sharar, S. R., & Patterson, D. R. (2011). Virtual reality as an adjunctive non-pharmacologic analgesic for acute burn pain during medical procedures. Ann. Behav. Med., 41, 183–191. doi: 10.1007/s12160-010-9248-7.

Search Google Scholar

Export Citation

Ivry, R. B., & Schlerf, J. E. (2008). Dedicated and intrinsic models of time perception. Trends Cogn. Sci., 12, 273–280. doi: 10.1016/j.tics.2008.04.002.

Search Google Scholar

Export Citation

Kuss, D. J., & Griffiths, M. D. (2012). Internet gaming addiction: a systematic review of empirical research. Int. J. Ment. Health Addict., 10, 278–296. doi: 10.1007/s11469-011-9318-5.

Search Google Scholar

Export Citation

Meissner, K., & Wittmann, M. (2011). Body signals, cardiac awareness, and the perception of time. Biol. Psychol., 86, 289–297. doi: 10.1016/j.biopsycho.2011.01.001.

Search Google Scholar

Export Citation

Miller, R. (2016) Oculus founder thinks VR may affect your ability to perceive time passingThe Vergehttps://www.theverge.com/2016/3/17/11258718/palmer-luckey-oculus-time-vr-virtual-reality-gdc-2016.

Search Google Scholar

Export Citation

Murray, C. D., & Gordon, M. S. (2001). Changes in bodily awareness induced by immersive virtual reality. CyberPsychol. Behav., 4, 365–371. doi: 10.1089/109493101300210268.

Search Google Scholar

Export Citation

Nuyens, F. M., Kuss, D. J., Lopez-Fernandez, O., & Griffiths, M.D. (2020). The potential interaction between time perception and gaming: A narrative review. Int. J. Ment.l Health Addict., 18, 1226–1246. doi: 10.1007/s11469-019-00121-1.

Search Google Scholar

Export Citation

Özoğlu, E., & Thomaschke, R. (2020). Knowing your heart reduces emotion-induced time dilation. Timing Time Percept., 8, 299–315. doi: 10.1163/22134468-bja10016.

Search Google Scholar

Export Citation

Pollatos, O., Yeldesbay, A., Pikovsky, A., & Rosenblum, M. (2014). How much time has passed? Ask your heart. Front. Neurorobot., 8, 15. doi: 10.3389/fnbot.2014.00015.

Search Google Scholar

Export Citation

Riva, G., Mantovani, F., Capideville, C. S., Preziosa, A., Morganti, F., Villani, D., Gaggioli, A., Botella, C., & Alcañiz, M. (2007). Affective interactions using virtual reality: the link between presence and emotions. CyberPsychol. Behav., 10, 45–56, doi: 10.1089/cpb.2006.9993.

Search Google Scholar

Export Citation

Schatzschneider, C., Bruder, G., & Steinicke, F. (2016). Who turned the clock? Effects of manipulated zeitgebers, cognitive load and immersion on time estimation. IEEE Trans. Vis. Comput. Graph., 22(4), 1387–1395. doi: 10.1109/tvcg.2016.2518137.

Search Google Scholar

Export Citation

Schneider, S. M., Kisby, C. K. & Flint, E. P. (2011). Effect of virtual reality on time perception in patients receiving chemotherapy. Support. Care Cancer, 19, 555–564. doi: 10.1007/S00520-010-0852-7.

Search Google Scholar

Export Citation

Tobin, S., Bisson, N., & Grondin, S. (2010). An ecological approach to prospective and retrospective timing of long durations: a study involving gamers. PloS ONE 5, e9271. doi: 10.1371/journal.pone.0009271.

Search Google Scholar

Export Citation

van der Ham, I. J. M., Klaassen, F., van Schie, K., & Cuperus, A. (2019). Elapsed time estimates in virtual reality and the physical world: the role of arousal and emotional valence. Comput. Hum. Behav., 94, 77–81. doi: 10.1016/j.chb.2019.01.005.

Search Google Scholar

Export Citation

Wittmann, M. (2009). The inner experience of time. Philos. Trans. R. Soc. B Biol. Sci., 364, 1955–1967. doi: 10.1098/rstb.2009.0003.

Search Google Scholar

Export Citation

Zakay, D. (1993). Time estimation methods ‒Do they influence prospective duration estimates?. Perception, 22, 91–101. doi: 10.1068/p220091.

Search Google Scholar

Export Citation

Zakay, D., & Block, R. A. (1995). An attentional-gate model of prospective time estimation. In M. Richelle, V. D. Keyser, G. d’Ydewalle & A. Vandierendonck (Eds), Time and the dynamic control of behavior (pp. 167–178). Liège, Belgium: Université de Liège.

Search Google Scholar

Export Citation


Portalco delivers an alternate method of delivery for any VR or non-VR application through advanced interactive (up-to 360-degree) projection displays. Our innovative Portal range include immersive development environments ready to integrate with any organisational, experiential or experimental requirement. The Portal Play platform is the first ready-to-go projection platform of it's type and is designed specifically to enable mass adoption for users to access, benefit and evolve from immersive projection technologies & shared immersive rooms.

Virtual Reality- Understanding The Screen Door Effect

Have you ever worn a virtual reality headset excited for the immersive experience, only to have it marred by a disruptive black grid blanketing your vision? That isn’t your eyes playing tricks on you: it’s actually a real tech phenomenon, known as the screen door effect.

Of course, the key to a successful immersive application is to captivate the user and involve themin the virtual world, not just offer them a view from the outside (we already have standard screens and monitors for that!). And if the entire world is blanketed by a strange meshing like the one you get on a screen door, it may feel as painful as being slammed in the face by said door...okay, maybe not that painful, but the full power of immersion can’t be unlocked.

Why exactly does the screen door effect happen?

Nowadays, most electronic screens are either LCD or OLED displays. They are made up of loads of pixels, each consisting of different-coloured subpixels (red, green and blue).

LCD

If you have a TV or a computer monitor, chances are it’s an LCD. All of the pixels are lit up by a singular backlight, which illuminates the entire screen.

OLED

OLEDs are most commonly found in smartphones and virtual reality headsets (though OLED TVs and monitors exist now too!). OLED displays tend to be crisper; the structure of the pixels is often quite different, and each individual pixel emits light by itself.

Wasted space

As illustrated in the above images, between each of the pixels (and subpixels) is an area of unlit space - and it’s this space that is the culprit of the intrusive screen meshing.

The screen door effect isn’t unique to VR displays; if you sit too close to the television or put a magnifying glass over your smartphone’s screen, you may be able to see the miniscule mosaic of red, green and blue shapes. But when you casually watch TV or text a friend, your eyes will be at a comfortable distance from the screen where you won’t even be able to make out the distance between each pixel.

However, with head-mounted devices, it’s a different story. When you wear a virtual reality headset, your eyes are much closer to the display than they would be looking at a phone, computer monitor or television. The lens magnification and all-encompassing field of view of the headset visuals only further exaggerate the gap between the pixels.

How can we prevent it?

Due to the way LCD and OLED screens work, the screen door effect is impossible to shut out completely (at least at the moment). That doesn’t mean there aren’t any ways we can make the effect less prevalent, but they’re not without their own drawbacks.

Needs more cowbell pixels

Probably the most obvious option. Increasing the amount of pixels per inch (ppi) decreases the amount of black space in the display, and more and more tech companies are recognising the need to improve the quality and crispness of their product’s visual screens. Nowadays, a commercial VR headset display can have upwards of 500 pixels per inch

However, a higher pixel density may require a larger display to host them all on, not to mention a larger amount of system memory to handle each pixel’s information without sacrificing motion smoothness.

Diffusion filters

Diffusion filters are a common and inexpensive visual trick for making images look softer and smoother. The material of the diffuser scatters the emitted light, creating the illusion of the pixels blending into one another and eliminating a lot of the explicit gaps between each one.

It’s important to note, however, that this technique also carries some burdens of its own. If not done carefully, a diffuser may make visuals look too blurry to be convincing, and even cause more disruptive artifacts such as sparkling and moiré-patterned (ripple-like) interference. Many users may also feel themselves having to make more effort to focus on the visuals, resulting in both hindered immersion and bad eye strain!

Low pass filters

A low-pass filter positioned over the screen is a straightforward way of filtering out the most high-frequency black space of the display.

Not only does implementing such a filter bump up the costs, but also power consumption (which is also expensive, and in today’s ecologically-minded climate, not ideal!)

Mechanical shifting

Researchers at Facebook Reality Labs, the developers of the Oculus Rift display, have proposed a way of mitigating the screen door effect via piezo actuators, devices which convert electrical energy into a high-speed force of displacement without any moving parts. Two of these devices would shift the pixels along the screen so that they occupy unlit gaps, minimising the perceived distance between them. While this sounds like it would make everything shaky, this would be done at such a rapid rate that the picture would still appear stable to the viewer.

Though studies into this method are currently limited, the researchers highlight that this method is most likely not ‘one-size-fits-all’, and the screen door effect in a display would need to be inspected and characterised before deciding how the pixels should be moved.

Microlenses

In 2018, Valve Corp (creators of the Valve Index VR headset and the Steam game marketplace) filed a patent for a display they believe can greatly mitigate the screen door effect.

Sandwiched between this display’s lens and the eye’s view is a phase optic, which consists of an array of microlenses (tiny lenses less than a millimetre in diameter, often in micrometres!). These would magnify the pixels and scatter light, making the meshing of gaps much more subtle. While this is probably the most sophisticated method mentioned, it’s also very expensive and fiddly to implement, so it may be a long time before we see this technique in use.