How virtual reality is redefining soft skills training- a Pwc Study.

Example pattern for desktop

Summary

7 minute read

The VR advantage

Employers are facing a dilemma: Their workforce needs to learn new skills, upgrade existing capabilities or complete compliance training, but may not be able to do so in person given the current environment. Yet, training is especially important now, with employees so keen to gain skills, and it may become even more critical when workers start returning to a changed workplace. So how can employers deal with the challenge?

One solution to this training problem comes from an unexpected place: virtual reality (VR).

VR is already known to be effective for teaching hard skills and for job skills simulations, such as a flight simulator to train pilots. But many employees also need to learn soft skills, such as leadership, resilience and managing through change.1:35Play Video

Tech effect

So how does VR measure up as a training tool for these and other soft skills?

PwC set out to answer this question with our study of VR designed for soft skills training. Selected employees from a group of new managers in 12 US locations took the same training — designed to address inclusive leadership — in one of three learning modalities: classroom, e-learn and v-learn (VR).

The results? The survey showed that VR can help business leaders upskill their employees faster, even at a time when training budgets may be shrinking and in-person training may be off the table, as people continue to observe social distancing.

VR learners were:
Statistics: VR learners

Five top findings about the value of VR in soft skills training

Here are five takeaways that can help you support your employees’ digital learning needs:

1. Employees in VR courses can be trained up to four times faster

US employees typically spend only 1% of their workweek on training and development, so employers need to be sure that they use that time productively. That’s where VR can help.

What took two hours to learn in the classroom could possibly be learned in only 30 minutes using VR. When you account for extra time needed for first-time learners to review, be fitted for and be taught to use the VR headset, V-learners still complete training three times faster than classroom learners. And that figure only accounts for the time actually spent in the classroom, not the additional time required to travel to the classroom itself.

Time to complete training
Pie chart: Employees VR courses

2. VR learners are more confident in applying what they’re taught

When learning soft skills, confidence is a key driver of success. In difficult circumstances, such as having to give negative feedback to an employee, people generally wish they could practice handling the situation in a safe environment. With VR, they can.

Because it provides the ability to practice in an immersive, low-stress environment, VR-based training results in higher confidence levels and an improved ability to actually apply the learning on the job. In fact, learners trained with VR were up to 275% more confident to act on what they learned after training — a 40% improvement over classroom and 35% improvement over e-learn training.Bar chart titledImprovement in confidence discussing issues and acting on issues of diversity and inclusion after the trainingDiscussing issuesActing on issuesClassroom166%198%E-learn179%203%VR245%275%Source: PwC VR Soft Skills training Efficacy Study, 2020

3. Employees are more emotionally connected to VR content

People connect, understand and remember things more deeply when their emotions are involved. (We learned that during the VR study and multiple BXT experiences, where we gathered different viewpoints and worked together to identify what matters most.) Simulation-based learning in VR gives individuals the opportunity to feel as if they’ve had a meaningful experience.

V-learners felt 3.75 times more emotionally connected to the content than classroom learners and 2.3 times more connected than e-learners. Three-quarters of learners surveyed said that during the VR course on diversity and inclusion, they had a wake-up-call moment and realized that they were not as inclusive as they thought they were.Bar chart titledAverage emotional connection felt to learning contentClassroom4.29E-learn5.29VR20.43Source: PwC VR Soft Skills training Efficacy Study, 2020

4. VR learners are more focused

Today’s learners are often impatient, distracted and overwhelmed. Many learners will not watch a video for its duration, and smartphones are a leading cause of interruption and distraction.

With VR learning, users are significantly less distracted. In a VR headset, simulations and immersive experiences command the individual’s vision and attention. There are no interruptions and no options to multitask. In our study, VR-trained employees were up to four times more focused during training than their e-learning peers and 1.5 times more focused than their classroom colleagues. When learners are immersed in a VR experience, they tend to get more out of the training and have better outcomes.

Comparison chart: How focused are VR learners?

5. VR learning can be more cost-effective at scale

In the past, VR was too expensive, complicated and challenging to deploy outside of a small group. Today, the cost of an enterprise headset ecosystem is a one-time fee of less than $1,000, and these units can be managed like any other enterprise mobile device and can be used repeatedly to deliver training. Studios of all sizes are developing compelling content, while vendors are creating software packages to enable non-VR developers to create their own content in a cost-effective way. Elsewhere, some big learning-management-system players are enabling VR content to be easily integrated into their platforms.

The value VR provides is unmistakable when used appropriately. In our study, we found that, when delivered to enough learners, VR training is estimated to be more cost-effective at scale than classroom or e-learning. Because VR content initially requires up to a 48% greater investment than similar classroom or e-learn courses, it’s essential to have enough learners to help make this approach cost-effective. At 375 learners, VR training achieved cost parity with classroom learning. At 3,000 learners, VR training became 52% more cost-effective than classroom. At 1,950 learners, VR training achieved cost parity with e-learn. The more people you train, the higher your return will likely be in terms of employee time saved during training, as well as course facilitation and other out-of-pocket cost savings.

Training modality cost per learner
Line graph: Training modality cost per learner

Building a blended learning curriculum

While VR will not replace classroom or e-learn training anytime soon, it should be part of most companies’ blended learning curriculum. VR learning differentiates itself by combining the elements of a well-planned BXT experience: business expertise to tackle challenges, a human-centered experience and the right technology to boost productivity without sacrificing quality. Ideally, an entire team would take this training and then have follow-up discussions to determine how they can apply the learned skills in their jobs.

VR can help people make more meaningful connections by allowing learners to practice skills that help them relate to diverse perspectives in the real world. For example, PwC developed a VR soft skills course that enables executives and staff to practice new sales approaches. Learners get to make a pitch to a virtual CEO, but if they rely on business-as-usual sales techniques, the virtual CEO asks them to leave her office. However, if learners apply skills that demonstrate how they can bring value to the CEO’s company, they get a “virtual contract” at the end of the conversation.

The simplicity of this technology is another good reason to start using VR at scale in your organization. In the study, our team was able to provision, deploy and manage a large fleet of VR headsets with a very small team. That success makes it easy to imagine a day when all employees will be issued their own headsets, along with the requisite laptops, on their first day on the job. That would be a truly new way of working.

Achieving Presence Through Evoked Reality

Jayesh S. Pillai1*, Colin Schmidt1,2 and Simon Richir1

The following report collates a variety of information and perspectives on our multiple realities and how these can impact an immersive experience but more so the human experience which is so critical to long-lasting and user-focused digital experiences that improve memorization, understanding and engagement as a whole.

The sense of “Presence” (evolving from “telepresence”) has always been associated with virtual reality research and is still an exceptionally mystifying constituent. Now the study of presence clearly spans over various disciplines associated with cognition. This paper attempts to put forth a concept that argues that it’s an experience of an “Evoked Reality (ER)” (illusion of reality) that triggers an “Evoked Presence (EP)” (sense of presence) in our minds. A Three Pole Reality Model is proposed to explain this phenomenon. The poles range from Dream Reality to Simulated Reality with Primary (Physical) Reality at the center. To demonstrate the relationship between ER and EP, a Reality-Presence Map is developed. We believe that this concept of ER and the proposed model may have significant applications in the study of presence, and in exploring the possibilities of not just virtual reality but also what we call “reality.”

Introduction

Research on presence has brought to our understanding various elements that certainly cause or affect the experience of presence in one way or another. But in order to evoke an illusion of presence, we in effect try to generate an illusion of reality different from our apparent (real world) reality through different mediations like Virtual Reality. The attempt to evoke an illusory reality is what brought researchers to think about presence in the first place. “Reality,” despite its being a major concept, is most often either overlooked or confused with other aspects that affect presence. To study presence we must first understand the reality evoked in one’s mind. It is this illusion of reality that forms a space-time reference in which one would experience presence. It is evident from the research in the field of virtual reality, that if a medium is able to create a convincing illusion of reality, there will certainly be a resultant feeling of presence. Various theories have been proposed, to explore and define the components of this mediated presence. We aim to abridge those theories in an efficient manner. Moreover, studies in the field of cognition and neuroscience confirm that the illusion of reality can as well be non-mediated (without the help of external perceptual inputs), that is purely evoked by our mind with an inception of corresponding presence. One of the most common but intriguing example of a non-mediated illusion of reality would be – a dream. This self evoking faculty of mind leading to the formation of presence is often neglected when observed from the perspective of virtual reality.

Sanchez-Vives and Slater (2005), suggest that presence research should be opened up, beyond the domain of computer science and other technologically oriented disciplines. Revonsuo (1995) proposed that we should consider both – the dreaming brain and the concept of Virtual Reality, as a metaphor for the phenomenal level of organization; they are excellent model systems for consciousness research. He argues that the subjective form of dreams reveals the subjective, macro-level form of consciousness in general and that both dreams and the everyday phenomenal world may be thought of as constructed “virtual realities.”

According to Revonsuo (2006), any useful scientific approach to the problem of consciousness must consider both the subjective psychological reality and the objective neurobiological reality. In Virtual Reality it’s not just the perceptual input and the technical faculties that contribute to a stronger illusion of reality but also various psychological aspects (Lombard and Ditton, 1997Slater, 20032009) relating to one’s emotion, attention, memory, and qualia (Tye, 2009) that help mold this illusion in the mind. In the case of non-mediated illusion of reality like dreams or mental imagery, the perceptual illusion is generated internally (Kosslyn, 19942005LaBerge, 1998). The dream images and contents are synthesized to fit the patterns of those internally generated stimulations creating a distinctive context for the dream reality (DR; Hobson and McCarley, 1977Hobson, 1988). Whether mediated or non-mediated, the illusion of reality is greatly affected by the context. “A context is a system that shapes conscious experience without itself being conscious at that time” (Baars, 1988, p. 138). Baars describes how some types of contexts shape conscious experience, while others evoke conscious thoughts and images or help select conscious percepts. In fact it’s a fine blend of perceptual and psychological illusions (explained in section The Illusion of Reality) that leads to a strong illusion of reality in one’s mind. We attempt to explore this subjective reality that is the fundamental source of experience for presence.

Presence and Reality

With the growing interest in the field of Virtual Reality, the subject of presence has evolved to be a prime area of research. The concept of presence, as Steuer (1992) describes, is the key to defining Virtual Reality in terms of human experience rather than technological hardware. Presence refers not to one’s surroundings as they exist in the physical world, but to the perception of those surroundings as mediated by both automatic and controlled mental processes.

Presence

Presence is a concept describing the effect that people experience when they interact with a computer-mediated or computer-generated environment (Sheridan, 1992). Witmer and Singer (1994) defined presence as the subjective experience of being in one environment (there) when physically in another environment (here). Lombard and Ditton (1997) described presence as an “illusion of non-mediation” that occurs when a person fails to perceive or acknowledge the existence of a medium in his/her communication environment and responds as he/she would if the medium were not there. Although their definition confines to presence due to a medium, they explained how the concept of presence is derived from multiple fields – communication, computer science, psychology, science, engineering, philosophy, and the arts. Presence induced by computer applications or interactive simulations was believed to be what gave people the sensation of, as Sheridan called it, “being there.” But the studies on presence progressed with a slow realization of the fact that it’s more than just “being there.” We believe that presence, whether strong or mild is the result of an “experience of reality.”

In fact “presence” has come to have multiple meanings, and it is difficult to have any useful scientific discussion about it given this confusion (Slater, 2009). There can be no advancement simply because when people talk about presence they are often not talking about the same underlying concept at all. No one is “right” or “wrong” in this debate; they are simply not talking about the same things (Slater, 2003). On the general problems in conveying knowledge due to the intersection of the conceptual, material, and linguistic representations of the same thing, there exists an attempt to explain the workings of communication and its mishaps (Schmidt, 1997a,b2009), which clearly states that scientists must always indicate which representation they speak of. In this article, we are mainly speaking about the phenomenon, which is the experience of presence.

Reality

The term “reality” itself is very subjective and controversial. While objectivists may argue that reality is the state of things as they truly exist and is mind-independent, subjectivists would reason that reality is what we perceive to be real, and there is no underlying true reality that exists independently of perception. Naturalists argue that reality is exhausted by nature, containing nothing supernatural, and that the scientific method should be used to investigate all areas of reality, including the human spirit (Papineau, 2009). Similarly a physicalist idea is that the reality and nature of the actual world conforms to the condition of being physical (Stoljar, 2009). Reality is independent of anyone’s beliefs, linguistic practices, or conceptual schemes from a realist perspective (Miller, 2010). The Platonist view is that reality is abstract and non-spatiotemporal with objects entirely non-physical and non-mental (Balaguer, 2009). While some agree that the physical world is our reality, the Simulation Argument suggests that this perceivable world itself may be an illusion of a simulated reality (SR; Bostrom, 2003). Still others would endeavor to say that the notion of physical world is relative as our world is in constant evolution due to technological advancement; also because of numerous points of view on its acceptation (Schmidt, 2008). Resolving this confusion about theories on reality is not our primary aim and is however beyond the scope of this study. So we reserve the term “Primary Reality” to signify the reality of our real world experiences, which would be explained later in this paper.

The Illusion of Reality

The factors determining the experience of presence in a virtual environment have been explored by many in different ways. For example, presence due to media has previously been reviewed as a combination of:

• Perceptual immersion and psychological immersion (Biocca and Delaney, 1995Lombard and Ditton, 1997).

• Perceptual realism and social realism (Lombard and Ditton, 1997).

• Technology and human experience (Steuer, 19921995).

• Proto-presence, core-presence, and extended-presence (Waterworth and Waterworth, 2006).

• Place illusion and plausibility illusion (Slater, 2009).

To summarize, the two main factors that contribute to the illusion of reality due to media are (1) Perceptual Illusion: the continuous stream of sensory input from a media, and (2) Psychological Illusion: the continuous cognitive processes with respect to the perceptual input, responding almost exactly how the mind would have reacted in Primary Reality. Virtual reality systems create highest levels of illusion simply because it can affect more senses and help us experience the world as if we were inside it with continuous updated sensory input and the freedom to interact with virtual people or objects. However other forms of media, like a movie (where the sensory input is merely audio-visual and there is no means to interact with the reality presented) can still create a powerful illusion if it manages to create a stronger Psychological Illusion through its content (for example a story related to one’s culture or past experiences, would excite the memory and emotional aspects). One of the obvious examples illustrating the strength of Perceptual illusion is a media that enforces stereoscopic view enhancing our depth perception (the illusion works due to the way our visual perception would work otherwise, without a medium). The resultant of the two, Perceptual Illusion and Psychological Illusion evokes an illusion of reality in the mind, although subjectively varying for each person – in strength and experience.

The Concept of “Evoked Reality”

We know that it’s not directly presence that we create but rather an illusion in our minds as a result of which we experience presence. When we use virtual reality systems and create convincing illusions of reality in the minds of users, they feel present in it. This illusion of reality that we evoke through different means in order to enable the experience of presence is what we intend to call “Evoked Reality (ER).” To explore this experience of presence we must first better understand what ER is.

As deduced earlier, all the factors influencing presence would essentially be categorized as Perceptual Illusion and Psychological Illusion. We believe that every media in a way has these two basic elements. Thus ER is a combined illusion of Perceptual Illusion and Psychological Illusion. This combined spatiotemporal illusion is what evokes a different reality in our minds (Figure 1) inducing presence.FIGURE 1

Figure 1. Spatiotemporal illusion due to mediation: reality so evoked generates the experience of presence

Evoked Reality

Even though the terms like telepresence and virtual reality are very recent, their evidence can be traced back to ancient times. The urge to evoke reality different from our Primary Reality (real world reality) is not at all new and can be observed through the evolution of artistic and scientific media throughout history. “When anything new comes along, everyone, like a child discovering the world, thinks that they’ve invented it, but you scratch a little and you find a caveman scratching on a wall is creating virtual reality in a sense. What is new here is that more sophisticated instruments give you the power to do it more easily. Virtual Reality is dreams.” Morton Heilig. (as quoted in Hamit, 1993, p. 57).

From Caves to CAVEs

Since the beginning of civilizations, man has always tried to “express his feelings,” “convey an idea,” “tell a story” or just “communicate” through a number of different media. For example, the cave paintings and symbols that date back to prehistoric times may be considered as one of the earliest forms of media used to convey ideas. As technology progressed media evolved as well (Figure 2) and presently we are on the verge of extreme possibilities in mediation, thus equivalent mediated presence.FIGURE 2

Figure 2. Evolution of media: from caves to CAVEs

We all like to experience presence different from our everyday happenings. To do so, we basically find methods to create an illusion of reality different from the reality that we are familiar with. With the help of different media we have already succeeded to evoke a certain amount of presence and we further aim for an optimum level – almost similar to our real world. Every form of mediation evokes a different kind of illusory reality and hence different degrees of presence. In the early examples of research in presence, studies were conducted based on television experiences before Virtual Reality became a more prominent field of research (Hatada and Sakata, 1980). While some types of media evoke mild illusion of presence, highly advanced media like Virtual Reality may evoke stronger presence. “But we must note that the basic appeal of media still lies in the content, the storyline, the ideas, and emotions that are being communicated. We can be bored in VR and moved to tears by a book” (Ijsselsteijn, 2003). This is precisely why the reality evoked (by media) in one’s mind depends greatly on the eventual psychological illusion, although it may have been triggered initially by a perceptual illusion. Media that could evoke mild or strong presence may range from simple paintings to photos to televisions to films to interactive games to 3D IMAX films to simulation rides to immersive Virtual Reality systems.

Evoked Reality

Evoked Reality is an illusion of reality, different from our Primary Reality (Physical Reality as referred in previous studies). ER is a transient subjective reality created in our mind. In the case of ER due to media, the illusion persists until an uninterrupted input of perceptual stimuli (causing perceptual illusion) and simultaneous interactions (affecting the psychological illusion) continue to remain. The moment at which this illusion of ER breaks due to an anomaly is when we experience what is called a “Break in Presence (BIP)” (Slater and Steed, 2000Brogni et al., 2003). Thus a BIP is simply an immediate result of the “Break in Reality (BIR)” experienced. Different kinds of media can evoke realities of different qualities and different strengths in our minds for different amount of time. It’s an illusion of space or events, where or during which we experience a sense of presence. Thus, it is this ER in which one may experience Evoked Presence (EP).

Evoked Presence

Depending on the characteristics of ER, an experience of presence is evoked. To be more specific this illusion of presence created by ER, we would like to refer to as EP. In this paper, the term “EP” would imply the illusion of presence experience (the sense of presence), while the term “presence” would be reserved for experience of presence in its broad sense (real presence and the sense of presence). EP is the spatiotemporal experience of an ER. We could say that so far it’s through the media like highly immersive virtual reality systems, that we were able to create ER that could evoke significantly strong EP.

Media-Evoked Reality and Self-Evoked Reality

As we saw before, ER is a momentary and subjective reality created in our mind due to the Perceptual Illusion and Psychological Illusion imposed by a media. It is clear that due to ER induced through media like Virtual Reality we experience an EP. This illusion of reality evoked through media, we would like to call “Media-Evoked Reality” or Media-ER.

As mentioned earlier, it’s not just through the media that one can evoke an illusion of reality. The illusion can as well be endogenously created by our mind evoking a seemingly perceivable reality; whether merely observable or amazingly deformable; extremely detailed or highly abstract; simple and familiar or bizarrely uncanny. Thus to fully comprehend the nature of presence, we must study this category of ER that does not rely on media. In fact, we always or most often undergo different types of presence without mediation. Sanchez-Vives and Slater (2005) proposed that the concept of presence is sufficiently similar to consciousness and that it may help to transform research within domains outside Virtual Reality. They argue that presence is a phenomenon worthy of study by neuroscientists and may help toward the study of consciousness. As rightly put by Biocca (2003), where do dream states fit in the two pole model of presence (Reality-Virtuality Continuum)? The psychological mechanisms that generate presence in a dream state have to be at least slightly different than psychological mechanisms that generate presence in an immersive, 3D multimodal virtual environment. Dreaming, according to Revonsuo (1995) is an organized simulation of the perceptual world and is comparable to virtual reality. During dreaming, we experience a complex model of the world in which certain types of elements, when compared to waking life, are underrepresented whereas others are over represented (Revonsuo, 2000). According to LaBerge (1998), theories of consciousness that do not account for dreaming must be regarded as incomplete. LaBerge adds, “For example, the behaviorist assumption that ‘the brain is stimulated always and only from the outside by a sense organ process’ cannot explain dreams; likewise, for the assumption that consciousness is the direct or exclusive product of sensory input.” It is very clear that one can think, imagine, or dream to create a reality in his mind without the influence of any media whatsoever. This reality evoked endogenously, without the help of an external medium, we would like to call “Self-Evoked Reality” or Self-ER (implying that the reality evoked is initiated internally by the mind itself).

Ground-breaking works by Shepard and Metzler (1971) and Kosslyn (19801983) in the area of Mental Imagery provide empirical evidence of our ability to evoke images or imagine stimuli without actually perceiving them. We know that Perceptual and Psychological Illusion are factors that affect Media-ER and corresponding EP. We believe that Self-ER essentially has Psychological Illusion for which the Perceptual element is generated internally by our mind. By generally overlooking or occasionally completely overriding the external perceptual aspects (sensorimotor cues), our mind endogenously creates the Perceptual Illusion required for the ER. It’s evident in the case of dreaming which according to LaBerge (1998), can be viewed as the special case of perception without the constraints of external sensory input. Rechtschaffen and Buchignani (1992) suggest that the visual appearance of dreams is practically identical with that of the waking world. Moreover, Kosslyn’s (19942005) work show that there are considerable similarities between the neural mappings for imagined stimuli and perceived stimuli.

Similar to Media-ER, one may feel higher or lower levels of presence in Self-ER, depending on the reality evoked. A person dreaming at night may feel a stronger presence than a person who is daydreaming (perhaps about his first date) through an on-going lecture with higher possibilities of BIRs. According to Ramachandran and Hirstein (1997) we occasionally have a virtual reality simulation like scenario in the mind (although less vivid and generated from memory representations) in order to make appropriate decisions in the absence of the objects which normally provoke those qualities. However, the vividness, strength, and quality of this internally generated illusion may vary significantly from one person to another. For example, the intuitive “self-projection” phenomenon (Buckner and Carroll, 2007; personal internal mode of mental simulation, as they refer to it) that one undergoes for prospection will certainly differ in experience and qualia from another person. It is a form of Self-ER that may not be as strong or prolonged as a picturesque dream, but strong enough to visualize possible consequences. It is clear that ER is either the result of media or induced internally. This dual (self and media evoking) nature of ER directs us toward a fresh perceptive – three poles of reality.

Three Poles of Reality

As we move further into the concept of ER and EP, we would like to define the three poles of reality to be clearer and more objective in the explanations that follow. Reality, as discussed earlier (in subsection Simulated Reality), has always been a term interpreted with multiple meanings and theories. To avoid confusion we would like to use an impartial term – “Primary Reality,” which would refer to the “experience” of the real world (or what we call physical world). It is the spatiotemporal reality in our mind when we are completely present in the real world. It would mean that any reality other than Primary Reality is a conscious experience of illusion of reality (mediated or non-mediated), or more precisely – ER.

Presence and Poles of Reality

Inherited from early telerobotics and telepresence research, the two pole model of presence (Figure 3) suggests that presence shifts back and forth from physical space to virtual space. Research on presence has been dominated ever since by this standard two pole psychological model of presence which therefore requires no further explanation.FIGURE 3

Figure 3. The standard two pole model of presence

Biocca (2003) took the study of presence model one step further. According to the model he proposed, one’s spatial presence shifts between three poles of presence: mental imagery space, the virtual space, and the physical space. In this three pole graphic model, a quasi-triangular space defined by three poles represented the range of possible spatial mental models that are the specific locus of an individual user’s spatial presence. His Model of presence attempted to offer a parsimonious explanation for both the changing loci of presence and the mechanisms driving presence shifts. Though the model explained the possibilities of presence shifts and varying levels of presence, it is vague about certain aspects of reality. It did not clarify what happens when we experience an extremely low level of presence (at the center of the model). How or why do we instantly return to our Primary Reality (in this model – Physical Space) as soon as a mediated reality or a DR is disrupted (Even though we may have entirely believed to be present in the reality evoked during a vivid dream)? Moreover it took into account only the spatial aspects but not the temporal aspects of shifts in presence.

We would like to define three poles of reality from the perspective of ER. The Three Pole Reality Model (Figure 4) may help overcome the theoretical problems associated with presence in the standard two pole model of presence as well as the model proposed by Biocca. According to us it’s the shifts in the type of reality evoked that create respective shifts in the level of presence evoked. For example if one experiences a highly convincing ER during a virtual reality simulation, he/she would experience an equivalently strong EP until a BIR occurs. The three poles of reality that we define are:

• DR (Threshold of Self-ER)

• Primary Reality (No ER)

• SR (Threshold of Media-ER)FIGURE 4

Figure 4. Three pole reality model

Primary reality

Primary reality refers to the reality of our real world. In Primary reality, the experience evoking stimulation arrives at our sensory organs directly from objects from the real world. We maintain this as an ideal case in which the stimulus corresponds to the actual object and does not deceive or misinform us. For instance, imagine yourself running from a tiger that is chasing you. It’s very near and is about to pounce on you. You scream in fear, and wake up to realize that you are safe in your bed, like every morning. You know for sure that this is the real world and the chasing tiger was just a part of the DR that your mind was in, some time before. So, Primary Reality is our base reality to which we return when we are not in any ER. In other words, when a BIR occurs, we come back to Primary Reality. Thus, as we can see in Figure 5, any point of reality other than Primary Reality is an ER. We could say that it’s this Primary Reality that we rely on for our everyday activities. It’s the reality in which we believe that we live in. Our experiences in this Primary Reality may form the basis for our experiences and expectations in an ER. For example, our understanding of the real world could shape how we experience presence in an immersive virtual reality environment, or even in a Dream. We could suppose that it’s the Primary Reality in which one believes this paper exists, or is being read.FIGURE 5

Figure 5. Three poles of reality: evoked reality constantly shifts between them

Simulated reality

In the case of Media-ER, an experience similar to Primary Reality is attempted to be achieved by interfering with the stimulus field, leading to an illusion of reality. For example virtual reality uses displays that would entirely mediate our visual perception in a manner that our head or eye movements are tracked and updated with appropriate images to maintain this illusion of receiving particular visual stimuli from particular objects. SR would be the most compelling and plausible reality that could ever be achieved through such mediations. It would be the reality evoked in our mind under the influence of a perfectly simulated virtual reality system. It’s the ultimate level that virtual reality aims to reach someday. At the moment an immersive virtual reality system, like flight simulators would be able to create ER considerably close to this pole. Its effectiveness is evident in the fact that pilots are able to perfectly train themselves being in that ER created by the simulator, helping them eventually to directly pilot a real plane. However, in the hypothetical condition of a perfectly SR our mind would completely believe the reality evoked by the simulation medium, and have no knowledge of the parent Primary Reality (Putnam, 1982Bostrom, 2003). In this state, it would be necessary to force a BIR to bring our mind back to Primary Reality. A Perfect SR is the Media-ER with strongest presence evoked and will have no BIRs.

Dream reality

In the case of Self-ER, the external perceptual stimuli are imitated by generating them internally. DR is an ideal mental state in which we almost entirely believe in the reality experienced, and accept what is happening as real. It does not return to the Primary Reality unless a BIR occurs. For instance, in the case of our regular dreams, the most common BIR would be “waking up.” Although internally generated, dream states may not be completely divorced from sensorimotor cues. There can be leakage from physical space into the dream state (Biocca, 2003). The experienced EP during a strong Dream can be so powerful that even the possible anomalies (causing BIRs) like external noises (an alarm or phone ringing) or even elements from physical disturbances (blowing wind, temperature fluctuations) may be merged into the DR, so as to sustain this ER for as long as possible. A Perfect DR is a Self-ER with the strongest presence evoked and will have no BIRs (similar to SR on the media side).

Presence Shifts and Presence Threshold

We are often under the effect of either Media or Self-ER. Imagine that we are not influenced by any mediation, nor any kind of thoughts, mental imagery, or dreams and our mind is absolutely and only conscious about the Primary Reality. In such an exceptional situation we would supposedly feel complete presence in the Primary Reality. Thus we presume that this perfect Primary Reality-Presence (or “real presence” as some may call) is the threshold of presence one’s mind may be able to experience at a point of time. It is clear that we can experience presence either in Primary Reality or in an ER. We cannot consciously experience presence in two or more realities at the same time, but our mind can shift from one reality to another voluntarily or involuntarily, thus constantly shifting the nature and strength of the presence felt. As pointed out by Garau et al. (2008), presence is not a stable experience and varies temporally. They explain how even BIPs could be of varying intensities. They also try to illustrate using different presence graphs the phenomenon of shifting levels of presence with the course of time and how subjective the experience is for different participants. Media like virtual reality aims to achieve the Presence Threshold at which one’s mind might completely believe the reality evoked. Though we have not however achieved it, or may never do, theoretically it’s possible to reach such a level of SR. Similarly if one experiences a Perfect Dream without any BIR, he/she would be at this threshold of presence exactly like being in the Primary Reality. SR and DR are the two extreme poles of reality at which the EP is at its threshold. These presence shifts due to the shifting of reality between these poles is something that we seldom apprehend, although we always experience and constantly adapt to them. In the following section we attempt to represent this phenomenon with a schematic model that would help us examine presence and reality from a clearer perspective.

Reality-Presence Map

Based on the three poles of reality and Presence Threshold we would like to propose the Reality-Presence Map (Figure 6). This map is a diagram of the logical relations between the terms herein defined. At any point of time one’s mind would be under the influence of either a Media-ER or a Self-ER when not in the Primary Reality (with no ER at all). Between the poles of reality, ER would constantly shift evoking a corresponding presence EP. As we can see in the map there is always a sub-conscious Parent Reality-Presence corresponding to the EP. This Parent Reality-Presence is very important as it helps our mind to return to the Primary Reality once the illusion of ER discontinues (or a BIR occurs). For a weaker EP, the Parent Reality-Presence is stronger (although experienced sub-consciously). When the ER manages to evoke very strong presence, the strength of Parent Reality-Presence drops very low (almost unconscious) and we start to become unaware of the existence of a Primary Reality; which is what an excellent immersive virtual reality system does. The shifting of presence is closely related to our attention. As soon as our attention from the ER is disrupted (predominantly due to interfering external perceptual elements), our attention shifts to the parent reality-presence sliding us back to Primary Reality (thus breaking our EP).FIGURE 6

Figure 6. Reality-presence map.

At the extreme poles, we would experience an Optimum Virtual Presence in a SR and similarly an Optimum Dream Presence in a DR. At these extreme points one may completely believe in the illusion of reality experienced almost or exactly like it is our Primary Reality, without the knowledge of an existing Parent Reality. At such a point, possibly a very strong BIR should be forced to bring one back to the parent Primary Reality. Experiencing a strong DR is one such example which many would relate to. During a very compelling but frightening dream, “waking up” acts as a very strong BIR, helping in the desperate attempt to leave the DR. After such a sudden and shocking change in reality most often our mind takes time to adjust back to the Primary Reality where everything would slowly turn normal and comforting.

Whenever there is an ER, the EP part of the presence (in the map) is what has our primary attention, and thus is the conscious part. Hence, the higher the EP, the lesser we are aware of our parent reality. Evidence of the sub-conscious Parent Reality-Presence can be observed in our experience of any media that exists today. Many studies have shown that in virtual environments, although the users behaved as if experiencing the real world, at a sub-conscious level they were certain that it was indeed “not” real. BIPs (that are used to measure presence) are in fact triggered by shifts in attention from the virtual world to the real world. For instance, virtual reality systems that help visually surround us completely with a virtual environment, elevates our presence (compared to a panorama view or television with visible frame boundaries) as our chances of shifting attention toward the real world drastically reduce in such higher levels of immersion (Grau, 2004Slater, 2009). Since ER is a subjective feeling, it can never be measured or even compared truthfully. This is the reason why we depend on the measurement of presence EP to determine if a system creates a stronger or weaker ER. Since the strength of presence itself is relative, the best way to measure is to compare between systems in similar context. “The illusion of presence does not refer to the same qualia across different levels of immersion. The range of actions and responses that are possible are clearly bound to the sensorimotor contingencies set that defines a given level of immersion. It may, however, make sense to compare experience between systems that are in the same immersion equivalent class” (Slater, 2009).

A major task for empirical consciousness research is to find out the mechanisms which bind the experienced world into a coherent whole (Revonsuo, 1995). This map provides a framework where the various experiences of ER could be mapped. Note that this map is not a “graph” that shows the strength of EP as directly proportional to the strength of ER. In fact it would help us represent every possible kind of ER as a point fluctuating between the two extreme poles of reality, with its respective strength of EP. We may refer to ER as stronger or weaker, when its qualia evoke stronger or weaker EP respectively. The Reality-Presence Map shows that if we can skillfully manipulate these qualia of ER (although subjective to each individual) bringing it closer to either of the two extreme poles, we may be able to evoke higher levels of EP. We should also note that, in order to introduce its basic concept, the Reality-Presence Map is presented here in a flattened two-dimensional manner. In the later sections we will illustrate how this map attempts to account for different experiences which were unable to be explained by previous presence models.

Subjectivity of Evoked Reality

As a matter of fact, the same mediation can create different subjective ER for different users depending on their personal traits. For example, two users reading the same book, or playing the same video game, or using the same Virtual Reality system would experience presence in an entirely different manner. EP (especially evoked by a medium) may be affected by one’s knowledge related to the context, degree of interest, attention, concentration, involvement, engagement, willingness, acceptance, and emotional attributes making it a very subjective experience. This is precisely why it is difficult to evaluate the efficiency of a particular Virtual Reality system by means of presence questionnaires. In fact many researchers confuse few of these terms above, with the concept of presence.

Therefore, to locate ER on the map, we have to examine “presence.” In fact finding reliable ways to measure presence has been a pursuit among many virtual reality and communication media researchers. In order to lead to testable predictions, we would rely on currently evolving measuring and rating systems, so as to determine an objective scale for presence (from Primary Reality to each extreme pole). Presently existing measuring techniques include questionnaires like “presence questionnaire” (Witmer and Singer, 1998Usoh et al., 2000), ITC-SOPI questionnaire (Lessiter et al., 2001), SUS questionnaire (Slater et al., 19941995), analysis of BIPs (Slater and Steed, 2000Brogni et al., 2003), objective corroborative measures of presence like psycho-physiological measures, neural correlates, behavioral measures, task performance measures (Van Baren and Ijsselsteijn, 2004), to mention a few. We can certainly predict the positions of different everyday experiences for a person in general (Figure 7); however it could be tested in the future only using above mentioned methods of measuring presence.FIGURE 7

Figure 7. An example range of Media-ER and Self-ER experiences mapped on reality-presence map, for an individual, that would occur at various points in time.

In virtual reality, distinction between “presence” and “immersion” has been made very clear previously in (Slater, 19992003). Though immersion (which is discussed extensively in the domain of virtual reality) is one of the significant aspects of EP, it falls under the technical faculty of a mediated system. “Immersion (in perceptual sense) provides the boundaries within which Place Illusion can occur” (Slater, 2009). Detailed aspects of presence related to immersive virtual reality are also discussed in (Slater et al., 2009). The characteristics like involvement, engagement, degree of interest, emotional response, may seem similar to presence, but are in fact different elements that may influence or be influenced by EP. The psychological impact of content, i.e., good and bad, exciting and boring, depends to a large extent on the form in which it is represented (Ijsselsteijn, 2003). Thus one of the most important aspects of Media-ER is its context. In most cases it forms a reference in one’s mind to how they may experience ER and hence the presence evoked. For example, in some contexts, especially in art and entertainment, it would invoke a “genre” that plays a major role in its communication. The context (whether artistic expression, communication, entertainment, medical application, education, or research) should be a core concern while designing a Virtual Reality System, in order to bring about a subjectively higher quality of ER. A descriptive account on the importance of context in Self-ER is given by Baars (1988). With examples of different sources and types (perceptual and conceptual) of contexts, he demonstrates how unconscious contexts shape conscious experience. In addition, he explains the importance of attention, which acts as the control of access to consciousness. Attention (in both Media-ER and Self-ER) can direct the mind toward or away from a potential source of qualia. The experience of an ER therefore depends also on the voluntary and involuntary characteristics of one’s attention.

According to the concept, our presence shifts continuously from one ER to another and does not require passing through Primary Reality to move from one side to another. This map does not provide a temporal scale per se. However in future (with the advancements in presence measurement techniques), the map can be used to trace presence at different times to study the temporal aspects of presence shifts.

Evoked Reality within Evoked Reality

There is an important question that arises now. How can we account for our thoughts or mental imagery experiences during VR simulations, games, movies, or most importantly books? It is the phenomena of experiencing Self-ER during a Media-ER experience.

Self-ER within media-ER

Whenever we experience an ER, our mind is capable of temporarily presuming it as the parent reality and reacting accordingly. The better the ER and stronger the EP, the easier it is for our mind to maintain the illusion. In such states Media-ER is experienced as a temporarily form of Primary Reality, and we are able to experience Self-ER within it. In fact that is the core reason why virtual reality systems and virtual environments work. This phenomenon is clearly displayed in such experiences, where the users require thinking, planning, and imagination in order to navigate in the virtual world, just like they would do in the real world. Below, it is demonstrated how this phenomenon may be represented with respect to the Reality-Presence Map (Figures 8 and 9). This scenario will ultimately be classified under Media-ER.FIGURE 8

Figure 8. An example of how Media-ER would temporarily act as a version of primary reality

Figure 9. An example of presence shift due to Self-ER within Media-ER (for e.g., thinking within a virtual environment).

Self-ER triggered during media-ER

“Self-ER within Media-ER” should be distinguished from the phenomenon of “Self-ER triggered during Media-ER.” This is similar to a well-known case of Self-ER – the phenomenon of mind-wandering that temporarily detaches us from the Primary Reality. It is otherwise known as “task unrelated thought,” especially with respect to laboratory conditions. Smallwood et al. (2003) define it as the experience of thoughts directed away from the current situation. It is in fact a part of (and closely related to) our daily life experiences (Smallwood et al., 2004McVay et al., 2009). Although studies on mind-wandering are principally focused on shifts between Self-ER and tasks relating to Primary Reality (falling under usual case of Self-ER experience – Figure 10), we propose that they are applicable to similar cases in Media-ER as well. It has been suggested that this involuntary experience may be both stable and a transient state. That means we can experience a stable EP during mind-wandering or an EP oscillating between the Self-ER, Media-ER, and the Primary Reality.FIGURE 10

Figure 10. The usual case of presence shift from primary reality to Self-ER

Therefore, when an unrelated Self-ER is triggered while experiencing a Media-ER (or when Self-ER within Media-ER traverse the presence threshold and becomes unaware of the Media-ER itself), it should be considered under the case of Self-ER (Figure 11).FIGURE 11

Figure 11. An example of presence shift toward Self-ER triggered during Media-ER.

Discussion

Our attempt was a novel idea, to fit together different concepts regarding presence into a single coherent graphical representation. Although this concept of ER and EP along with the proposed map provides us a simplified way to look at reality and presence, it raises plenty of questions. Can the experience of an altered state of consciousness (ASC) like hallucination, delusion, or psychosis due to mental disorders be a kind of Self-ER? Revonsuo et al. (2009) redefines ASC, as the state in which consciousness relates itself differently to the world, in a way that involves widespread misrepresentations of the world and/or the self. They suggest that, to be in an ASC is to deviate from the natural (world-consciousness) relation in such a way that the world and/or self tend to be misrepresented (as evident in reversible states like dreaming, psychotic episodes, psychedelic drug experiences, epileptic seizures, and hypnosis). According to Ramachandran and Hirstein (1997) we have internal mental simulations in the mind using less vivid perceptual attributes, in the absence of the regular external sensory inputs. If they possessed full-strength perceptual quality, that would become dangerous leading to hallucinations. They argue that in cases like temporal lobe seizures, this illusion (Self-ER) may become indistinguishable to real sensory input losing its revocability and generating incorrect sense of reality (creating a permanent ER situation that makes it difficult to return to Primary Reality). So can hallucinations due to Self-ER be compared to Augmented Reality due to Media-ER?

In contrast to Presence, is there an “Absence” and do we experience that? If so, how? Can it be compared to a dreamless sleep? Can Presence Threshold itself be subjective and differ from person to person? With reference to the Reality-Presence Map, is there a possibility of an experience analogous to uncanny valley when ER is nearest to the two extreme poles? Is this the reason why many experience anomalies during exceptionally vivid nightmares or lucid dreams? Similarly on the Media-ER side, can simulator sickness due to inconsistencies during virtual reality simulations be compared to this phenomenon? Other than the obvious difference between Media-ER and Self-ER that was discussed before, they have another main differentiation. In most cases of Media-ER, multiple users could share the experience of a common ER at the same time (naturally, with subjective differences, especially due to psychological illusion). While in the case of Self-ER, every person’s mind experiences unique ER. Thus a Dream is typically an individual experience (as far as our present technological advancements and constraints suggest), while SR may be shared.

Furthermore, the Reality-Presence Map helps us investigate into potential ideas on Reality, for instance the possibility of Simulation within a Simulation (SWAS). The Map could be extended to and be applicable for any level of reality, in which we believe there’s a Primary Reality – the base reality, to which we return to in case of absence of any form of ER. Let’s imagine that someday we achieve a perfect SR. As per our proposition, one’s mind would accept it as the Primary Reality as long as the experience of presence continues (or till a “BIR” occurs). It would imply that at such a point, one can experience presence exactly as in the Primary Reality. In this perfect SR if one experiences Media-ER (e.g., virtual reality) or Self-ER (e.g., dream), as soon a BIR occurs they return back to it since it’s the immediate Parent Reality. Figure 12 attempts to illustrate such a situation with DR and SR as two orthogonal Poles of Reality. Similarly in the Self-ER side, one’s mind could experience a Dream within a Dream (DWAD). When one wakes up from such a dream, he could find himself in the parent DR from which he would have to wake up again into the Primary Reality. Can this be how people experience such false awakenings [a hallucinatory state distinct from waking experience (Green and McCreery, 1994)]? Figure 13 attempts to illustrate such a situation of DWAD.FIGURE 12

Figure 12. Simulation within a simulation

Figure 13. Dream within a dream

In fact it makes us curious about the even bigger questions. Can there be an ultimate reality beyond Primary Reality or even beyond the scope of this map. The Simulation argument claims that we are almost certainly living in a computer simulation (Bostrom, 2003), in which case what we believe to be our Primary Reality might itself be a SR [similar to Brains in a vat scenario (Putnam, 1982)]. Metzinger (2009) proposes that our experience of the Primary Reality is deceptive and that we experience only a small fraction of what actually exists out there. He suggests that no such thing as “self” exists and the subjective experience is due to the way our consciousness organizes the information about outside world, forming a knowledge of self in the first person. He claims that everything we experience is in fact a SR and the on-going process of conscious experience is not so much an image of reality as an “ego tunnel” through reality. So, is our Primary Reality in fact the base reality? Or are we always under an ER of some kind? Figure 14 attempts to put together different levels of reality as a Reality Continuum. It would make us wonder if it’s probable, to how many levels would one be able to go? Do we already visit them unknowingly through our dreams? Would the levels of reality in the figure be represented as a never ending fractal structure? In any case, will we be able to understand someday all these aspects of our experience of reality?FIGURE 14

Figure 14. Reality continuum (illustrating the levels of reality).

Conclusion

In this paper we explored presence and different elements that contribute to it. Presence is not just “being there” but a combination of multiple feelings and most importantly “experiencing the reality.” The two main factors affecting presence due to mediation are Perceptual Illusion and Psychological Illusion. These factors evoke an illusion of reality in our mind in which we feel presence. We are constantly subjected to such illusions of reality, during which we experience presence differently from that of our apparent real world. This illusion of reality is called ER.

Evoked Reality is not just media-evoked but can also be self-evoked. Media-ER may range from the mild effect of a painting to an extremely plausible immersive Virtual Reality experience while a Self-ER may range from a simple thought to an exceptionally believable DR (the strength of ER may not necessarily be in the same order, as it depends on one’s qualia and personal characteristics). This dual nature of ER led us to define three poles of reality: primary reality – the unaltered and unmediated Real World, SR – the ultimate Media-ER (a perfect Virtual Reality condition) and DR – the ultimate Self-ER (a perfect dream condition). Thus ER is an illusion of reality formed in our mind, which is different from Primary Reality. It’s a combined illusion of space and events, or at least one of them. It is in this ER, one would experience presence. Thus EP is the spatiotemporal experience of an ER.

The proposed Reality-Presence Map attempts to graphically illustrate the concept of ER and EP. This map provides a framework where the various experiences of ER could be mapped. The subjectivity of ER qualia and how these subjective factors affect Media-ER and EP were explained. The idea of Presence Threshold was also explored which formed the basis for different levels of EP and temporal Presence Shifts. Different possibilities like SWAS and DWAD conditions were discussed with respect to the proposed model. However certain elements still demand clarifications to fill in the theory. The concept presented here is an inception of a potential future research. We believe that ER and the proposed Reality-Presence Map could have significant applications in the study of presence and most importantly in exploring the possibilities of what we call “reality.”

The full report including references can be found here

Immersive & Interactive Audio for Extended Reality

Extended Reality (XR) is an umbrella term that embraces the concepts of virtual/augmented/mixed reality. Today XR technologies are being increasingly adopted in various fields, such as entertainment, health, social network, education/training, marketing and tourism. The common aim of XR applications is to provide the technology user with an immersive and interactive experience, i.e., sense of being in a virtual or augmented environment, interacting with virtual beings or objects. To allow for a high level of multi-sensory immersive experience, it is crucial for XR technologies to produce auditory cues necessary for accurate sound localisation and the perception of various spatial attributes, as well as realistic visual and haptic cues. Furthermore, sensorimotor contingencies afforded by XR technologies (e.g. head and motion tracking-based controls) allow for a realistic interaction with the virtual environment and beings, which would lead to a more immersive experience.

The key is to explore the recent developments in immersive and interactive audio technologies for XR applications, and to provide new insights into future research to advance this field. Although the last decade saw a great advance in three-dimensional (3D) audio technologies for multichannel and binaural recording, reproduction and transmission, there still exist various challenges specific to XR contexts that need to be overcome through further research, e.g. natural and efficient rendering of dynamic cues for 6-degrees-of-freedom (6DoF) experience, individualisation and optimisation of head-related transfer functions, sound field capture and post-processing techniques, efficient virtual acoustic modeling, etc. Furthermore, rapid advances in artificial intelligence, human-computer interaction, internet of things, and cloud and edge processing technologies open up new opportunities for the development of new immersive and interactive audio technologies. In all of these, trade-off between perceived accuracy/quality and computational efficiency need to be considered, and therefore more research into perceptual-model-based approaches might be required.

Audio & acoustic sinal processing equipment for XR applications:
• Ambisonics
• Auditory-visual interaction
• Audio-haptic feedback
• Binaural synthesis techniques
• HRTF individualisation and optimisation
• Holographic auditory display
• Immersive audio using Internet-of-Things
• Machine learning-based approaches
• Microphone array techniques
• Objective metrics for auditory immersiveness measurements
• Six-degrees-of-freedom audio capture and processing
• Sound field synthesis methods
• Virtual acoustic simulation techniques

Audio is fully built into immersive Holospatial environments, but what is interesting in that taking a more targetted approach focused on the sensory system we can really understand the depth of impact that such an approach can have, and whilst most people think of audio as just noise, the variations in these technologies, peripheral virtual technologies and multi-frequency devices enables you to create a well-rounded sensory experience.

This is particularly important when it comes to devising applications, tools, simulations & training programs because audio is a far more powerful tool than what we are used to understanding- so when planning your immersive projection as well as other virtual XR technologies, remember that the ultimate goal is sensory immersion, it's not about a novelty, it's about being able to naturally interact.

Keywords: 3D Audio Technologies, Immersive Audio, Interactive Audio, Mixed Reality, Virtual Reality, Augmented Reality, Virtual Acoustics

This research topic is hosted on www.frontiersin.org where further information, reading and subject materials can be found.

Virtual Reality Rooms & Immersing Autism

Initially thought of as a novelty entertainment platform, virtual reality has recently emerged as an accessible, affordable technology for a variety of use purposes, including education and training. Researchers have also highlighted that it may offer a way to improve the lives of people with disabilities, especially autism.

The National Autistic Society estimates that around 700,000 people in the UK are on the autistic spectrum; that is almost one in 100. It’s a condition that can make a social, multi-sensory world as ours extremely difficult to navigate. While autism is impossible to cure, we know that certain types of therapy can help people on the spectrum overcome their challenges, and virtual reality (VR) has shown early promise in being one of these, in several different ways!

Immersive Learning

Spatial reality environments can be used as a tool to help autistic people improve their social and communication skills. One research study has explored the effects of a VR learning environment, teaching social skills to children aged 10-14. The findings of the study have been extremely promising, with the children having demonstrated improved performance in social tasks thanks to the application.

VR Therapy

VR projections don’t just serve to educate: they can also serve as excellent preparation  environments. Some people on the spectrum take comfort in a regular routine; any unexpected change or disruption to this could be a huge trigger for stress. Spatial reality applications could be used to help people prepare for unfamiliar situations, by giving them a safe, controlled atmosphere where they can learn what to expect and how to respond.

Spatial Sensory Environments

Many autistic people have difficulty processing sensory information. For example, they may be extremely sensitive to loud noise, or fixated by lights of a particular colour.

Sensory rooms (also known as ‘snoezelen’) Not only are they often calming spaces for individuals to relax in, but they also provide a space where one can develop and engage their senses in a controlled environment. However, they can be rather expensive to set up and run, thanks to all the specialist equipment and electricity they require!Given that autism is a spectrum, not all individuals with the condition would want the same solutions: an object that one autistic person may find pleasant could completely stress out another! Fortunately, just like a physical sensory room, a virtual spatial snoezelen could easily be adapted for the individual’s needs.

Explaining Autism Itself

Since many of its challenges are the result of perceiving the world differently, autism is a disability that can be tricky to explain or understand unless you have it yourself. Immersive films and applications have proven to be a useful medium for illustrating how the everyday world seems to an autistic person.

For example, in 2017, The Guardian launched ‘The Party’, a short 360-degree film told from the perspective of an autistic teenager as she slowly becomes overwhelmed by the hustle and bustle of her mother’s birthday celebration. We believe it does an amazing job of communicating how situations that are mundane or pleasurable for most people can be unbearable for someone with autism. Maybe a VR version of a video like this could feel even more impactful.

https://www.youtube.com/watch?v=OtwOz1GVkDg


A standard VR headset can be rather bulky to wear. For an autistic person with sensory issues (amongst many other people), this could make for a rather uncomfortable experience. Here at Holospatial, however, we develop solutions offering all the eye-popping visuals without the need for a headset, offering a much more comfortable virtual adventure. If you work with autistic people and are tempted by the idea of providing them with comfortable and enjoyable virtual spaces, feel free to check out our low-cost, accessible solutions.


Delicious Data: Will We Ever Taste Our Computer Applications?

Just imagine you are in a restaurant. A virtual restaurant. You feel as if you are really sitting at a table on a chair. You hear the chatter and plate clattering of your nearby diners. You see the posh lighting, finely decorated tables and velvety restaurant wallpaper. You pick up virtual menus and look through what you fancy eating. Maybe you can even smell the ambience of all that delicious food being prepared from in the chefs’ kitchen. But then, your meal arrives. It looks convincing and appetising, and you pick up your digital knife and fork to cut it up and try it...but it’s bland. There’s no mouth feel, no tingling taste buds...no eating experience.

The sense of taste is one which has often been overlooked in the world of technology. As graphic displays, audio speakers and even haptic devices have all advanced over the years, the idea of gustatory devices has been left in the dust. Not because people aren’t interested in making digital taste happen, but rather that a lot more research needs to be done to understand how we can make it a reality. The sense of taste is not quite as straightforward a sense as you may expect...

How does taste work?

Of course, one of the secrets to how we can appreciate the different flavours of what we eat is in the tens of thousands of taste buds in our tongue. Within each taste bud is a cluster of taste receptor cells.  There are actually only five taste sensations the tongue can experience: sweetness, saltiness, sourness, bitterness and umami/savouriness, which are triggered by the detection of sugar, sodium chloride, acids, alkaloids and glutamates respectively.

But how is it that can we taste the difference between lemon juice and vinegar if both are sour? That’s where your nose comes into play, as the sense of smell and taste are actually very closely linked. As you eat or drink, the chemicals in what you consume trigger the olfactory receptors in your nasal cavity as well as your taste buds, and it is the combination of these responses that signal flavours to your brain. Next time you eat something tasty, hold your nose while you put it in your mouth and chew...and it’s all but guaranteed you won't have the gustatory explosion you would have felt in your mouth otherwise.

What will taste tech look like?

The world of taste-inducing technology is still in the conception stage, let alone its infancy. Due to the complex, chemical nature of our sense of taste, thinking of ways to simulate human taste sensations artificially and safely have proven difficult. No product or standard for taste-simulating technology currently exists in the market, and it’s currently impossible to break down smells into categories or elements. However, a few researchers have come up with some potential ideas as to how taste can work...

Tokyo University’s Takuji Narumi drew attention to the close link between the senses of smell and taste, and explored how exposure to different visual and olfactory stimuli could affect how we taste. At a computing conference in Canada in 2011, he demonstrated the Meta Cookie system, a head-mounted visual and olfactory display which was worn while eating an unflavoured biscuit. The headset’s display laid an image of a different-coloured biscuit over the original using augmented reality, as a perfume scent travelled through tubes attached to the nose.

In 2012, another team of researchers at the National University of Singapore, led by Nimesha Ranasinghe, explored how technology could tantalise the taste buds directly. They developed an experimental tongue-mounted device which aimed to replicate rudimentary taste sensations via controlled electrical stimulation. The results of this experiment found that the device was most effective at replicating sour taste sensations, and was capable of doing so in three degrees of intensity.

Six years later, another team of researchers proposed a similar taste-actuating interface of their own, this time activating the taste buds by changing temperature. This method could stimulate sweetness much better than any other taste, indicating that different taste sensations may require different strategies.

Where will we see taste tech used?

Imagine a culinary arts training simulation where you can actually learn what flavours to look out for when sampling your cooking. Or a virtual marketing campaign where you can sample beverages without physically having to drink them (want to taste a wine but need to drive back home?). Or maybe a virtual travel experience where you can actually get a taster for another country’s cuisine, all within one of our Portals perhaps!

Integrating the sense of taste into technology is still ages away from being a possibility, but it’s nonetheless exciting to think about the doors it’ll open up for what kind of immersive applications we can create.

Citations

Karunanayaka, K., Johari, N., Hariri, S., Camelia, H., Bielawski, K.S. & Cheok, A.D. (2018). New Thermal Taste Actuation Technology for Future Multisensory Virtual Reality and Internet. IEEE Transactions on Visualization and Computer Graphics. 24 (4). pp. 1496–1505.

Narumi, T., Nishizaka, S., Kajinami, T., Tanikawa, T. & Hirose, M. (2011). Augmented reality flavors: Gustatory display based on Edible Marker and cross-modal interaction. Conference on Human Factors in Computing Systems - Proceedings.

Ranasinghe, N., Nakatsu, R., Nii, H. & Gopalakrishnakone, P. (2012). Tongue Mounted Interface for Digitally Actuating the Sense of Taste. In: 2012 16th International Symposium on Wearable Computers. [Online]. June 2012, Newcastle, United Kingdom: IEEE, pp. 80–87. Available from: http://ieeexplore.ieee.org/document/6246147/. [Accessed: 1 November 2020].

Stereoscopy In Projection Environments- Is It The Future Or The Past?

A quick timeline

1838 - Stereoscopes: the very first 3D viewer

In 1838, English inventor Charles Wheatstone made the groundbreaking discovery that would kickstart this entire branch of technology in the first place:

“The mind perceives an object of three dimensions by means of the two dissimilar pictures projected by it on the two retinae.”

In other words, each of our two eyes views the same object from a slightly different position and angle. Our brain, however, combines both signals into a singular picture, which is where our sense of depth perception stems from.

To demonstrate this, Wheatstone developed the preliminary stereoscope: a device consisting of a pair of mirrors positioned at 45 degrees to the user’s eyes. Each mirror reflected an identical drawing positioned off to the side and parallel from each other, creating an illusion of one singular image being displayed rather than two separate ones.

The user would find that the reflected image in the mirror had more depth and volume than the original drawing alone.

1849 - Lenticular stereoscope

It wasn’t too long before David Brewster, a scientist who specialised in optics, took note of Wheatstone’s observations and set out to improve upon the original 3D viewing device. He proposed what he termed the ‘lenticular stereoscope’ in 1849. Dubbed the world’s first portable 3D viewer, this model used two lenses in lieu of mirrors, and the user would look through the device as they would a pair of binoculars.

1853 - Anaglyph 3D (a.k.a the retro cinema glasses)

When you think of 3D glasses, the first ones that come to mind are those paper ones with the colourful lenses, aren’t they?

Anaglyphic photographs and films display both eyes’ views on the same image simultaneously: they are positioned slightly apart and encoded by colour (usually red and cyan). When the viewer puts on those retro glasses to view this funky film, each of the cellophane lenses filters out its own colour so the eye gets to view its own designated picture.

Super anaglyph

One of the biggest disadvantages of the anaglyph format is that the resulting image isn’t always faithful to the original. Because the coloured lenses filter out particular hues, an anaglyph photograph may not have the full range of colours found in the original. Some companies have acknowledged this flaw and built upon the anaglyph idea by developing more sophisticated methods for separating the left and right visual channels.

Dolby 3D uses what’s known as an interference filter system (a.k.a. “super anaglyph”); rather than separate the left and right eye images by one single colour, both images are in full colour albeit at slightly different wavelengths (not different enough for the naked eye to tell apart). The lenses in the special glasses separate the display into bands of colour (red, green and blue), only letting through the picture meant for the individual eye. That way, the Dolby 3D system gives viewers the best of both worlds: eye-popping pictures in eye-popping colour!

1890 - Polarisation: light angles are the new colours

First demonstrated in the 1890s, the idea of polarised 3D works very similarly to anaglyph 3D: the images for each eye are positioned on the same display, differentiated by a filter and viewed through special glasses. The key difference is that while an anaglyph display uses colour, a polarised 3D uses, you guessed it, polarisation.

But what exactly is polarisation?

Light travels in the form of electromagnetic waves. Most of the time, the waves that make up the light you see travel in all angles. When light is polarised, however, it means it’s filtered so that only waves travelling at a particular angle pass through.

Polarised 3D takes advantage of this to offer audiences the three-dimensional illusion without any impact on colour. The two eyes’ images are projected onto the same screen, albeit polarised in opposite directions. The viewer puts on glasses that contain polarised filters in the lenses; each lens lets in the light polarised in one direction and blocks out all light polarised in the other.

Because of its low cost and minimal impact on picture quality, this is nowadays the most common type of 3D system used in cinemas.

1922 - Active 3D

Active 3D is when the glasses or lenses used to view are ‘active’ in the sense of being powered, rather than just serve as mere panels to look through. Though in the world of electronic media it’s a relatively modern technique, the concept itself dates back to 1922.

Laurens Hammond, best known for inventing the Hammond organ, devised a then-novel lens for viewing cinema films in 3D: the Teleview. As two projectors showed each eye’s films with the shutters out of sync, the audience would look through a viewing device attached to their seat. This device would block out the appropriate eye, thanks to a rotary shutter operating in sync with the projectors. The experimental system was only used in one cinema viewing, because it was expensive to implement

Only one of both eye’s images are presented on the screen at once, and the glasses are constantly ‘winking’: the left eye’s view is blocked when the screen displays a picture for the left eye and vice versa. The winking is controlled via a timing signal synchronised with the screen’s refresh rate.

The frame rate of a film projected through an active 3D display has to be double that of a standard one. Another disadvantage of the active 3D system is that it doesn’t work with most modern monitors due to the extremely high refresh rate required to view the picture without any flickering.

1985 - Autostereoscopy: glasses not required

The term ‘autostereoscopy’ doesn’t just refer to a single method: it’s an umbrella term covering all stereoscopic display types that don’t require glasses or lenses to view.

Perhaps one of the most well-known examples of this form of stereoscopy is the Nintendo 3DS. The 3D effect in the top screen works via a parallax barrier, an array of strips placed over the LCD screen which project the display at just the right angles for giving the player a sense of depth perception. The parallax barriers can be disabled if the player wants to game without the 3D effect.

The Power Behind Immersive Projection For Human Experience

Immersion is at the heart of our ability to enjoy the most engaging of media and technology. It’s when, without even trying, you focus your attention away from your physical environment and into the world of a whole new experience.

With more advanced media displays, we can now experience not just a high level of immersion, but also what’s known as suspension of disbelief. It sounds complex, but really it’s just the human ability to accept things that don’t necessarily make sense in order to experience something enjoyable or meaningful.

But how does all this magic really happen? What exactly is going on behind the scenes? Let’s break it down to understand the secrets of a perfect immersive experience.

Different flavours of immersive projection

To understand the ins and outs of immersion, we actually need to take a good look at the concept as a whole and break it down; there is no one size fits all explanation for what makes the content we consume feel as engaging and exciting as it does.

Perhaps one of the most rudimentary ways to look at it is to refer to subtypes of immersion defined by game design consultant Ernest Adams:

Tactical 360 immersion

Ever watched a ping pong match and noticed the sheer amount of concentration and determination painted across each player’s face, as they pace back and forth firing back the feather-light ball as it zooms across the table? This is what tactical immersion looks like; it’s that feeling of putting all your energy into your physical skills and reflex.

Many of the most thrilling video game genres rely on this type of immersion for optimal enjoyment, from intensive fighters and first-person shooters, to fast-moving platformers and rhythm games.

Strategic/cognitive immersive projection

Strategic immersion is for your brainpower what tactical immersion is for your actions and reactions. It’s that ‘in the zone’ feeling you get where you are so dedicated to thinking something through, solving a tricky puzzle or making the right decision.

This is the type of trance chess players enter when they’re thinking of exactly the right move to make; they need to prepare for how they can defeat their opponent and how they can protect themselves from being overthrown, so they have to make each move count.

Narrative/emotional immersion

Imagine you’re using your laptop late at night, streaming your favourite box set series online. You tell yourself, “Right, I’m going to limit myself to just one episode and log off at midnight”...you finish episode one, but it was so good, you just need to know what happens next. You keep trying to stop yourself from watching anymore, but every time the end credits roll, it gets harder and harder to resist as you binge your way through the series until before you know it, you check your clock...it’s 3am.

If this has ever happened to you, then congratulations: you are highly susceptible to falling into a state of narrative immersion. Whether it’s in a film, game or book, storytelling is a powerful tool, and if used correctly can suck your mind into a fantastic world that’s hard to break out of.


However, these three definitions alone barely scratch the surface; they are only types of low-level immersion. In other words, they explain scenarios where your concentration feels like it’s drifted away from the real world, but nothing else. Your senses and spatial awareness aren’t involved in this immersion, nor is your mind actually tricked by what’s going on around you.

With today’s technology, we can actually instil immersion of a much higher plane. And that’s why game computing scholars Staffan Björk and Jussi Holopainen,

Spatial immersion

This is where the excitement of virtual worlds comes into play! Spatial immersion is established when a media display goes beyond acting as...just that, and forms the foundations of a whole new environment to explore, marvel at and even interact with. When this type of immersion comes into play, the audience feels a strong sense of presence in a place they’re not actually in.

Of course, spatial immersion isn’t exclusive to virtual and augmented reality (e.g. have you ever felt your stomach tighten as you cross a precarious bridge in a 3D action adventure game?), but headsets, AR apps and portals take it one step further than a traditional media display could.

Psychological immersion

If a virtual environment is convincing and powerfully crafted enough (or perhaps a little too much so depending on how you look at it!), the audience may go past the stage of suspension of disbelief...to complete eradication of disbelief.

This means there should be nothing in the environment that makes the digital aspect of it too obvious: no distractions like weighty headsets, mic booms or intrusive bulky equipment or  wires lying around everywhere. If everything looks, feels and sounds convincing enough, your ability to understand and interpret your surroundings as intended becomes effortless, and you may for a moment actually forget that everything they are interacting with is, in fact, virtual!

Sensory immersion

We often take our senses for granted, but they perhaps have one of the largest roles to play when it comes to our ability to appreciate high-level immersion!

We have two eyes to see around and behind us in an immersive world, two ears to determine distance and origin of aural objects thanks to spatial sound, fingers to interact with controls and virtual objects, and a sense of balance and temperature to make sense of our physical presence in the world.

Of course, we’re not forgetting our nose and mouth for smelling and tasting, but Smell and taste technologies are still in their very early infancy yet...when they eventually blossom, however, we are confident they could make a heap of difference to your application!

Memory

The more immersive an application is, the better we remember it. This isn’t just our excitement talking here; many researchers have backed this claim up. And when we delve into how the human memory system works, it all makes sense.

Our memory has three main stages: long-term, short-term and sensory. Sensory memory is the first and most immediate of these, with compartments known as registers picking up and beginning to make sense of the information around us. Each of our six senses has its own dedicated sensory register:

Because an immersive application typically delivers more complex sensory information taken in effortlessly than a conventional content display, the sensory registers take in a lot more at once. The links between all the registers’ tidbits of information are very strong, so when everything you’ve taken in travels into short-term memory, a lot of it carries on over to long-term memory and sticks.

Everything in moderation

So, with all that in mind, does this mean you have to cram in as many sights, sound effects, sensations and funky smells in your application as possible? Well, you’d definitely succeed in giving your audience an experience they’ll remember for a lifetime...but not one they’d look back at too fondly.

You see, the key to optimal immersion is to fill the brain with as much information as possible without overwhelming it. If there isn’t enough happening, the experience may not feel as authentic for your audience and they may become bored. But a full-on sensory assault of flashy colours and noxious noises could result in stress, frustration and even nausea. In fact, you may also end up bringing on the opposite effect of what you wanted: too much going on at once may break the suspension of disbelief, completely destroying the sense of immersion you set out to sustain.

With all that in mind, the world is your oyster! With our portals, there is nothing stopping you from pushing the boundaries of immersion even further and offering an experience your customers will never forget...for all the right reasons, of course!

Project Immersion 101

“Virtual reality”, “augmented reality”, “immersive tech”...the landscape of electronic media is becoming more advanced and interactive everyday, and as such most of us have heard some of these terms used all the time, but what do these mean?

What is immersion?

Think about how you watch your favourite films, play your favourite games and binge through your favourite box sets. Your eyes are glued to the screen as you sit at the very edge of your seat. You feel so engrossed into the characters and their lives and stories. But you are only a spectator, looking into their world through a digital, two-dimensional lens.

Immersion is what happens when technology can deceive your mind into believing everything you are sensing is actually real and physically in your reach, rather than just a digital recording or animation.

Project virtual reality

This is probably the term you hear the most in discussions revolving around immersive technology. Virtual reality (VR). Many people immediately think of applications experienced through a head-mounted display such as the Oculus Rift. However, the term “virtual reality” also includes mobile applications and online applications recorded with a 360° panoramic view.

Augmented reality

Augmented reality allows us to see and interact with virtual objects through a digital window into our real world. The most widely seen example of augmented reality is mobile apps where users can view virtual objects augmented onto live real-world footage capture.

Many mobile games such as the successful Pokemon Go are powered by augmented reality, while cosmetics brands have also used the technology to let potential customers try on their products virtually before buying them physically.

Project spatial reality

Virtual and augmented reality applications, no matter how convincing or real they may feel, still function more like interactive lenses than virtual worlds when you look at how they work. But what if you want a technology that truly transports your users into a whole new virtual environment?

This is where spatial reality comes in (our speciality!). It's a whole new way to explore the immersive world of VR and interact with virtual objects like those in AR, all without a headset or any other ‘window’ to look through. Instead, you step into a portal with an all-encompassing environment, taking you on a journey and offering you a mesmerising experience.

Here at Holospatial, we believe spatial reality solutions are the future of immersive technology and how we consume our favourite content. That is why we offer a range of bespoke portal packages to suit you, offering your audience the opportunity to go on a captivating virtual journey like no other.

The Impact of Immersive Projection On Active Gaming & Immersing eSport

As the appetite for immersive content grows, video gaming will be at the forefront and so perhaps will 360 projection and visualisation techniques. It’s estimated that the world is home to almost two billion gamers, with this figure only set to further skyrocket in time to come. But gone are the days where gaming is a mere unproductive hobby; the global industry is worth almost $85 billion (£60 billion), and an industry worth over $900 million has accumulated in esports alone.

Almost a billion of us tune into gaming tournaments, where we watch players across the world battle it out on many titles for trophies and top cash prizes.

Immersive technologies unlock further potential for esports to flourish further and offer a more enriching experience to both contestants and spectators.

Put the ‘sport’ in ‘esports’ with 360 projection

Who says esports tournaments always have to be done with the players all sitting down in chairs? Immersive spaces allow users to move physically around a virtual space, and support gesture control and motion tracking.

Not only does this pave the way for the creation of even more captivating games relying on physical movement as well as strategy and skill, but it also carries the potential to revolutionise the entire world of esports, by making it more like a physically active sport in its own right.

A greater immersion creates a greater depth of field

With their environment all around them rather than merely presented through a two-dimensional screen, players can get a greater sense of where they are in the game's virtual world. They can even see in-game objects from any perspective and interact with them as if they were really there.

A whole new league of immersive experience

Whether it’s Super Smash Bros, Fortnite or Overwatch, many of the most popular competitive games have their own dedicated leagues (and not necessarily just one!). And now, leagues exist specifically for VR games and players who use the technology.

Industry 4.0 Technologies: Why We Need Immersive Virtual & Projection Technology More Than Ever

We are now in the era of Industry 4.0; the fourth industrial revolution. And it’s the fastest one yet...

Almost 80% of global senior IT leaders are confident that the upcoming five years have a century’s worth of technological advancements in store for us. It isn’t hard to see why, with the exponential development and incorporation of the Internet of Things, machine learning and the smart factory. Technology is everywhere, in almost all facets of our work and home life, and there’s simply no way to avoid it: if businesses can’t adapt to it quickly enough, they will get left behind.

We now live in a world where technology isn’t just a novelty or a convenience we work with; it’s a necessity that now works with us. And this brings to its head a huge job for immersive technology to carry out during this era: to bring humans and machines even closer together seamlessly. Here are just some of the ways companies have realised the immense potential of virtual and augmented technologies to make the smart factory become even smarter.

Augmented reality - manufacturing

The use of augmented reality can make an excellent interactive environment where workers can learn while practising on the job. The following video shows an augmented reality training application in use, serving as a visual production guide for the assembly of a Ducati motorcycle engine.

Through projected instructions, sound effects and colour-coding, the system clearly walks the user through the manufacturing process and can measure the exact torques and angles of tightened screws. It can also detect errors such as a screw being tightened at the incorrect torque, and alerts the user of these through visual and aural feedback.

Even fully trained assemblers working with complicated products quickly could benefit greatly from such a system, to make sure their output is consistent and free from human error. Such a tool may prove particularly handy for rush jobs while preventing sacrifice of quality!

Virtual reality - product design

Virtual reality offers product designers the opportunity to test their prototype products in the form of interactive computer-generated models. You can get an idea of what ideas would work and what wouldn't within having to invest in materials and assembly, so you can save resources for only building the products you are most confident in.

Here is an example of such an application in action, designed for Jacuzzi to test out a prototype for one of their showers. The user can step into the computer-generated simulation of a shower cubicle to assess space and comfort. Then they can adjust the virtual shower head through gesture control to get a feel or its ergonomics and ease of use.

Augmented reality - quality control

As well as making assembly much smoother, augmented reality can prove to be invaluable in the quality control process. Even inspectors with the best eye for detail can overlook particular product defects (this is part and parcel of being human, after all!), but a digital application can highlight these quickly and safely. This is a proof of concept for such a system developed by German software provider CAQ AG, demonstrated with the inspection of a ballpoint pen.

Powered by gesture control, the system allows users to select inspection processes to carry out, which it then assists through instructions and visual cues. Not unlike the aforementioned Ducati production guide, the application can measure component parts and highlight any unacceptable deviations from product standards.

Virtual reality - simulation training

This is perhaps the use case we have heard of most often in the workplace: more and more companies have adopted the use of interactive virtual reality applications to train their employees, where they can develop soft skills and an understanding of health and safety guidelines. There are many benefits for virtual reality training applications over traditional methods:

In fact, the virtual reality training application has become so widespread and versatile in recent years that we even have a separate article with use cases and more information, which you can check out here! link here immersion and training).