British Computing Society Human Computer Interaction Conference- Sponsored By Holospatial

From the 11th to the 13th of July our team took a break from holo-ing up the projection industry, to focus on one of the key incentives for Holospatial projection interfaces- Human Computer Interaction.

With academics from across the UK and from as far as Japan, there was a wonderful collective of academia, students, PHDs, Masters and Doctorates. An incredible interaction gallery adorned the central atrium with workshops, keynotes and a superb line-up of talks ensuring the event was continuously a-buzz. The interaction gallery included submissions from Napier University and Demontfort as well as many other exemplary displays of human-computer interaction in the full line-up as below:

Tuesday

  1. Susan Lechelt, Inge Panneels and Ingi Helgason. Data-Driven Innovation for Sustainable Creative Practice (80)
  2. John Morrison, Andrew Mckelvey and Matthew Kranicz. Experiential Dialogues: Extended Reality as an Attentive Means of Listening and Knowing Care Identity (52)
  3. Sandra Woolley, Seb Heron, Joash Abejide and Matthew Chau. Digital Art, Interactive Animation and Creative Expression in the Computer Science Curriculum (76)
  4. Alan Dix. Tools for community heritage and digitisation
  5. Tanis Grandison, Tom Flint and Kirstie Jamieson. Digi-Mapping: Creative Placemaking with Psychogeography (24)
  6. Tom Flint and Linda Shore. Emotional Data Visualised (EDV) (28)

Wednesday

  1. Lynne Hall, Gill Hagan-Green and Samiullah Paracha. Alternative Views of Cyber Security: Innovation, Art and Collaborative Practice (74)
  2. Craig Appleby and Sophy Smith. A Cybernetic Performance System for Live Performance (17)
  3. Sandra Woolley, Tim Collins, Erlend Gehlken, Richard Rhodes and Mustafa Dhar. Interactive 3D Viewer Interfaces for Virtual Museum Artefacts (77)
  4. Fiona Stewart, Tom Flint, Tanis Grandison, Gemma Webster and Craig Tyrie. Let’s Play the Arcade Machines (30)
  5. James Mitchell. An Augmented Reality Virtual Museum ‘Takeout’ Interaction (83)

Holospatial created a digitised inception-like immersive projection display to slide alongside and create a live view-within-a-view of the main conference/expo space. It was great discussing our innovations with world leading academics and even greater was being able to part-take and learn from these discussions, key-notes and speeches and collaborative create ideas, solutions or new pathways of thinking.

Our highlight was a wonderful workshop by Gordon Rugg on Day 1 (hydeandrugg.com) entitled "Generating, Developing and Evaluating Ideas Systematically".

But having said that the BBQ on the 12th on the grand lawns of Keele Hall looking over a spectacularly beautiful landscaped gardens and forest was also right up there.

Big thanks to Sandra Woolley and Ed De Quincey as organisers as well as all the other supporting sponsors and every single person we (and the Holospatial platform) was able to interact with. We absolutely enjoyed being apart of the conference, discussions and sessions and we can't wait to reconnect at the next one at York University in 2023.

Interactive CAVEs & Fully Immersive Projection

What is a CAVE? It's a Customised Audio Visual Environment. It's quite an umbrella term without being anything quite in particular- and is often mis-understood.

They are pretty big, high-tech, complex and most often projected utilising high-power projectors that are mounted in a permanent installation usually on a pretty bulky frame. You'll find them mainly in behind-closed-doors environments for product design, architectural review and a number of specialist industries.

How Does A CAVE Work?

A CAVE combines multiple projectors, screens, sound variations and multiple low-level interactions to create a high-fidelity simulation environment. They were originally incepted in the US in the early 90's and the systems and their dependencies haven't heavily evolved- projection technology has remained expensive and so too have the media processors and heavy graphical requirements that they depend on.

Further to this CAVE installations are quite primal, they use 3D stereoscopics and often require the users to wear tracking devices. They are super-exciting and as the name entails can be customised to exacting specifications.

Got Room For A CAVE?

Good! CAVEs are hugely empowering devices that enable users to interact with digital content in ways that on-screen computing can't even consider. And whilst systems can vary, most often they focus on the same high-powered, high-end technology to ensure optimum output. This is critical as delivering immersion is not as simple as just enveloping and targeting senses, but ensuring that smaller elements don't have collateral damage on the overall immersive effect.

A CAVE projection installation is ultimately a customised metal-framework, hyper-conditioned to achieve immersion that results in a room-type environment. There isn't many producers of CAVEs so your options are well considered and easy-to-manage, the key in reality is before embarking into a CAVE is to create a pipeline, a strategy, what does your immersive journey look like? What might your immersive journey look like? Who are you trying to immerse? Why are you trying to immerse them?

Talk to us about your customised audio visual environment- we'll assist you in scoping your project, aligning your team and considering your immersive journey for today and tomorrows needs.

Let's makeyourspacework@holospatial.co.uk

Staffordshire Connected & (Immersive) Intelligent Mobility Innovation Accelerator

Holospatial is delighted to confirm our participation and acceptance into the Staffordshire Connected & Intelligent Mobility Innovation Accelerator aka SCIMIA.

SCIMIA is an exciting accelerator focusing on Holospatials' immersive intelligent mobility credentials alongside the UKs leading university in the sector, with applications and innovations focused on reducing travel requirements, travelling smarter or travelling altogether without leaving the office. This also extends to smart training, meetings in full-360, conferences, remote viewing and monitoring of locations.

These are this very first steps in our participation of the accelerator program that will be taking place throughout the next 12-months- we can't wait to share more information with you.

For more information on our intelligent mobility innovations contact us via support@holospatial.co.uk

SCIMIA is a dedicated project led by Staffordshire University driving research and innovation through collaborative knowledge exchange.

SCIMIA is a dedicated project led by Staffordshire University driving research and innovation through collaborative knowledge exchange between Staffordshire University and Stoke-on-Trent & Staffordshire LEP SMEs to develop innovative solutions for the intelligent mobility market.

Defined as the smarter, greener and more efficient movement of people and goods around the world, Intelligent Mobility is a sector of the wider transport industry which is predicted to be worth around £900 billion a year by 2025 (Transport Systems Catapult).

New and innovative solutions have a potential to:

Intelligent mobility is an exciting market where opportunities are identified as the interconnections made through a range of industries and technologies including:

Immersive Innovation areas

So you want to be a VR developer - what you need

So you want to be a VR content developer? We are so thrilled to hear that!

Perhaps you have found yourself completely mesmerised after trying out a VR headset or playing an AR game on your phone.

Or perhaps, like us, you have grown frustrated by content developers not taking full advantage of this fantastic medium, and wish to see that change.

In any case, you can stop wishing. There are loads of tools and resources out there for any aspiring VR developers to become an active one, and this article will cover all the basics skills and equipment you need before you can make your virtual dreams a virtual reality.

Coding knowhow

As we know, the key ingredient of all software (and the backbone of the VR industry in general) is code. And lots of it. This, of course, means you’ll have to have at least some experience in working with it. Depending on which engine you decide to make your content in (which we’ll discuss a bit later), C# and C++ are both wise choices of programming language to learn in the field.

Whether you’re completely new to the world of programming or feel your skills are a little rusty, there are many free helpful resources for learning how to code online. As we believe immersive reality should be available for everyone, you can always check out our educational resources for tips and tricks on how you can get started!

An engine

So now you’ve got impeccable coding skills under your belt, you’ve got to start putting them all into practice. But getting to work on building your amazing creation from the ground up seems rather intimidating...and frankly, it probably wouldn't be worth all the trouble. At least, not when game engines exist.

Of course, a game engine can’t make your game for you with the press of a button, but by offering a user-friendly framework to get started from, it sure does accelerate development!

There are several game engines to choose from, each for different needs, but we personally recommend Unity: it is free to download and use, based on C#, has a very large support base, and comes with loads of tools and resources specifically for XR content development out of the box.

Unreal Engine is also an extremely powerful engine, albeit fuelled by C++. It is a little bit more difficult to master, but the end results are very often breathtaking.

SDKs

Game engines have granted you a solid framework to help you get started on your VR masterpiece, but you'd still need some extra tools and features to make sure you can export a working application on all of your target operating systems. That is where software development kits come in. IN PROGRESS

We have developed a software development kit for Unity to help you get your ideas from the PC to the portal!

A VR-ready computer

Let’s face it: virtual reality applications are intensive and resource-hungry.

You may think to yourself “but I’m not playing VR games; I’m just creating them” and set out to make do with your trusty laptop and its integrated graphics card. But just like every great chef tastes their creation, every great software developer tests their own software before they’re ready to release it.

That doesn’t mean you have to throw out your old machine and break the bank on an all-singing, all-dancing gaming rig, however.

GameSpot has made an extremely handy guide on getting your computer set up for VR gaming and production (while it’s 4 years old at the time of writing this article, a lot of the advice still holds up!)

How To Make Sure Your PC Is VR-Ready - YouTube

An idea!

Metaverse Junior: The Implications of an Immersive Internet for Children

A Metaverse-like Internet dominating our lives is looking more like a matter of when than if. And when it does eventually arrive, among the most enthusiastic to jump head-first into it would be kids and teens. After all, they have their whole future ahead of them - a future shaped by all the tech evolutions on the horizon.

Life in the Metaverse wouldn’t all be fun and games, however. There will be dangers and threats that will require some street-smarts to dodge. Children naturally don't have these yet, so parents need to protect their sons and daughters from losing their way in the Metaverse, and content developers have to be mindful of what their youngest users may get up to.

Inappropriate Content

The Internet has amassed a huge wealth of all sorts of information and content. But as we all know, there is a lot of stuff out there that isn’t appropriate for younger viewers (e.g. pornography, strong violence or obscene language). Because of the huge and open nature of the World Wide Web where most content isn’t monitored, censoring children from problematic websites and comments is more difficult than stopping them from accessing age-restricted films or video games.

PARENTS SHOULD

Consider implementing a firewall to stop children and teenagers from accessing inappropriate websites and spaces. It's important to remember, however, that this isn't a one size fits all solution. Not all parents may see eye-to-eye when it comes to what is suitable for their young ones and what isn't - this all depends on the age and maturity of their children, and individual family values. You'd be wise to enable profanity filters while your 7-year-old browses the Metaverse, but your 16-year-old may be downright insulted by the same treatment!

If

DEVELOPERS SHOULD

Prohibit minors from accessing their spaces or using their apps if they host adult content are otherwise not suitable for younger viewers.

It's all too easy for children to lie about their age on the Internet and say they're over the age of 18 to get access to content they shouldn't have. But

Immersive Data Collection

Websites on the Internet we currently use may already collect enough information as they do, be it our viewing and search history, email addresses, pictures we upload, and in some cases, even our sign-in location. But the information collected of us via the Metaverse is bound to be even more intensive as headsets and motion-capturing devices can read our body movements, facial features and other biometrics.

PARENTS SHOULD

Carefully read the relevant sections of terms and conditions as to what data an app or space may collect, and how it would be used. Children should also be aware of how they should keep themselves safe online (e.g. not sharing too much personal information or joining spaces they don't know).

DEVELOPERS SHOULD

Be clear and upfront about the data that is collected of their users, and ensure sufficient privacy procedures are in place if their app or space permits children. Many countries enforce legal regulations on how data of users under 18 should be collected and processed - the United States is particularly serious about data collection of children in accordance to the Children's Online Privacy Protection Act (COPPA).

Immersive Screen Time

Virtual reality applications are even more immersive than traditional media formats. So immersive, in fact, that they can make you lose your sense of time passing by! Researchers have identified this phenomenon as 'time compression' and it's surprisingly common among VR headset users. Thanks to time compression, timing yourself within the Metaverse without a clock nearby would be virtually impossible...and could lead to addiction!

Not only can too much screen time be harmful for a child's eyes, but it can also disrupt their sleep schedule, in turn affecting their performance in school and their energy levels throughout the day.

PARENTS SHOULD

Set timers if their child is particularly prone to tech addiction, both to limit the amount of time spent in the Metaverse and to ensure they can log off to unwind, just in time for bed.

DEVELOPERS SHOULD

Add real-time cues to help users identify the time, such as a clock or timed lighting. It'll be surprising

Microtransactions

Each year, more and more news stories are popping up of young children racking up thousands in debt on their parents’ credit cards...and not to buy new toys or designer trainers.

Video games are more popular now than they ever have been. Many of the most popular titles are free to play, but with a catch. As the difficulty increases, so does the amount of ads tempting the player to buy cheap accessories or tools to help them get through that one tricky level. Some games may even have loot boxes, a lottery-like system of surprise bundles where you don’t know what items you’ll get until you buy one.

In a Metaverse world, you’ll probably have the option to dress your virtual avatar however you want, and decorate your personal space with digital furniture and goodies. It’s ridiculously easy to get kids pestering their parents to buy them anything if it’s marketed as the in thing to show off to their mates. They don’t understand the value of money or the greed of the game industry, and that is why they’re a prime target of such predatory business models.

PARENTS SHOULD

Establish a system of trust and transparency when it comes to spending money in the Metaverse. If your child really wants to buy some accessories for their avatar or space, place a limit on how much they spend.

DEVELOPERS SHOULD

In an ideal world, not add microtransactions to spaces aimed at kids! Sadly, this is wishful thinking as many companies place profit before principle.

Grooming and Bullying

For many young people, the Internet has allowed them to make new friends from around the world, many of whom they had never met in person. Unfortunately, the web has also become a hotbed for two of every parents' worst nightmares: cyberbullying and sexual grooming. Bullies who are too cowardly to say boo to a goose face-to-face suddenly have the medium to troll and harass other users without the fear of being personally exposed.

However, the most evil threats lurking in the Metaverse are usually the ones who act the friendliest. As users would assume the form of a computer-generated avatar as they roam around the Metaverse, it may be even harder for children to forget that the people they meet online may not be who they say they are. This means they could be talking to absolutely anyone. It's this anonymity and a child's naivete that paedophiles take advantage of; they know exactly how to let a young person's guard down by pretending to be another child their age or making them feel like nobody understands them like they do.

PARENTS SHOULD

Make sure their children understand the dangers and risks of talking to strangers online. The vast majority of kids know not to start talking to random people they spot on the street, but that caution seems to go by the by online.

Parents should also encourage their children to talk to them if they see or hear anything that has made them uncomfortable or distressed.

DEVELOPERS SHOULD

Keep their spaces safe for everyone by implementing a zero-tolerance policy for bullying or predatory behaviours. Children should be allowed to report problematic users to a moderation team.

Post Pandemic, The Office Is Shared, Immersive, Social. It's Collaborative, It's Open, It's Flexible

Why we need immersive tech more than ever in a post-pandemic world

2020 was a game-changing year for the entire world. We humans are social beings by nature, so naturally, we all took being able to meet up with our friends, family and work colleagues for granted.

But then COVID-19 struck…

While the lockdowns and social distancing that followed suit were necessary to keeping people as safe as possible from the disease, it also brought about an epidemic of loneliness. In a climate where we couldn't go out to work or even meet up with anyone outside our homes physically, technology was the glue that kept us together. From business meetings on Zoom to birthday parties in Animal Crossing, we relied on our electronics to keep in touch as our everyday life was flipped on its head.

But even now, as things slowly start to go back to normal (or rather, we’re transitioning to a “new normal”), the past year has made something clearer than ever: we need to continue finding new ways to keep the world spinning that are safe, accessible and affordable.

Alternative Ways To Move

Cancelled flights on airport board panel

The tourist industry was among the most badly-hurt by the coronavirus. Travel restrictions and international outbreaks meant millions were forced to cancel their holidays, and hotels and tourist destinations struggled to keep afloat in their losses. In recent times, people have turned to the power of VR to satisfy their quench for globetrotting without leaving the house.

The National Geographic Explore VR app offers adventurers a whole package of exciting expeditions, so you can traverse around Machu Picchu temples, kayak through the Antarctic, and climb El Capitan, all from the comfort of your sofa! And if your trip was postponed for work, then you have even more options, with multiple ways to connect locations, Portals or viewpoints together, and visit them, without a boarding pass. While most of us still long to do it all properly, virtual travel may become a resident of the post-pandemic world rather than just a visitor.

A lot of travellers and consumers want a seamless experience, to go from looking to booking with minimal clicks, instant gratification and saving as much time as possible.

Ralph Hollister

Even when the lockdown lifts and we can all go abroad again, our headsets may still come in handy for booking plane tickets, viewing interactive hotel tours, and getting up close to wild animals (something you wouldn't be able to do safely otherwise!)

Sports matches 

The pandemic has also squashed sports fans’ hopes of flying over to watch their favourite matches and tournaments. However, the show must go on. Most of those widely anticipated matches still happen, albeit behind closed arena doors. What makes the whole experience of watching a sports match in person is a strong sense of ‘togetherness’, not just out of loyalty for your favourite teams or players, but with fellow supporters. And the VR headset just can’t offer that. With a large-scale spatial reality portal, however, fans can still get together and cheer at their favourite sporting events with a 360-degree panoramic view, all while socially distanced. 

Product trials 

Social distancing measures don’t just end at having to keep away from everyone else: they also entail other restrictions on what you can do in public places. Changing rooms are sealed off and free sample booths are packed away, so shoppers are left with no choice but to try out products after purchasing them. But to be fair, this has been the case with loads of products even before COVID crept in - you wouldn’t curl your eyelashes with a mascara you had just grabbed from a store shelf at the best of times!

Beauty brand L’Oreal is particularly passionate about this idea. Their tech division ModiFace is dedicated to developing AR mobile apps, where potential customers can see how different hair dyes and makeup products would look on them, just by using their camera. The apps also use artificial intelligence to ensure the resulting edited image faithfully represents how the actual product would look on the user.

Meetings and gatherings

If you’ve ever attended a concert, festival or convention, you’re familiar with the chaos that ensues: scrambling to meet up with friends or colleagues, trying to beat the crowds (and food lines), and madly searching for merch or booths. Chances are, you probably felt a little under the weather within the next few days with good old con crud (i.e. a cold or flu following a convention). Simply put, large events and airborne diseases don’t mix…and COVID has made us even more mindful of that.

Does this mean such massive meetups are a thing of the past, however? Not necessarily.

Mesmerise, a Manchester-based tech startup, has developed a revolutionising solution for keeping the conference alive in a safe and modern form. The Gatherings VR platform lets guests navigate booths and exhibitions within a virtual main hall, and stream live and on-demand speaker sessions with a front seat view. Because the only limits are software and technology, hosts can put on events that would not otherwise be feasible via physical means.  

There is no better way to get together than by creating a common stage. Virtual media and social hubs are the new hotspots for entertainment, marketing, and tourism - they're only going to become more plentiful and wonderful in the coming years, and we're excited to see how they evolve in our constantly evolving society in the next few years

An Interactive and Multimodal Virtual (reality) Mind Map for Future Immersive Projection Workplace: Research Article

The following is a featured report authored by David Kutak, Milan Dolezal, Bojan Kerous, Zdenek Eichler, Jiri Vasek and Fotis Liarokapis. We keep well on top of the latest industry perspectives and researchers from academics globally- it's our business.

So to bring that to you we share our favourite reports on a monthly basis to give greater exposure to some of the leading minds and researchers in the mixed reality, immersive technology and projection fields- enjoy!

Traditional types of mind maps involve means of visually organizing information. They can be created either using physical tools like paper or post-it notes or through the computer-mediated process. Although their utility is established, mind maps and associated methods usually have several shortcomings with regards to effective and intuitive interaction as well as effective collaboration. Latest developments in virtual reality demonstrate new capabilities of visual and interactive augmentation, and in this paper, we propose a multimodal virtual reality mind map that has the potential to transform the ways in which people interact, communicate, and share information. The shared virtual space allows users to be located virtually in the same meeting room and participate in an immersive experience. Users of the system can create, modify, and group notes in categories and intuitively interact with them. They can create or modify inputs using voice recognition, interact using virtual reality controllers, and then make posts on the virtual mind map. When a brainstorming session is finished, users are able to vote about the content and export it for later usage. A user evaluation with 32 participants assessed the effectiveness of the virtual mind map and its functionality. Results indicate that this technology has the potential to be adopted in practice in the future, but a comparative study needs to be performed to have a more general conclusion.

1. Introduction

Modern technologies offer new opportunities for users to communicate and interact with simulated environments, to quickly find information and knowledge when needed and also to learn anytime and anywhere (Sharples, 2000). When humans communicate, the dynamics of this social interaction are multimodal (Louwerse et al., 2012) and provide several different patterns, like entrainment of recurrent cycles of behavior between partners, suggesting that users are coordinated through synchronization and complementarity, i.e., mutual adjustments to each other resulting in corresponding changes in their behavior occurring during the interaction (Sadler et al., 2009). When users are engaging in a collaborative task, then synchronization takes place between them through multiple modalities. Some of these include gestures, facial expression, linguistic communication (Louwerse et al., 2012) or eye-movement patterns (Dale et al., 2011). By designing multimodal interfaces, it is possible to improve the accessibility and usability of mind mapping systems to achieve a natural and intuitive experience to the users.

Traditional types of mind maps involve some means of the visual organization of information. During the past few years, there have been some initial approaches to developing 3D mind maps to boost productivity. A number of human-computer interaction (HCI) technologies exist nowadays that address this topic, and some of them tend to work well in specific situations and environments. A virtual reality (VR) solution for shared space can be realized in several ways depending on the required level of immersion of the end-user in addition to the requirements of the application. One of the most accepted definitions states that immersion refers to the objective level of sensory fidelity a VR system provides whereas presence addresses user's subjective psychological response to a VR system (Slater, 2003). The level of immersion is directly interrelated with the end-user perception and promoted if tactile devices are used. VR, therefore, has the potential to augment processes of our everyday life and to mitigate difficult problems.

An open problem is a co-location in the office of the future environment. Meeting with people on site is costly because often people need to travel to the location from various cities or countries. The Internet enables to connect these people from the technical point of view while VR allows to achieve much more immersive and natural cooperation. Nowadays, several immersive virtual solutions that address co-location exist, such as collaborative applications for cave automatic virtual environment (CAVE) systems (Cruz-Neira et al., 1993), where users do not wear head-mounted displays (HMDs) and are able to see each other and interact directly due to their location in the same physical space. Nowadays, collaborative immersive VR allows users to be co-located in the same space or in different locations and achieve communication through internet (Dolezal et al., 2017). The availability of current HMDs allows for the easier creation of strongly immersive user experiences. Typical sub areas of shared spaces for VR include visualization, communication, interaction and collaboration, and VR-based mind map workflow overlaps and relies on all four aspects of the experience.

The main focus of this research is on multimodal VR collaborative interfaces that facilitate various types of intelligent ideation/brainstorming (or any other mostly creative activity). Participants can be located in different environments and have a common goal on a particular topic within a limited amount of time. Users can group (or ungroup) actions (i.e., notes belonging in a specific category) and intuitively interact with them using a combination of different modalities. Ideally, the multimodal interface should allow users to create actions (i.e., post-it note) and then post it on the virtual mind map using one or more intuitive methods, such as voice recognition, gesture recognition, and through other physiological or neurophysiological sources. When a task is finished, users should be able to access the content and assess it.

This paper presents a novel shared virtual space where users are immersed in the environment (i.e., same meeting room) and participate in a multimodal manner (through controllers and voice recognition). Emphasis is given on the (a) shared VR environment; (b) effective performance of the multimodal interface; and (c) assessment of the whole system as well as the interaction techniques. The tasks are typically moderated by one or two individuals who facilitate the process, take care of the agenda, keep the schedule, and so on. Such ideation exercise can be used on various occasions but is typically associated with the creative process in the company where the output of the exercise is uncertain before it is executed. During a particular task users can create and manipulate shared nodes (equivalent to real-world sticky notes), modify their hierarchical or associative relationships and continuously categorize, cluster, generalize, comment, prioritize, and so on. Moderator's role is to guide the discussion and regulate the voting phase.

With the appearance of novel interfaces and mediums, such as VR and increasing presence of sensors and smart devices in our environment, it has become apparent that the typical way we interact with the computer is changing rapidly. Novel ways of achieving fluent interaction in an environment saturated with sources of useful behavioral or physiological data need to be explored to pave the way for new and improved interface designs. These interfaces of the future hold the promise of becoming more sophisticated, informative, and responsive by utilizing speech/gesture recognition, novel peripherals, eye-tracking, or affect recognition. The role of multimodal interfaces is to find ways to combine multiple sources of user input and meaningful ways of leveraging diverse sources of data in real time to promote usability. These sources can be combined in one of three levels, as outlined in Sharma et al. (1998), that depends on the level of integration (fusion) of distinct sources of data. There is a real opportunity to mitigate the difficulties of a single modality-based interface by combining other inputs.

Collaborative Virtual Environments (CVEs) may be considered as shared virtual environments operating over a computer network (Benford et al., 2001). They have different application domains ranging from health-care (McCloy and Stone, 2001Rizzo et al., 2011), cultural heritage (White et al., 2007Liarokapis et al., 2017), education (Redfern and Galway, 2002Pan et al., 2006Faiola et al., 2013Papachristos et al., 2013) to psychology (Loomis et al., 1999), and neuroscience (Tarr and Warren, 2002). One of the main disadvantage of CVEs is that they do not support non-verbal communication cues (Redfern and Galway, 2002). The typical solution to overcome this problem is to include a representation of the participants in a form of avatars. Although this does not solve the problem, it allows for some form of limited non-verbal communication. As a result, participants of CVEs can interact with objects or issue commands while being observed by the virtually co-located collaborator.

The benefits, design, and evaluation in the field of designing speech and multimodal interactions for mobile and wearable applications were recently presented (Schaffer and Reithinger, 2016). Having a multimodal VR interface can be beneficial for several complex operations as well as new applications, ranging from automotive styling to museum exhibitions. The multimodality can also be achieved by providing different visual representations of the same content so the user can choose the most suitable one (Liarokapis and Newman, 2007). The main principle of the concept of multimodality is that it allows users to switch between different types of interaction technologies. Multimodal interfaces can greatly expand the accessibility of computing to diverse and non-specialist users, for example, by offering traditional means of input like the keyboard and also some uncommon ones like specialized or simplified controllers. They can also be used to promote new forms of computing and improve the expressive power and efficiency of interfaces (Oviatt, 2003).

The flexibility of multimodal interfaces allows for the better alternation of input modalities, preventing overuse and physical damage arising from a repeated action during extended periods of use. Furthermore, multimodal interfaces can be used to provide customizable digital content and scenarios (White et al., 2007) while, on the other hand, they can bring improvements by combining information derived from audio and visual cues (Krahnstoever et al., 2002). Acquisition of knowledge is also augmented through the use of such multimodal MR interface compared to a traditional WIMP-based (Windows, Icons, Menu and Pointer) interface (Giraudeau and Hachet, 2017). In fact, one example implementation of a mind-map based system reported in Miyasugi et al. (2017) allows multiple users to edit a mind map by using hand gestures and voice input and share it through VR. Initial comparative experiments with representative mind map support software (iMindMap1) found that the task completion time for creating and changing the key images was shorter than that of iMindMap. Currently, there are several software alternatives for mind map creation; XMind2 and iMindMap being the most famous ones, but most of these solutions are aimed at a single user and support only traditional non-VR enabled interfaces. In the world of VR applications, Noda3 is one of the most progressive alternatives. Noda utilizes spatial mind maps with nodes being positioned anywhere in the three-dimensional space, while it does not offer collaboration possibilities.

Having a three-dimensional mind map presents some advantages like increased ability to exploit spatial thinking and theoretically infinite place for storing ideas. On the other hand, spatial placement might decrease the clarity of the mind map as some nodes might be hidden behind the user or other nodes. The one-to-one correspondence with traditional mind mapping software is lost as well, which makes it hard to export the results for later processing and review. This would decrease the usability of the outputs created inside the VR, and it is the reason why our approach works with two-dimensional (2D) mind maps.

Another alternative tool is Mind Map VR4 offering more or less same functionalities as Noda. An interesting feature of the Mind Map VR is the ability to change the surroundings for a different looking one. When concerned about collaborative platforms, rumii5, CocoVerse (Greenwald et al., 2017), and MMVR6 (Miyasugi et al., 2017) are closely related to our work. In its core, all of these systems provide users a possibility to cooperate in VR. Rumii is, however, aimed mostly at conferencing and presentation, while CocoVerse is aimed at co-creation mainly via drawing so although mind mapping is theoretically possible, the application is not designed for this purpose.

As MMVR is focused on mind mapping and online collaboration in VR, it does have a similar purpose as our application. MMVR utilizes hand gestures to create mind maps with nodes positioned in three-dimensional space. On the contrary, in our system, VR controllers are used for the interaction, and the map canvas is two-dimensional. Similarly to Noda, authors of MMVR decided to take a slightly different approach than we did regarding the mind map representation. Besides already mentioned things, we tried to make the mind mapping process more related to the real-world one—VR controller acts as a laser pointer while 2D canvas is a virtual representation of a whiteboard. MMVR also excludes features related to brainstorming, such as voting.

3. System Architecture

The traditional way of brainstorming using post-it notes presents several drawbacks related to a reshuffling or modifying of notes during the whole process as post-it notes often fall from the wall and trying to do it multiple times makes them not staying on the wall anymore. Besides that, taking multiple notes from multiple places to move them to some other place is cumbersome. Mapping relationships between post-it notes is another difficult task, one needs to make lines among post-it notes and to label the lines if needed, but to do this one must often re-shuffle the post-it notes to make the relationships visible. Elaborating on a particular topic (for example deeper analysis requiring more post-it notes) in one part of the exercise is also difficult as all of the other post-it notes need to be reshuffled again to make space for the new exercise. It is challenging to draw on the post-it note when needed and then stick it on the wall. Finally, post-exercise analysis is difficult; it typically involves a photograph of the result and then manual transcription into a different format; for example, brainstorming “tree” and word document as meeting minutes. If it is necessary to perform a remote brainstorming, the disadvantages are much more significant, and there does not exist a flawless solution.

Our system is designed in such a way to try to take the best of both the interpersonal brainstorming and software approaches and merge it into one application. The main part of our system is a canvas with a mind map. The canvas serves as a virtual wall providing users space to place their ideas and share them with others. All nodes are positioned at this 2D wall to keep the process as close as possible to the real-world while also providing similar visual style as conventional mind mapping software tools. To simplify collective manipulations, our system introduces a simple gesture. The user draws a shape around the nodes he or she wishes to select and then simply moves them around using a cursor button which appears after the selection ends. This feature is described in more detail in section 3.2. One of the big challenges of VR technology lies in the interaction side. Existing speech-to-text tools were integrated into our system to allow users to use voice-based interaction.

When the process of brainstorming finishes, voting about the best ideas takes place. In real-world exercise, it is hard to make sure that all participants obey the voting rules. Some participants might distribute a different number of points than they should, or they can be influenced by other participants. Our tool provides a voting system, ensuring that the voting points are distributed correctly and without being influenced by other participants. Brainstorming exercise is usually performed with more people at the same place while one of them serves as a moderator. This might not be a problem for teams sharing a workspace, but when it is desired to collaborate with people being physically far away, things are much more complicated. Our tool provides a VR environment where all users can meet, although they might be located at different places in the world. The overview of the different parts of the system is shown in Figure 1. Figure 2 shows a screenshot of the application while brainstorming is in progress.

Figure 1. System overview.

Figure 2. Mind map canvas while brainstorming is in progress.

The software was developed in Unity and C#. The networking system presented in Dolezal et al. (2017) was incorporated into the application. Interaction with the HMDs is possible thanks to Virtual Reality Toolkit (also VRTK) plugin. To run the application, SteamVR is required as it provides application interfaces for VR devices. Even if the application is designed to be operational with HMDs, it is also possible to use it just with personal computers like desktop or laptop—without HMD. In this case, the keyboard and mouse are required as input devices. If a microphone is present as well, speech recognition service can be still utilized as an input modality. Regarding the HMDs, the system is implemented to work with HTC Vive (Pro) and one controller. Our focus was on making the controls as simple as possible. For this reason, the user is required to work only with two of the controller's buttons (touchpad and trigger) to have complete control over the system features. When the user presses the touchpad, laser pointer is emitted from the VR controller. Trigger serves as an “action” button; when the user is pointing at some element, pressing the trigger initiates the appropriate action. Video showing some of the system functionality and interaction is in the Supplementary Material.

3.1. Map Nodes

Map nodes are the core component of the system. Each node is represented by a visual element having color and a label. Map nodes can be modified in several ways - they can be moved, deleted, modified, and being updated with new visual styles. It is also possible to make a relation between nodes represented by lines between appropriate nodes. Two types of relations were implemented - parent/child and general ones. Former ones represent a “strong” relation where each node can have only one parent and is dependent on its antecedents - when some of them are moved or removed, this node is modified as well. Latter type of relations is there mainly for semantic purpose. Each node can have as many of these relations with other nodes as desired while persisting independency. Modifications of the nodes are done using radial menus shown while pointing at a node. This allows users to perform the most important actions while still being focused on the mind map. The content of the node's radial menu is shown in Figure 3. Blue buttons provide the functionality to add the aforementioned relations to other nodes. Red button removes nodes while the green button creates a new node as a child of the currently selected node. The last button allows users to record a text for this node.FIGURE 3

Figure 3. Radial menu which opens when a single node is selected.

3.2. Selection of Multiple Nodes

The multiple selection is handled in such a way that a user is required to draw a shape around the nodes he wishes to select. Selection shape is drawn using a controller by pressing touchpad and the trigger buttons at the same time while pointing at the canvas. When the selection is finished, the user can move the selected nodes or perform some actions provided by the appropriate radial menu. Thanks to this feature, selecting several nodes and changing their visual style, position, or relations is quite simple. In the background, this feature is based on a point-in-polygon test. The selection shape is visually represented as a long red line (technically a polyline) which is converted into a polygon (based on vertices of individual line segments of a polyline) after the drawing is finished. Then, for each node, it is computed whether its center lies in a resulting polygon.

3.3. Voice Recognition

Language technology is easier to accept for participants only if it is implemented in an intuitive and easy to use way (Schaffer and Reithinger, 2016). Text input is a big challenge for all VR applications as a traditional keyboard cannot be properly used due to not being visible. It also disallows the user to move freely. The most straightforward idea is to use a virtual keyboard, but this approach is not very effective, especially with only one controller. For this reason, we decided to use speech-to-text technology. Our system is using Wit.ai service to provide this functionality. The user uses an appropriate button in the radial menu to start the recording, says the desired input, and then ends the recording. The rest is handled by our system in cooperation with the service mentioned above. In the background, voice recognition operates in such a way that the user's recording is converted into an audio file which is uploaded to the Wit.ai servers. These servers process the audio and return a response containing the appropriate text. The whole process is running on a separate thread to not block the program while speech is transformed into the result.

3.4. Voting

Voting is a special state of the application during which nodes cannot be edited and which provides an updated user interface where each node is accompanied by plus and minus buttons and a text box with points. This allows participants to assign points easily. Voting consists of several rounds where during each round, one color to be voted about is chosen. Voting is led by a moderator of the brainstorming who decides about the colors to vote about and assigns the number of points to distribute between the voted ideas. For each such voting round, participants see only the number of points they assigned, and they have to distribute all points. When a moderator tries to end the voting round, the system checks whether all participants distributed their points and if not, then the round cannot be closed. When the voting ends, all participants see the summary of points for each node. Winners in each category are made visually distinct.

3.5. Online Collaboration

The core of the network-related part of the system is Collaborative Virtual Environments (CVR) platform (Dolezal et al., 2017) utilizing Unity Networking (UNET) technology (its core structure is shown at Figure 4). The system works on a host/client model, in which one user is a server and a client at the same time while other users are just clients. Each user is represented as an abstract representation of a capsule (as shown in Figure 5) with HMD and VR controller in hands. Both the positions of the avatar and controller are synchronized over the network. Online collaboration also includes a system of node-locking, preventing users from modifying a node while another user is currently working with it, and controller-attached laser pointer which allows users to get immediate feedback about the place they or other user are pointing to. Regarding the node-locking, this functionality is based on the concept of node managers. When a client points at a node, system locally checks whether the node is locked for this client or not. If the node is already locked, it is not selected. Otherwise, the client sends a request to the server to lock this node. Server processes these requests sequentially and for each request verifies whether the claimed node is without a manager, otherwise denies the request. If the node has no manager yet, the server makes requesting user the manager of this node and sends an remote procedure call (RPC) to the rest of the clients that this node is locked. If a node is deselected, unlock message is sent to the server, which then propagates this information down to the clients.FIGURE 4

Figure 4. UNET function calls.FIGURE 5

Figure 5. Representation of the user in VR environment with overlayed image of real users.

3.6. Mind Map Export

At any time during the brainstorming, users of our system can export the mind map into the open-source XMind format. Possibilities of the format are fully utilized, therefore most of the important information like node visuals, relations, or points received during voting are exported. Support of mind map export provides an ability to access the brainstorming results later on or even modify them in other software tools. The mind map is also regularly saved to a backup file which is stored in a custom JavaScript Object Notation (JSON)-like format. This format was designed to be as simple and as fast as possible while still storing all the necessary information. The backup file can be loaded at any time during the brainstorming, making it therefore possible to restore mind mapping progress in case of some failure like lost internet connection.

4. Methodology

This section presents the methodology of the experiment performed for collecting information about the application.

4.1. Participants and Questionnaires

The study consisted of a total of 32 healthy participants (19 males, 13 females) and testing took place in pairs (16 individual groups). Participants were a voluntary sample, recruited based on their motivation to participate in the study. All subjects signed informed consent to participate in the study and to publish their anonymous data. They were aged from 18 to 33 years old, and all of them were regular computer users. They were rather inexperienced with mind maps and generally had some experience with remote collaboration. The very first step was to explain the workflow of the experiment to participants. Then, statistical and demographic data were collected. After the completion of the experiment, subjects were asked to fill in questionnaires related to the recent experience. Two questionnaires were used. The first one focused on measuring presence in VR (Witmer and Singer, 1998Witmer et al., 2005). The second questionnaire aimed at assessing the cognitive workload and was based on the NASA Task Load Index (Hart, 2006). The subjects were also asked to fill in a free-form debriefing session questionnaire, where they provided qualitative feedback for the whole experiment.

4.2. Procedure

The procedure of user testing consisted of two main steps. Participants were located in different rooms, and during the first 10–15 min, depending on the skill of the individual user, each of them was alone in the virtual environment while being introduced to the system and presented with its features. While trying the system functionality, the participant's feedback was gathered. The second part of the evaluation consisted of participants trying to brainstorm on a scenario. To assess the functionality of the system, a number of different brainstorming scenarios were designed. The topics that were chosen include: (a) How to cook an egg properly, (b) What is best to do on Friday night, (c) How will artificial intelligence take over the world, (d) Wine selection for dinner, and (e) Propose your own topic. The given topic for the experiment was “What is best to do on Friday night.” The process was directed by a moderator and contained the following steps:

1. Participants were asked to present possibilities how to spend Friday night using nodes on the wall together

2. Participants were asked to assign other specific properties to ideas from previous exercise and to use different color of nodes

3. Each participant was asked to select one idea and add nodes describing concrete proposal

4. Participants were asked to present to each other results of previous exercise

5. Participants ran a voting session. One of the participants took a role of a voting moderator, the second one was acting as a voting participant.

Time of completion for each of the steps was measured and the behavior of the participants was monitored in order to get another type of feedback.

5. Results

5.1. Qualitative Results

The participants provided us with valuable feedback necessary for further improvements. The feedback was gathered not only by direct communication with participants but also by watching their behavior during the actual scenario. Thanks to this approach, it was possible to collect useful information during the whole time of the testing. During the debriefing, we asked participants whether they know any other tools which can be used for remote brainstorming or collaboration and if they can find some (dis)advantages of our system in comparison to these tools. The mentioned tools included services like Skype, Google Docs/Hangouts, Slack, Facebook, Team Speak, IBM Sametime, and video conferencing platforms.

The most commonly mentioned advantage of our system was immersion. Quoting one of the users, “It makes you feel like you are brainstorming in the same room on a whiteboard (…).” Similarly, the ability to see what is going on was praised, mainly the fact that the users are represented as avatars with a laser pointer instead of abstract rectangles with names as is common in some applications. Another advantage, in comparison to other tools known to participants, was an absence of outside distractions. Also, it was mentioned several times, that our application is more fun than comparable tools. Regarding the disadvantages, the inability to see other peoples' faces was mentioned. Many users also pointed out the necessity to have appropriate hardware, i.e., that such an application requires more equipment and preparations than the tools they know. Another drawback was physical discomfort, mainly the requirement to wear a HMD. Some users mentioned that it takes more time to get familiar with the interface in comparison to common tools they know. Also, the speed with which the ideas can be generated was considered by some participants to be slower than in the case of conventional platforms.

At the end of the experiment, users gave us general feedback about the application. We expanded this feedback by insights we collected by observing their behavior. The most mentioned drawback of the system was the position of the mind map canvas. It was positioned too high, forcing users to look up all the time, which resulted in physical discomfort and decreased the readability of nodes which were positioned at the top of the canvas. Some users also had some remarks about the speed and reliability of the used speech-to-text service. The application itself was generally considered as responsive, although the user interface has space for improvement. Mainly at the beginning, users tended to forget to stop the voice recording after they finished saying the desired text for a node. Also, the difference between parent-child relations and general relations was not clear enough. Regarding the environment, some participants spoke favorably about the space surroundings; on the other hand, one user mentioned that there exists a risk of motion sickness or nausea for some people. Others mentioned that the text at the top of the canvas is hardly readable. Unfortunately, the pixel density of HMD is not good enough at such distance, so it is necessary to consider this drawback when designing similar types of applications. We also noticed that the system of node locking sometimes slows down the work.

Participants also provided some ideas for further improvements. One mentioned that it would be good to have the possibility to decide whether to hide or show activity (including laser pointers) of other people during the voting. Another one pointed out that the current selection of color themes is not very visually pleasing and that it might be good to use some better color palette. One participant said that it might be useful to have more control over voice so you can mute yourself or others, for example, when saying a text for a node. Ability to change the size of the node's text would also be welcomed addition for some users. Overall, the application seemed to be quite immersive but for the price of increased physical demand and possibly slower pacing.

5.2. Quantitative Results

The first part of this section presents compound histogram summarizing participants' evaluations of core system features. Each user was assigning one (= poor) to five (= excellent) points to each feature.

Figure 6 confirms observed behavior which is that users had no major problems when trying to create or delete nodes. The delete might perform a bit worse because when a node is deleted its radial menu remains open until the user points elsewhere. Although the menu is not working anymore, it is a bit confusing that it is still present. This behavior is going to be addressed in the future to deliver a smoother user experience. Distribution of yellow colored responses in Figure 6 shows that the mechanism for moving nodes was not as user-friendly as desired for some participants. This might be caused by the fact that moving of nodes fails (i.e., the node returns to the previous state) when both of the controller buttons are released at the same time. This was a slight complication for some users. Red values, revealing evaluations of change text feature, have a distribution with a mean of 2.94, it can be therefore said that the speech recognition in its current state is acceptable. The question is, how would it perform if different scenario with more complicated words was used? Hence, although the performance is not entirely bad, there is a space for improvement in both the user interface and recognition quality. Then, it might be worth considering whether to stick to the current speech recognition solution or try something else. Another idea to think about is to utilize multimodality even in text input. It was not unusual that the user said a word which was recognized as a different one, but also very similar, to what he wanted, so the difference was just a few letters. It might come as handy to have a quick way of fixing these errors, either in the form of a virtual keyboard or some dictionary-like mechanism.FIGURE 6

Figure 6. Evaluation of usability of system features.

Table 1 presents the results obtained based on Spearman's correlation. An interesting point is the relation between stated physical demand and frustration. When users felt physical discomfort, caused, for example, by too highly placed canvas or weight of the HMD, they became more frustrated. Physical demand can be partly decreased by improving the application's interface, but as long as HMD is used, there will always be a certain level of discomfort. Another interesting output is the correlation between the TLX temporal demand and effort. Participants considering the pace of the task hurried felt that they have to work harder in order to accomplish the task. In this case, improvement of speech-to-text service might be helpful. There was also a strong correlation between answers on “How easy did you find the cooperation within the environment?” and “How quickly did you adjust to the VR environment?” A negative correlation was found between satisfaction with "change text" functionality and answers to TLX questions regarding the feeling of frustration and physical demand. Since this is a key feature from the system perspective, it is used a lot, and when the user does not feel comfortable with it, it might make him or her tired both physically and mentally. Finally, users who considered the visual display quality of HMD as distracting and unsatisfactory felt like the task was more physically demanding. This is partially due to the technological limits of current HMDs but also certain design aspects could be improved. The idea is to improve the colors and sizes of UI elements to decrease the users' eye strain caused by the relatively low pixel density of HMDs.TABLE 1

Table 1. Outputs of selected Spearman's correlation coefficients.

5.3. Log Results

The activity of participants during the testing scenario was logged in order to get more information about their behavior as well as the effectiveness of our platform. Stored information contains general actions performed by the participant (e.g., node creation and deletion) and visualizations of mind map canvas interactions. The median of collaboration times for the scenario was 19 min and 5 s (excluding explanations of each step). Nodes were created only during the first three steps of the scenario, the median of these times is 14 min and 20 s. This accounts for an average speed of ~1–2 nodes per minute since the median of nodes created during the scenario was 24. It is worth mentioning that the speed, respectively duration, of the brainstorming depends on the creativity of the users. The fastest pair was able to create 3.3 nodes per minute on average while slowest one achieved the speed of nearly one node per minute. The relation between the number of nodes at the end of the exercise and total time is shown in figure 7.

Figure 7. Scatter plot of collaboration times and number of nodes (two testing pairs had exactly the same time and node count so there are only 15 visible points).

This could be justified in several ways. First, the users with higher node count might have been simply more creative than the rest, and so it was easier for them to come up with new ideas. Moreover, as each step of the study was not limited by time but rather by a rough minimum of the number of nodes, participants had no problems creating more than enough nodes to continue. The flow of the session was also not interrupted so much by the time spent on thinking about possible ideas. The effect can also be caused by differences in the communication methods between participants. In any case, this confirms that the speed of the brainstorming does not depend only on the system capabilities. Another results from the logs are shown is created by merging heatmaps of all tested users. The points in the image (Figure 9) represent positions in the mind map canvas, which were “hit” by a laser pointer while aiming at a canvas and selecting. The RGB colors determine the relative amount of hits at a given pixel with red being the most “hit” pixels while blue being the least “hit” pixels, whereas green pixels are somewhere in between. Figure 9 shows an averaged heatmap of selected nodes of all users. This determines positions where nodes were selected for the longest time - in this case holds that the bigger the opacity is, the longer this position was covered by a selected node.FIGURE 8

Figure 8. Merged heatmap with pointer movements of all users.

Figure 9. Merged heatmap highlighting positions of selected nodes.

An observation regarding both heatmaps is the fact that the space near the corners of the mind map is unused. This suggests a tendency of users to place most of the nodes near the central root node. Another interesting point is the significant difference in the density of heatmap in the bottom and the upper half of the canvas. This confirms that there might be reduced readability in the upper half of the canvas and users are therefore preferring nodes which are closer to them, i.e., at the bottom of the canvas. Figure 9 also reveals that users generally like to move the nodes around as they wish, and they do not just stick to the default automatic circular placement. This means that it is necessary to have a good interface for node movement. Regarding the movement, in order to be more precise, Figures 10 and 11 show two heatmaps which clusters the users into two categories. The first type is less common and prefers to stick to the default node placement and does only minor changes while the second category of users is more active in this regard. This is also related to another observed user behavior—some people use the laser pointer nearly all the time while others use it only when necessary.FIGURE 10

Figure 10. Example heatmap of the first type of users reorganizing nodes rather rarely.

Figure 11. Example heatmap of the second type of users with more active mind map reorganization.

6. Conclusions and Future Work

This paper presented a collaborative multimodal VR mind map application allowing several participants to fully experience the process of brainstorming. It covered both aspects (a) idea-generation phase as well as (b) voting procedure. Multimodality was achieved based on the combination of speech and VR controller. To verify the usability of the system, experiment with 32 participants (19 males, 13 females) was conducted. Users were tested in pairs and filled several questionnaires summarizing their experience. The results indicate that the system performs according to its specifications and does not show critical problems. In terms of user feedback, comments include mainly minor issues of the usability of the environment and can be clustered as design issues.

Furthermore, there are many possibilities on how to improve and extend the application. Besides general improvements to the interface, avatars will be exchanged for some more realistic ones. Also, name tags will be added to identify individual participants. Thanks to the integrated voice solution, some speech-related features will be added, for example, automatic muting of users when they are saying a label for a node. Moreover, there is also going to be visual feedback, like icon or mouth animation, to make it clear which user is speaking. Possibilities of hand gesture controls will be examined as well. Finally, a comparative user study will be made between traditional platforms for remote collaboration and the VR mind map to assess the advantages and disadvantages of each approach.


Portalco delivers an alternate method of delivery for any VR or non-VR application through advanced interactive (up-to 360-degree) projection displays. Our innovative Portal range include immersive development environments ready to integrate with any organisational, experiential or experimental requirement. The Portal Play platform is the first ready-to-go projection platform of it's type and is designed specifically to enable mass adoption for users to access, benefit and evolve from immersive projection technologies & shared immersive rooms.

Achieving Presence Through Evoked Reality

Jayesh S. Pillai1*, Colin Schmidt1,2 and Simon Richir1

The following report collates a variety of information and perspectives on our multiple realities and how these can impact an immersive experience but more so the human experience which is so critical to long-lasting and user-focused digital experiences that improve memorization, understanding and engagement as a whole.

The sense of “Presence” (evolving from “telepresence”) has always been associated with virtual reality research and is still an exceptionally mystifying constituent. Now the study of presence clearly spans over various disciplines associated with cognition. This paper attempts to put forth a concept that argues that it’s an experience of an “Evoked Reality (ER)” (illusion of reality) that triggers an “Evoked Presence (EP)” (sense of presence) in our minds. A Three Pole Reality Model is proposed to explain this phenomenon. The poles range from Dream Reality to Simulated Reality with Primary (Physical) Reality at the center. To demonstrate the relationship between ER and EP, a Reality-Presence Map is developed. We believe that this concept of ER and the proposed model may have significant applications in the study of presence, and in exploring the possibilities of not just virtual reality but also what we call “reality.”

Introduction

Research on presence has brought to our understanding various elements that certainly cause or affect the experience of presence in one way or another. But in order to evoke an illusion of presence, we in effect try to generate an illusion of reality different from our apparent (real world) reality through different mediations like Virtual Reality. The attempt to evoke an illusory reality is what brought researchers to think about presence in the first place. “Reality,” despite its being a major concept, is most often either overlooked or confused with other aspects that affect presence. To study presence we must first understand the reality evoked in one’s mind. It is this illusion of reality that forms a space-time reference in which one would experience presence. It is evident from the research in the field of virtual reality, that if a medium is able to create a convincing illusion of reality, there will certainly be a resultant feeling of presence. Various theories have been proposed, to explore and define the components of this mediated presence. We aim to abridge those theories in an efficient manner. Moreover, studies in the field of cognition and neuroscience confirm that the illusion of reality can as well be non-mediated (without the help of external perceptual inputs), that is purely evoked by our mind with an inception of corresponding presence. One of the most common but intriguing example of a non-mediated illusion of reality would be – a dream. This self evoking faculty of mind leading to the formation of presence is often neglected when observed from the perspective of virtual reality.

Sanchez-Vives and Slater (2005), suggest that presence research should be opened up, beyond the domain of computer science and other technologically oriented disciplines. Revonsuo (1995) proposed that we should consider both – the dreaming brain and the concept of Virtual Reality, as a metaphor for the phenomenal level of organization; they are excellent model systems for consciousness research. He argues that the subjective form of dreams reveals the subjective, macro-level form of consciousness in general and that both dreams and the everyday phenomenal world may be thought of as constructed “virtual realities.”

According to Revonsuo (2006), any useful scientific approach to the problem of consciousness must consider both the subjective psychological reality and the objective neurobiological reality. In Virtual Reality it’s not just the perceptual input and the technical faculties that contribute to a stronger illusion of reality but also various psychological aspects (Lombard and Ditton, 1997Slater, 20032009) relating to one’s emotion, attention, memory, and qualia (Tye, 2009) that help mold this illusion in the mind. In the case of non-mediated illusion of reality like dreams or mental imagery, the perceptual illusion is generated internally (Kosslyn, 19942005LaBerge, 1998). The dream images and contents are synthesized to fit the patterns of those internally generated stimulations creating a distinctive context for the dream reality (DR; Hobson and McCarley, 1977Hobson, 1988). Whether mediated or non-mediated, the illusion of reality is greatly affected by the context. “A context is a system that shapes conscious experience without itself being conscious at that time” (Baars, 1988, p. 138). Baars describes how some types of contexts shape conscious experience, while others evoke conscious thoughts and images or help select conscious percepts. In fact it’s a fine blend of perceptual and psychological illusions (explained in section The Illusion of Reality) that leads to a strong illusion of reality in one’s mind. We attempt to explore this subjective reality that is the fundamental source of experience for presence.

Presence and Reality

With the growing interest in the field of Virtual Reality, the subject of presence has evolved to be a prime area of research. The concept of presence, as Steuer (1992) describes, is the key to defining Virtual Reality in terms of human experience rather than technological hardware. Presence refers not to one’s surroundings as they exist in the physical world, but to the perception of those surroundings as mediated by both automatic and controlled mental processes.

Presence

Presence is a concept describing the effect that people experience when they interact with a computer-mediated or computer-generated environment (Sheridan, 1992). Witmer and Singer (1994) defined presence as the subjective experience of being in one environment (there) when physically in another environment (here). Lombard and Ditton (1997) described presence as an “illusion of non-mediation” that occurs when a person fails to perceive or acknowledge the existence of a medium in his/her communication environment and responds as he/she would if the medium were not there. Although their definition confines to presence due to a medium, they explained how the concept of presence is derived from multiple fields – communication, computer science, psychology, science, engineering, philosophy, and the arts. Presence induced by computer applications or interactive simulations was believed to be what gave people the sensation of, as Sheridan called it, “being there.” But the studies on presence progressed with a slow realization of the fact that it’s more than just “being there.” We believe that presence, whether strong or mild is the result of an “experience of reality.”

In fact “presence” has come to have multiple meanings, and it is difficult to have any useful scientific discussion about it given this confusion (Slater, 2009). There can be no advancement simply because when people talk about presence they are often not talking about the same underlying concept at all. No one is “right” or “wrong” in this debate; they are simply not talking about the same things (Slater, 2003). On the general problems in conveying knowledge due to the intersection of the conceptual, material, and linguistic representations of the same thing, there exists an attempt to explain the workings of communication and its mishaps (Schmidt, 1997a,b2009), which clearly states that scientists must always indicate which representation they speak of. In this article, we are mainly speaking about the phenomenon, which is the experience of presence.

Reality

The term “reality” itself is very subjective and controversial. While objectivists may argue that reality is the state of things as they truly exist and is mind-independent, subjectivists would reason that reality is what we perceive to be real, and there is no underlying true reality that exists independently of perception. Naturalists argue that reality is exhausted by nature, containing nothing supernatural, and that the scientific method should be used to investigate all areas of reality, including the human spirit (Papineau, 2009). Similarly a physicalist idea is that the reality and nature of the actual world conforms to the condition of being physical (Stoljar, 2009). Reality is independent of anyone’s beliefs, linguistic practices, or conceptual schemes from a realist perspective (Miller, 2010). The Platonist view is that reality is abstract and non-spatiotemporal with objects entirely non-physical and non-mental (Balaguer, 2009). While some agree that the physical world is our reality, the Simulation Argument suggests that this perceivable world itself may be an illusion of a simulated reality (SR; Bostrom, 2003). Still others would endeavor to say that the notion of physical world is relative as our world is in constant evolution due to technological advancement; also because of numerous points of view on its acceptation (Schmidt, 2008). Resolving this confusion about theories on reality is not our primary aim and is however beyond the scope of this study. So we reserve the term “Primary Reality” to signify the reality of our real world experiences, which would be explained later in this paper.

The Illusion of Reality

The factors determining the experience of presence in a virtual environment have been explored by many in different ways. For example, presence due to media has previously been reviewed as a combination of:

• Perceptual immersion and psychological immersion (Biocca and Delaney, 1995Lombard and Ditton, 1997).

• Perceptual realism and social realism (Lombard and Ditton, 1997).

• Technology and human experience (Steuer, 19921995).

• Proto-presence, core-presence, and extended-presence (Waterworth and Waterworth, 2006).

• Place illusion and plausibility illusion (Slater, 2009).

To summarize, the two main factors that contribute to the illusion of reality due to media are (1) Perceptual Illusion: the continuous stream of sensory input from a media, and (2) Psychological Illusion: the continuous cognitive processes with respect to the perceptual input, responding almost exactly how the mind would have reacted in Primary Reality. Virtual reality systems create highest levels of illusion simply because it can affect more senses and help us experience the world as if we were inside it with continuous updated sensory input and the freedom to interact with virtual people or objects. However other forms of media, like a movie (where the sensory input is merely audio-visual and there is no means to interact with the reality presented) can still create a powerful illusion if it manages to create a stronger Psychological Illusion through its content (for example a story related to one’s culture or past experiences, would excite the memory and emotional aspects). One of the obvious examples illustrating the strength of Perceptual illusion is a media that enforces stereoscopic view enhancing our depth perception (the illusion works due to the way our visual perception would work otherwise, without a medium). The resultant of the two, Perceptual Illusion and Psychological Illusion evokes an illusion of reality in the mind, although subjectively varying for each person – in strength and experience.

The Concept of “Evoked Reality”

We know that it’s not directly presence that we create but rather an illusion in our minds as a result of which we experience presence. When we use virtual reality systems and create convincing illusions of reality in the minds of users, they feel present in it. This illusion of reality that we evoke through different means in order to enable the experience of presence is what we intend to call “Evoked Reality (ER).” To explore this experience of presence we must first better understand what ER is.

As deduced earlier, all the factors influencing presence would essentially be categorized as Perceptual Illusion and Psychological Illusion. We believe that every media in a way has these two basic elements. Thus ER is a combined illusion of Perceptual Illusion and Psychological Illusion. This combined spatiotemporal illusion is what evokes a different reality in our minds (Figure 1) inducing presence.FIGURE 1

Figure 1. Spatiotemporal illusion due to mediation: reality so evoked generates the experience of presence

Evoked Reality

Even though the terms like telepresence and virtual reality are very recent, their evidence can be traced back to ancient times. The urge to evoke reality different from our Primary Reality (real world reality) is not at all new and can be observed through the evolution of artistic and scientific media throughout history. “When anything new comes along, everyone, like a child discovering the world, thinks that they’ve invented it, but you scratch a little and you find a caveman scratching on a wall is creating virtual reality in a sense. What is new here is that more sophisticated instruments give you the power to do it more easily. Virtual Reality is dreams.” Morton Heilig. (as quoted in Hamit, 1993, p. 57).

From Caves to CAVEs

Since the beginning of civilizations, man has always tried to “express his feelings,” “convey an idea,” “tell a story” or just “communicate” through a number of different media. For example, the cave paintings and symbols that date back to prehistoric times may be considered as one of the earliest forms of media used to convey ideas. As technology progressed media evolved as well (Figure 2) and presently we are on the verge of extreme possibilities in mediation, thus equivalent mediated presence.FIGURE 2

Figure 2. Evolution of media: from caves to CAVEs

We all like to experience presence different from our everyday happenings. To do so, we basically find methods to create an illusion of reality different from the reality that we are familiar with. With the help of different media we have already succeeded to evoke a certain amount of presence and we further aim for an optimum level – almost similar to our real world. Every form of mediation evokes a different kind of illusory reality and hence different degrees of presence. In the early examples of research in presence, studies were conducted based on television experiences before Virtual Reality became a more prominent field of research (Hatada and Sakata, 1980). While some types of media evoke mild illusion of presence, highly advanced media like Virtual Reality may evoke stronger presence. “But we must note that the basic appeal of media still lies in the content, the storyline, the ideas, and emotions that are being communicated. We can be bored in VR and moved to tears by a book” (Ijsselsteijn, 2003). This is precisely why the reality evoked (by media) in one’s mind depends greatly on the eventual psychological illusion, although it may have been triggered initially by a perceptual illusion. Media that could evoke mild or strong presence may range from simple paintings to photos to televisions to films to interactive games to 3D IMAX films to simulation rides to immersive Virtual Reality systems.

Evoked Reality

Evoked Reality is an illusion of reality, different from our Primary Reality (Physical Reality as referred in previous studies). ER is a transient subjective reality created in our mind. In the case of ER due to media, the illusion persists until an uninterrupted input of perceptual stimuli (causing perceptual illusion) and simultaneous interactions (affecting the psychological illusion) continue to remain. The moment at which this illusion of ER breaks due to an anomaly is when we experience what is called a “Break in Presence (BIP)” (Slater and Steed, 2000Brogni et al., 2003). Thus a BIP is simply an immediate result of the “Break in Reality (BIR)” experienced. Different kinds of media can evoke realities of different qualities and different strengths in our minds for different amount of time. It’s an illusion of space or events, where or during which we experience a sense of presence. Thus, it is this ER in which one may experience Evoked Presence (EP).

Evoked Presence

Depending on the characteristics of ER, an experience of presence is evoked. To be more specific this illusion of presence created by ER, we would like to refer to as EP. In this paper, the term “EP” would imply the illusion of presence experience (the sense of presence), while the term “presence” would be reserved for experience of presence in its broad sense (real presence and the sense of presence). EP is the spatiotemporal experience of an ER. We could say that so far it’s through the media like highly immersive virtual reality systems, that we were able to create ER that could evoke significantly strong EP.

Media-Evoked Reality and Self-Evoked Reality

As we saw before, ER is a momentary and subjective reality created in our mind due to the Perceptual Illusion and Psychological Illusion imposed by a media. It is clear that due to ER induced through media like Virtual Reality we experience an EP. This illusion of reality evoked through media, we would like to call “Media-Evoked Reality” or Media-ER.

As mentioned earlier, it’s not just through the media that one can evoke an illusion of reality. The illusion can as well be endogenously created by our mind evoking a seemingly perceivable reality; whether merely observable or amazingly deformable; extremely detailed or highly abstract; simple and familiar or bizarrely uncanny. Thus to fully comprehend the nature of presence, we must study this category of ER that does not rely on media. In fact, we always or most often undergo different types of presence without mediation. Sanchez-Vives and Slater (2005) proposed that the concept of presence is sufficiently similar to consciousness and that it may help to transform research within domains outside Virtual Reality. They argue that presence is a phenomenon worthy of study by neuroscientists and may help toward the study of consciousness. As rightly put by Biocca (2003), where do dream states fit in the two pole model of presence (Reality-Virtuality Continuum)? The psychological mechanisms that generate presence in a dream state have to be at least slightly different than psychological mechanisms that generate presence in an immersive, 3D multimodal virtual environment. Dreaming, according to Revonsuo (1995) is an organized simulation of the perceptual world and is comparable to virtual reality. During dreaming, we experience a complex model of the world in which certain types of elements, when compared to waking life, are underrepresented whereas others are over represented (Revonsuo, 2000). According to LaBerge (1998), theories of consciousness that do not account for dreaming must be regarded as incomplete. LaBerge adds, “For example, the behaviorist assumption that ‘the brain is stimulated always and only from the outside by a sense organ process’ cannot explain dreams; likewise, for the assumption that consciousness is the direct or exclusive product of sensory input.” It is very clear that one can think, imagine, or dream to create a reality in his mind without the influence of any media whatsoever. This reality evoked endogenously, without the help of an external medium, we would like to call “Self-Evoked Reality” or Self-ER (implying that the reality evoked is initiated internally by the mind itself).

Ground-breaking works by Shepard and Metzler (1971) and Kosslyn (19801983) in the area of Mental Imagery provide empirical evidence of our ability to evoke images or imagine stimuli without actually perceiving them. We know that Perceptual and Psychological Illusion are factors that affect Media-ER and corresponding EP. We believe that Self-ER essentially has Psychological Illusion for which the Perceptual element is generated internally by our mind. By generally overlooking or occasionally completely overriding the external perceptual aspects (sensorimotor cues), our mind endogenously creates the Perceptual Illusion required for the ER. It’s evident in the case of dreaming which according to LaBerge (1998), can be viewed as the special case of perception without the constraints of external sensory input. Rechtschaffen and Buchignani (1992) suggest that the visual appearance of dreams is practically identical with that of the waking world. Moreover, Kosslyn’s (19942005) work show that there are considerable similarities between the neural mappings for imagined stimuli and perceived stimuli.

Similar to Media-ER, one may feel higher or lower levels of presence in Self-ER, depending on the reality evoked. A person dreaming at night may feel a stronger presence than a person who is daydreaming (perhaps about his first date) through an on-going lecture with higher possibilities of BIRs. According to Ramachandran and Hirstein (1997) we occasionally have a virtual reality simulation like scenario in the mind (although less vivid and generated from memory representations) in order to make appropriate decisions in the absence of the objects which normally provoke those qualities. However, the vividness, strength, and quality of this internally generated illusion may vary significantly from one person to another. For example, the intuitive “self-projection” phenomenon (Buckner and Carroll, 2007; personal internal mode of mental simulation, as they refer to it) that one undergoes for prospection will certainly differ in experience and qualia from another person. It is a form of Self-ER that may not be as strong or prolonged as a picturesque dream, but strong enough to visualize possible consequences. It is clear that ER is either the result of media or induced internally. This dual (self and media evoking) nature of ER directs us toward a fresh perceptive – three poles of reality.

Three Poles of Reality

As we move further into the concept of ER and EP, we would like to define the three poles of reality to be clearer and more objective in the explanations that follow. Reality, as discussed earlier (in subsection Simulated Reality), has always been a term interpreted with multiple meanings and theories. To avoid confusion we would like to use an impartial term – “Primary Reality,” which would refer to the “experience” of the real world (or what we call physical world). It is the spatiotemporal reality in our mind when we are completely present in the real world. It would mean that any reality other than Primary Reality is a conscious experience of illusion of reality (mediated or non-mediated), or more precisely – ER.

Presence and Poles of Reality

Inherited from early telerobotics and telepresence research, the two pole model of presence (Figure 3) suggests that presence shifts back and forth from physical space to virtual space. Research on presence has been dominated ever since by this standard two pole psychological model of presence which therefore requires no further explanation.FIGURE 3

Figure 3. The standard two pole model of presence

Biocca (2003) took the study of presence model one step further. According to the model he proposed, one’s spatial presence shifts between three poles of presence: mental imagery space, the virtual space, and the physical space. In this three pole graphic model, a quasi-triangular space defined by three poles represented the range of possible spatial mental models that are the specific locus of an individual user’s spatial presence. His Model of presence attempted to offer a parsimonious explanation for both the changing loci of presence and the mechanisms driving presence shifts. Though the model explained the possibilities of presence shifts and varying levels of presence, it is vague about certain aspects of reality. It did not clarify what happens when we experience an extremely low level of presence (at the center of the model). How or why do we instantly return to our Primary Reality (in this model – Physical Space) as soon as a mediated reality or a DR is disrupted (Even though we may have entirely believed to be present in the reality evoked during a vivid dream)? Moreover it took into account only the spatial aspects but not the temporal aspects of shifts in presence.

We would like to define three poles of reality from the perspective of ER. The Three Pole Reality Model (Figure 4) may help overcome the theoretical problems associated with presence in the standard two pole model of presence as well as the model proposed by Biocca. According to us it’s the shifts in the type of reality evoked that create respective shifts in the level of presence evoked. For example if one experiences a highly convincing ER during a virtual reality simulation, he/she would experience an equivalently strong EP until a BIR occurs. The three poles of reality that we define are:

• DR (Threshold of Self-ER)

• Primary Reality (No ER)

• SR (Threshold of Media-ER)FIGURE 4

Figure 4. Three pole reality model

Primary reality

Primary reality refers to the reality of our real world. In Primary reality, the experience evoking stimulation arrives at our sensory organs directly from objects from the real world. We maintain this as an ideal case in which the stimulus corresponds to the actual object and does not deceive or misinform us. For instance, imagine yourself running from a tiger that is chasing you. It’s very near and is about to pounce on you. You scream in fear, and wake up to realize that you are safe in your bed, like every morning. You know for sure that this is the real world and the chasing tiger was just a part of the DR that your mind was in, some time before. So, Primary Reality is our base reality to which we return when we are not in any ER. In other words, when a BIR occurs, we come back to Primary Reality. Thus, as we can see in Figure 5, any point of reality other than Primary Reality is an ER. We could say that it’s this Primary Reality that we rely on for our everyday activities. It’s the reality in which we believe that we live in. Our experiences in this Primary Reality may form the basis for our experiences and expectations in an ER. For example, our understanding of the real world could shape how we experience presence in an immersive virtual reality environment, or even in a Dream. We could suppose that it’s the Primary Reality in which one believes this paper exists, or is being read.FIGURE 5

Figure 5. Three poles of reality: evoked reality constantly shifts between them

Simulated reality

In the case of Media-ER, an experience similar to Primary Reality is attempted to be achieved by interfering with the stimulus field, leading to an illusion of reality. For example virtual reality uses displays that would entirely mediate our visual perception in a manner that our head or eye movements are tracked and updated with appropriate images to maintain this illusion of receiving particular visual stimuli from particular objects. SR would be the most compelling and plausible reality that could ever be achieved through such mediations. It would be the reality evoked in our mind under the influence of a perfectly simulated virtual reality system. It’s the ultimate level that virtual reality aims to reach someday. At the moment an immersive virtual reality system, like flight simulators would be able to create ER considerably close to this pole. Its effectiveness is evident in the fact that pilots are able to perfectly train themselves being in that ER created by the simulator, helping them eventually to directly pilot a real plane. However, in the hypothetical condition of a perfectly SR our mind would completely believe the reality evoked by the simulation medium, and have no knowledge of the parent Primary Reality (Putnam, 1982Bostrom, 2003). In this state, it would be necessary to force a BIR to bring our mind back to Primary Reality. A Perfect SR is the Media-ER with strongest presence evoked and will have no BIRs.

Dream reality

In the case of Self-ER, the external perceptual stimuli are imitated by generating them internally. DR is an ideal mental state in which we almost entirely believe in the reality experienced, and accept what is happening as real. It does not return to the Primary Reality unless a BIR occurs. For instance, in the case of our regular dreams, the most common BIR would be “waking up.” Although internally generated, dream states may not be completely divorced from sensorimotor cues. There can be leakage from physical space into the dream state (Biocca, 2003). The experienced EP during a strong Dream can be so powerful that even the possible anomalies (causing BIRs) like external noises (an alarm or phone ringing) or even elements from physical disturbances (blowing wind, temperature fluctuations) may be merged into the DR, so as to sustain this ER for as long as possible. A Perfect DR is a Self-ER with the strongest presence evoked and will have no BIRs (similar to SR on the media side).

Presence Shifts and Presence Threshold

We are often under the effect of either Media or Self-ER. Imagine that we are not influenced by any mediation, nor any kind of thoughts, mental imagery, or dreams and our mind is absolutely and only conscious about the Primary Reality. In such an exceptional situation we would supposedly feel complete presence in the Primary Reality. Thus we presume that this perfect Primary Reality-Presence (or “real presence” as some may call) is the threshold of presence one’s mind may be able to experience at a point of time. It is clear that we can experience presence either in Primary Reality or in an ER. We cannot consciously experience presence in two or more realities at the same time, but our mind can shift from one reality to another voluntarily or involuntarily, thus constantly shifting the nature and strength of the presence felt. As pointed out by Garau et al. (2008), presence is not a stable experience and varies temporally. They explain how even BIPs could be of varying intensities. They also try to illustrate using different presence graphs the phenomenon of shifting levels of presence with the course of time and how subjective the experience is for different participants. Media like virtual reality aims to achieve the Presence Threshold at which one’s mind might completely believe the reality evoked. Though we have not however achieved it, or may never do, theoretically it’s possible to reach such a level of SR. Similarly if one experiences a Perfect Dream without any BIR, he/she would be at this threshold of presence exactly like being in the Primary Reality. SR and DR are the two extreme poles of reality at which the EP is at its threshold. These presence shifts due to the shifting of reality between these poles is something that we seldom apprehend, although we always experience and constantly adapt to them. In the following section we attempt to represent this phenomenon with a schematic model that would help us examine presence and reality from a clearer perspective.

Reality-Presence Map

Based on the three poles of reality and Presence Threshold we would like to propose the Reality-Presence Map (Figure 6). This map is a diagram of the logical relations between the terms herein defined. At any point of time one’s mind would be under the influence of either a Media-ER or a Self-ER when not in the Primary Reality (with no ER at all). Between the poles of reality, ER would constantly shift evoking a corresponding presence EP. As we can see in the map there is always a sub-conscious Parent Reality-Presence corresponding to the EP. This Parent Reality-Presence is very important as it helps our mind to return to the Primary Reality once the illusion of ER discontinues (or a BIR occurs). For a weaker EP, the Parent Reality-Presence is stronger (although experienced sub-consciously). When the ER manages to evoke very strong presence, the strength of Parent Reality-Presence drops very low (almost unconscious) and we start to become unaware of the existence of a Primary Reality; which is what an excellent immersive virtual reality system does. The shifting of presence is closely related to our attention. As soon as our attention from the ER is disrupted (predominantly due to interfering external perceptual elements), our attention shifts to the parent reality-presence sliding us back to Primary Reality (thus breaking our EP).FIGURE 6

Figure 6. Reality-presence map.

At the extreme poles, we would experience an Optimum Virtual Presence in a SR and similarly an Optimum Dream Presence in a DR. At these extreme points one may completely believe in the illusion of reality experienced almost or exactly like it is our Primary Reality, without the knowledge of an existing Parent Reality. At such a point, possibly a very strong BIR should be forced to bring one back to the parent Primary Reality. Experiencing a strong DR is one such example which many would relate to. During a very compelling but frightening dream, “waking up” acts as a very strong BIR, helping in the desperate attempt to leave the DR. After such a sudden and shocking change in reality most often our mind takes time to adjust back to the Primary Reality where everything would slowly turn normal and comforting.

Whenever there is an ER, the EP part of the presence (in the map) is what has our primary attention, and thus is the conscious part. Hence, the higher the EP, the lesser we are aware of our parent reality. Evidence of the sub-conscious Parent Reality-Presence can be observed in our experience of any media that exists today. Many studies have shown that in virtual environments, although the users behaved as if experiencing the real world, at a sub-conscious level they were certain that it was indeed “not” real. BIPs (that are used to measure presence) are in fact triggered by shifts in attention from the virtual world to the real world. For instance, virtual reality systems that help visually surround us completely with a virtual environment, elevates our presence (compared to a panorama view or television with visible frame boundaries) as our chances of shifting attention toward the real world drastically reduce in such higher levels of immersion (Grau, 2004Slater, 2009). Since ER is a subjective feeling, it can never be measured or even compared truthfully. This is the reason why we depend on the measurement of presence EP to determine if a system creates a stronger or weaker ER. Since the strength of presence itself is relative, the best way to measure is to compare between systems in similar context. “The illusion of presence does not refer to the same qualia across different levels of immersion. The range of actions and responses that are possible are clearly bound to the sensorimotor contingencies set that defines a given level of immersion. It may, however, make sense to compare experience between systems that are in the same immersion equivalent class” (Slater, 2009).

A major task for empirical consciousness research is to find out the mechanisms which bind the experienced world into a coherent whole (Revonsuo, 1995). This map provides a framework where the various experiences of ER could be mapped. Note that this map is not a “graph” that shows the strength of EP as directly proportional to the strength of ER. In fact it would help us represent every possible kind of ER as a point fluctuating between the two extreme poles of reality, with its respective strength of EP. We may refer to ER as stronger or weaker, when its qualia evoke stronger or weaker EP respectively. The Reality-Presence Map shows that if we can skillfully manipulate these qualia of ER (although subjective to each individual) bringing it closer to either of the two extreme poles, we may be able to evoke higher levels of EP. We should also note that, in order to introduce its basic concept, the Reality-Presence Map is presented here in a flattened two-dimensional manner. In the later sections we will illustrate how this map attempts to account for different experiences which were unable to be explained by previous presence models.

Subjectivity of Evoked Reality

As a matter of fact, the same mediation can create different subjective ER for different users depending on their personal traits. For example, two users reading the same book, or playing the same video game, or using the same Virtual Reality system would experience presence in an entirely different manner. EP (especially evoked by a medium) may be affected by one’s knowledge related to the context, degree of interest, attention, concentration, involvement, engagement, willingness, acceptance, and emotional attributes making it a very subjective experience. This is precisely why it is difficult to evaluate the efficiency of a particular Virtual Reality system by means of presence questionnaires. In fact many researchers confuse few of these terms above, with the concept of presence.

Therefore, to locate ER on the map, we have to examine “presence.” In fact finding reliable ways to measure presence has been a pursuit among many virtual reality and communication media researchers. In order to lead to testable predictions, we would rely on currently evolving measuring and rating systems, so as to determine an objective scale for presence (from Primary Reality to each extreme pole). Presently existing measuring techniques include questionnaires like “presence questionnaire” (Witmer and Singer, 1998Usoh et al., 2000), ITC-SOPI questionnaire (Lessiter et al., 2001), SUS questionnaire (Slater et al., 19941995), analysis of BIPs (Slater and Steed, 2000Brogni et al., 2003), objective corroborative measures of presence like psycho-physiological measures, neural correlates, behavioral measures, task performance measures (Van Baren and Ijsselsteijn, 2004), to mention a few. We can certainly predict the positions of different everyday experiences for a person in general (Figure 7); however it could be tested in the future only using above mentioned methods of measuring presence.FIGURE 7

Figure 7. An example range of Media-ER and Self-ER experiences mapped on reality-presence map, for an individual, that would occur at various points in time.

In virtual reality, distinction between “presence” and “immersion” has been made very clear previously in (Slater, 19992003). Though immersion (which is discussed extensively in the domain of virtual reality) is one of the significant aspects of EP, it falls under the technical faculty of a mediated system. “Immersion (in perceptual sense) provides the boundaries within which Place Illusion can occur” (Slater, 2009). Detailed aspects of presence related to immersive virtual reality are also discussed in (Slater et al., 2009). The characteristics like involvement, engagement, degree of interest, emotional response, may seem similar to presence, but are in fact different elements that may influence or be influenced by EP. The psychological impact of content, i.e., good and bad, exciting and boring, depends to a large extent on the form in which it is represented (Ijsselsteijn, 2003). Thus one of the most important aspects of Media-ER is its context. In most cases it forms a reference in one’s mind to how they may experience ER and hence the presence evoked. For example, in some contexts, especially in art and entertainment, it would invoke a “genre” that plays a major role in its communication. The context (whether artistic expression, communication, entertainment, medical application, education, or research) should be a core concern while designing a Virtual Reality System, in order to bring about a subjectively higher quality of ER. A descriptive account on the importance of context in Self-ER is given by Baars (1988). With examples of different sources and types (perceptual and conceptual) of contexts, he demonstrates how unconscious contexts shape conscious experience. In addition, he explains the importance of attention, which acts as the control of access to consciousness. Attention (in both Media-ER and Self-ER) can direct the mind toward or away from a potential source of qualia. The experience of an ER therefore depends also on the voluntary and involuntary characteristics of one’s attention.

According to the concept, our presence shifts continuously from one ER to another and does not require passing through Primary Reality to move from one side to another. This map does not provide a temporal scale per se. However in future (with the advancements in presence measurement techniques), the map can be used to trace presence at different times to study the temporal aspects of presence shifts.

Evoked Reality within Evoked Reality

There is an important question that arises now. How can we account for our thoughts or mental imagery experiences during VR simulations, games, movies, or most importantly books? It is the phenomena of experiencing Self-ER during a Media-ER experience.

Self-ER within media-ER

Whenever we experience an ER, our mind is capable of temporarily presuming it as the parent reality and reacting accordingly. The better the ER and stronger the EP, the easier it is for our mind to maintain the illusion. In such states Media-ER is experienced as a temporarily form of Primary Reality, and we are able to experience Self-ER within it. In fact that is the core reason why virtual reality systems and virtual environments work. This phenomenon is clearly displayed in such experiences, where the users require thinking, planning, and imagination in order to navigate in the virtual world, just like they would do in the real world. Below, it is demonstrated how this phenomenon may be represented with respect to the Reality-Presence Map (Figures 8 and 9). This scenario will ultimately be classified under Media-ER.FIGURE 8

Figure 8. An example of how Media-ER would temporarily act as a version of primary reality

Figure 9. An example of presence shift due to Self-ER within Media-ER (for e.g., thinking within a virtual environment).

Self-ER triggered during media-ER

“Self-ER within Media-ER” should be distinguished from the phenomenon of “Self-ER triggered during Media-ER.” This is similar to a well-known case of Self-ER – the phenomenon of mind-wandering that temporarily detaches us from the Primary Reality. It is otherwise known as “task unrelated thought,” especially with respect to laboratory conditions. Smallwood et al. (2003) define it as the experience of thoughts directed away from the current situation. It is in fact a part of (and closely related to) our daily life experiences (Smallwood et al., 2004McVay et al., 2009). Although studies on mind-wandering are principally focused on shifts between Self-ER and tasks relating to Primary Reality (falling under usual case of Self-ER experience – Figure 10), we propose that they are applicable to similar cases in Media-ER as well. It has been suggested that this involuntary experience may be both stable and a transient state. That means we can experience a stable EP during mind-wandering or an EP oscillating between the Self-ER, Media-ER, and the Primary Reality.FIGURE 10

Figure 10. The usual case of presence shift from primary reality to Self-ER

Therefore, when an unrelated Self-ER is triggered while experiencing a Media-ER (or when Self-ER within Media-ER traverse the presence threshold and becomes unaware of the Media-ER itself), it should be considered under the case of Self-ER (Figure 11).FIGURE 11

Figure 11. An example of presence shift toward Self-ER triggered during Media-ER.

Discussion

Our attempt was a novel idea, to fit together different concepts regarding presence into a single coherent graphical representation. Although this concept of ER and EP along with the proposed map provides us a simplified way to look at reality and presence, it raises plenty of questions. Can the experience of an altered state of consciousness (ASC) like hallucination, delusion, or psychosis due to mental disorders be a kind of Self-ER? Revonsuo et al. (2009) redefines ASC, as the state in which consciousness relates itself differently to the world, in a way that involves widespread misrepresentations of the world and/or the self. They suggest that, to be in an ASC is to deviate from the natural (world-consciousness) relation in such a way that the world and/or self tend to be misrepresented (as evident in reversible states like dreaming, psychotic episodes, psychedelic drug experiences, epileptic seizures, and hypnosis). According to Ramachandran and Hirstein (1997) we have internal mental simulations in the mind using less vivid perceptual attributes, in the absence of the regular external sensory inputs. If they possessed full-strength perceptual quality, that would become dangerous leading to hallucinations. They argue that in cases like temporal lobe seizures, this illusion (Self-ER) may become indistinguishable to real sensory input losing its revocability and generating incorrect sense of reality (creating a permanent ER situation that makes it difficult to return to Primary Reality). So can hallucinations due to Self-ER be compared to Augmented Reality due to Media-ER?

In contrast to Presence, is there an “Absence” and do we experience that? If so, how? Can it be compared to a dreamless sleep? Can Presence Threshold itself be subjective and differ from person to person? With reference to the Reality-Presence Map, is there a possibility of an experience analogous to uncanny valley when ER is nearest to the two extreme poles? Is this the reason why many experience anomalies during exceptionally vivid nightmares or lucid dreams? Similarly on the Media-ER side, can simulator sickness due to inconsistencies during virtual reality simulations be compared to this phenomenon? Other than the obvious difference between Media-ER and Self-ER that was discussed before, they have another main differentiation. In most cases of Media-ER, multiple users could share the experience of a common ER at the same time (naturally, with subjective differences, especially due to psychological illusion). While in the case of Self-ER, every person’s mind experiences unique ER. Thus a Dream is typically an individual experience (as far as our present technological advancements and constraints suggest), while SR may be shared.

Furthermore, the Reality-Presence Map helps us investigate into potential ideas on Reality, for instance the possibility of Simulation within a Simulation (SWAS). The Map could be extended to and be applicable for any level of reality, in which we believe there’s a Primary Reality – the base reality, to which we return to in case of absence of any form of ER. Let’s imagine that someday we achieve a perfect SR. As per our proposition, one’s mind would accept it as the Primary Reality as long as the experience of presence continues (or till a “BIR” occurs). It would imply that at such a point, one can experience presence exactly as in the Primary Reality. In this perfect SR if one experiences Media-ER (e.g., virtual reality) or Self-ER (e.g., dream), as soon a BIR occurs they return back to it since it’s the immediate Parent Reality. Figure 12 attempts to illustrate such a situation with DR and SR as two orthogonal Poles of Reality. Similarly in the Self-ER side, one’s mind could experience a Dream within a Dream (DWAD). When one wakes up from such a dream, he could find himself in the parent DR from which he would have to wake up again into the Primary Reality. Can this be how people experience such false awakenings [a hallucinatory state distinct from waking experience (Green and McCreery, 1994)]? Figure 13 attempts to illustrate such a situation of DWAD.FIGURE 12

Figure 12. Simulation within a simulation

Figure 13. Dream within a dream

In fact it makes us curious about the even bigger questions. Can there be an ultimate reality beyond Primary Reality or even beyond the scope of this map. The Simulation argument claims that we are almost certainly living in a computer simulation (Bostrom, 2003), in which case what we believe to be our Primary Reality might itself be a SR [similar to Brains in a vat scenario (Putnam, 1982)]. Metzinger (2009) proposes that our experience of the Primary Reality is deceptive and that we experience only a small fraction of what actually exists out there. He suggests that no such thing as “self” exists and the subjective experience is due to the way our consciousness organizes the information about outside world, forming a knowledge of self in the first person. He claims that everything we experience is in fact a SR and the on-going process of conscious experience is not so much an image of reality as an “ego tunnel” through reality. So, is our Primary Reality in fact the base reality? Or are we always under an ER of some kind? Figure 14 attempts to put together different levels of reality as a Reality Continuum. It would make us wonder if it’s probable, to how many levels would one be able to go? Do we already visit them unknowingly through our dreams? Would the levels of reality in the figure be represented as a never ending fractal structure? In any case, will we be able to understand someday all these aspects of our experience of reality?FIGURE 14

Figure 14. Reality continuum (illustrating the levels of reality).

Conclusion

In this paper we explored presence and different elements that contribute to it. Presence is not just “being there” but a combination of multiple feelings and most importantly “experiencing the reality.” The two main factors affecting presence due to mediation are Perceptual Illusion and Psychological Illusion. These factors evoke an illusion of reality in our mind in which we feel presence. We are constantly subjected to such illusions of reality, during which we experience presence differently from that of our apparent real world. This illusion of reality is called ER.

Evoked Reality is not just media-evoked but can also be self-evoked. Media-ER may range from the mild effect of a painting to an extremely plausible immersive Virtual Reality experience while a Self-ER may range from a simple thought to an exceptionally believable DR (the strength of ER may not necessarily be in the same order, as it depends on one’s qualia and personal characteristics). This dual nature of ER led us to define three poles of reality: primary reality – the unaltered and unmediated Real World, SR – the ultimate Media-ER (a perfect Virtual Reality condition) and DR – the ultimate Self-ER (a perfect dream condition). Thus ER is an illusion of reality formed in our mind, which is different from Primary Reality. It’s a combined illusion of space and events, or at least one of them. It is in this ER, one would experience presence. Thus EP is the spatiotemporal experience of an ER.

The proposed Reality-Presence Map attempts to graphically illustrate the concept of ER and EP. This map provides a framework where the various experiences of ER could be mapped. The subjectivity of ER qualia and how these subjective factors affect Media-ER and EP were explained. The idea of Presence Threshold was also explored which formed the basis for different levels of EP and temporal Presence Shifts. Different possibilities like SWAS and DWAD conditions were discussed with respect to the proposed model. However certain elements still demand clarifications to fill in the theory. The concept presented here is an inception of a potential future research. We believe that ER and the proposed Reality-Presence Map could have significant applications in the study of presence and most importantly in exploring the possibilities of what we call “reality.”

The full report including references can be found here

Virtual Realities' Bodily Awareness, Total Immersion & Time Compression Affect

VR and Time Compression- A Great Example of How Deeply Immersion Works

Time flies when you’re having fun. When you find yourself clock-watching in a desperate hope to get something over and done with, it often feels like the hands of the clock are moving like treacle. But when you find yourself really enjoying something,

It’s no surprise at all to hear that this phenomenon is particularly prevalent when it comes to virtual reality. After all, we all know that the more immersive the experience, the much more engaging and enjoyable it often tends to be. Researchers have in fact given this case of technology warping our sense of time a name: time compression.

Intel HTC virtual Reality Accident Reduction
We don't only get lost in the concept of time, but we feel the benefits too!

The Marble Game Experiment

Grayson Mullen and Nicolas Davidenko, two Psychology professors, conducted a survey in 2020 to see if there was any measurable scientific proof to this widely-reported phenomenon. And indeed there was!

They invited 41 undergraduate university students to play a labyrinth-like game, where the player would rotate a maze ball to navigate the marble inside to the target. One sample group played the game via a conventional monitor, while the other played within a virtual reality environment. The participants were asked to stop playing and press a yellow button at the side of the maze once they had sensed five minutes had passed.

With all the responses timed and recorded, the study ultimately found that the students who played the VR version of the labyrinth game pushed the button later than their conventional monitor counterparts, spending around 28.5% more real time playing!

Why does it happen?

We don’t exactly know how VR locks us in a time warp. There’s no denying that video games in general can be extremely addictive for some players. Even conventional games are so easy to get immersed into that you could forget whereabouts in the day you are.

Palmer Luckey, founder of Oculus, thinks it could boil down to the way we rely on the environment around us to sense the passage of time. Here is what he said during an interview at the 2016 Game Development Conference:

“I think a lot of times we rely on our environments to gain perceptual cues around how much time is passing. It's not just a purely internal thing. So when you're in a different virtual world that lacks those cues, it can be pretty tough...You've lived your whole life knowing roughly where the sun is [and] roughly what happens as the day passes…

In VR, obviously, if you don't have all those cues — because you have the cues of the virtual world — then you're not going to be able to make those estimates nearly as accurately.”

When you play a game on a conventional platform such as a console or a PC, you’ve got other things going on around you to give you a good indication of what the time is, like the sun and the lighting, and any background noises (e.g. the sounds of rush-hour traffic). With virtual reality, you block all this out, so you can’t rely on these to help you tell the time anymore.

What does this mean for immersion & us?

Time compression isn’t just relevant when it comes to enjoying entertainment: we can also use it to help people in other contexts. For example, Susan M Schneider led a clinical trial exploring the possibility of incorporating virtual reality experiences into chemotherapy sessions. This medical procedure can be very stressful for cancer patients, but the results of the trial found clear evidence for the VR simulation reducing anxiety levels and perceived passage of time, acting as a comforting distraction from the chemotherapy.

But despite all these potential benefits, we can’t forget the elephant in the room of gaming addiction. The time-warping effect of virtual reality also sadly means it’s easier for players to spend hour after hour stuck in their virtual world, which sacrifices their health as well as their time! Not only does this increase the risk of motion sickness, but it can also throw off your natural body clock, negatively affecting how well you sleep and thus your overall wellbeing.

It kind of sounds like one step away from the Lotus Casino from Rick Riordan’s Percy Jackson series - a casino where time never stops and nobody ever wants to leave. In their study, Mullen and Davidenko urge game developers not to take a leaf from the Lotus Eaters’ book. While a near-addictive  feeling in your audience is a positive sign of a successful immersive application, it shouldn’t be something you exploit to put them at risk.

Here are a couple of recommendations to help players know when it’s time to stop:

Bibliography

Miller, R. (2016). Oculus founder thinks VR may affect your ability to perceive time passing. [online] The Verge. Available at: https://www.theverge.com/2016/3/17/11258718/palmer-luckey-oculus-time-vr-virtual-reality-gdc-2016

Mullen, G. & Davidenko, N. (2021). Time Compression in Virtual Reality. Timing & Time Perception. 9 (4). pp. 377–392.

Schneider, S.M., Kisby, C.K. & Flint, E.P. (2011). Effect of Virtual Reality on Time Perception in Patients Receiving Chemotherapy. Supportive Care in Cancer. 19 (4). pp. 555–564.

To view the full report on


Holospatial delivers an alternate method of delivery for any VR or non-VR application through advanced interactive (up-to 360-degree) projection displays. Our innovative Portal range include immersive development environments ready to integrate with any organisational, experiential or experimental requirement. The Holospatial platform is the first ready-to-go projection platform of it's type and is designed specifically to enable mass adoption for users to access, benefit and evolve from immersive projection technologies & shared immersive rooms.

Nvidias' Immersive Omniverse of Virtual Tools Making Immersion a Reality- Expanding the industry

Nvidia has been venturing into the Omniverse for sometime now. It's a set of tools for software developers aimed at helping them create a "metaverse" of three-dimensional virtual worlds and a concurrent set of facilities to help you create and manage your metaverse.

Obviously the metaverse has been hitting the headlines, despite the fact this only truely exists in your head- but that's a different thought path altogether. The digital metaverse or omniverse in Nvidias' case is right in the heart of Portal territory with most of these platforms and immersive innovations all ploying away at the immersive opportunity which however fast organisations realise it, IS going to be the future and if you look even now at our reliance on digitised 2D information/platforms it's a pretty easy link to realise that stepping into 3D interfaces is where the digital "verse" (whichever pre-fix you choose) is really going to crank-up.

At the company's annual technology conference, Nvidia released Omniverse Enterprise- which has been on the agenda all year. It will start at $9,000 per year and be sold by partners such as Dell Technologies and Lenovo Group- two majorly significant distributors of Nvidia chips.

Omniverse Application

The Omniverse tools help various apps used to create three-dimensional worlds out of current technologies, software tools and applications. Without doubt, it's a bold step into the right (and the immersive) direction.

In an interview with Reuters, Richard Kerris, vice president of the Omniverse platform at Nvidia, called it "the plumbing of the virtual worlds. That's what we've built. And that's what we're building with all of these partners."

At current this is a business behind-the-scenes type game. And it's technologies such as Portals which allow a totally different delivery medium for such tech advances. The Omniverse is a step towards a Portal-world.

Kerris told Reuters that Nvidia worked with more than 700 companies to test and develop the software, including firms like telecommunications equipment maker Ericsson, which used the software to create a "digital twin" of a city that it used to test cell phone signal coverage before rolling out physical trucks to install real-world antennas.

Earlier this month, Wells Fargo analyst Aaron Rakers wrote that software and other tools for creating virtual worlds could be a $10 billion market opportunity for Nvidia over the next five years - especially as firms like Meta Platforms Inc (the new/old Facebook), entice people to spend more time in what it calls the metaverse.

Nvidia's stock market value has surged $191 billion since Facebook's capital expenditure announcement on Oct. 25, a two-week gain that is nearly as large as rival Intel's entire market capitalization of $209 billion an indicator of just how significant the 3D-verse will eventually be.

The "verse" is the future, and more so it is the prized asset which the largest and most forward-thinking companies and investors are looking at to add true value to their organisations, processes all of which to ultimately benefit it's people and wider audience.