During our meetup on March 29th, 2022, Stephen Brewster presented the current state of research of ViAjeRo — a project aiming to radically improve passenger experiences by utilizing XR.

About Stephen Brewster

Stephen Brewster is a professor of Human-Computer Interaction in the Department of Computing Science at the University of Glasgow and leads the Multimodal Interaction Group. His research focuses on multiple sensory modalities and control mechanisms (particularly sound, haptics, and gestures) to create a rich and natural interaction. With an experimental focus, he applies perceptual research to practical situations.

Project ViAjeRo

Aimed at radically improving passenger experiences by utilizing XR, ViAjeRo explores a range of AR and VR scenarios for different modes of transportation, focusing on entertainment, work, and collaboration activities. Through research, many challenges have been found and “significant progress has been made, but there are still many issues to solve,” said Brewster. The main research challenges revolve around interaction in confined spaces, social acceptability, and motion sickness.

Benefits of XR in Transit

XR has the potential to alter, augment, or replace the physical space of passengers in transit, whether that be a plane or a car. Using phones, tablets, or laptops is somewhat limited compared to the setup people might have at home or the workplace. With AR, interesting interactive elements can be added to the environment, and with VR, the environment can be entirely replaced. As Brewster put it, “instead of being stuck in that economy plane seat, squished between two people, you could be in outer space,” or the sensed motion of the vehicle can be incorporated into the XR experience, such that “30kph feels like 60kph.”

Activities: Entertainment, Work, and Collaboration

One of the scenarios created within entertainment is VR Cinema at-a-distance, enabling passengers to gather to watch a movie and have a shared experience during long-distance traveling in the backseat of a car or plane.

Another scenario is playing an AR zombie game as a car passenger. To play the game, the passenger wears a headset and, with the controller, detects vehicles on the road that supplies ammo or health as they look out for and shoots zombies.

AR Zombie Game

A typical workspace setup consists of one or multiple monitors, a keyboard, and a mouse or trackpad, which, in transit, is not feasible. In VR, however, a large workspace can be created, and all that is needed is a headset. The video below shows different interactions tested for a desktop setup in VR.

A collaboration scenario, suggested Brewster, can involve working remotely or having “fun with my family members” who might be at home. In the video below, two people collaborate in an underwater scene.

When creating and testing these scenarios, three main research challenges emerged: (1) interaction in confined spaces, (2) social acceptability, and (3) motion sickness.

Research Challenges

1. Interaction in Confined Spaces

VR interactions typically involve moving the body about in a space, which is not feasible in confined spaces like cars or plane seats. Questions are:

• How can high bandwidth-rich interactions be supported in limited situations?
• Can smaller movements be mapped onto a larger space?
• How can the physical environment be utilized?

Since large movements involving controllers or hand tracking are not optimal for confined spaces, the objective is to find viable alternatives. For one of the experiments, the physical surfaces in the environment (e.g., arm leans, tray tables) were utilized to test the possible types of movements and most effective surfaces for the interaction.

2. Social Acceptability

Generally, the actions performed in a plane or train environment are different from those performed comfortably in a home environment. The question is how to design interactions that people are comfortable performing in a confined space among strangers. To that end, experiments were conducted to find:

• What are preferred screen layouts in AR?
• What are socially acceptable actions?
• When in VR, what is deemed necessary information from the outside environment?

In testing preferred screen layouts, an experiment entailed asking participants to place screens in VR as if they are in an AR scenario (as seen in the images below). The results indicated that the preferred screen layout varies across the mode of transportation. For example, participants placed the screens low on two rows in the subway and train scenarios, while the screens were placed in a vertical column in the plane scenario. Brewster added that the participants placed the screens much lower than expected in the subway scenario, likely to avoid staring directly at the passengers across from them.

Screen layout, subway

Screen layout, train

Screen layout, plane

Similarly, when testing for the type of interactions people deem socially acceptable was shown to depend on the mode of transportation. Foot-tapping, for example, was found to be “not acceptable at all” in face-to-face situations, such as on a train. Ultimately, interactions that potentially involve accidental body contact with another passenger are deemed not socially acceptable.

3. Motion Sickness

Motion sickness is induced when there is a mismatch between the vestibular and visual systems. Immersive content induces motion sickness due to being removed from the visual cues the vestibular system perceives. With VR Cinema at-a-distance, for example, motion sickness is induced because it involves looking at head-locked content without being able to see the outside environment. This particular challenge has two parts: detecting when the mismatch starts and then mitigating it.

In detecting motion sickness, various sensors were used for capturing physiological data, from blood pressure to body temperature. Together with psychologists, it was found that a change in skin temperature, particularly on the face and fingertips, is a key marker for motion sickness. Two solutions for mitigating motion sickness were tested: brain stimulation and peripheral visual cues.

Brain stimulation methods included neurostimulation, specifically tACS (transcranial Alternating Current Stimulation), and galvanic vestibular stimulation. Each method involves stimulating parts of the ear, bringing noise into the vestibular system so that the brain relies on the visual system instead. The second solution, peripheral visual cues, involves creating an alignment of the visual cues and the vestibular cues. In the video below, a 360 video is placed in the center, and the movement of the outside environment is seen in the periphery.

In another implementation, the movement of the video content and the outside environment is synchronized.

During the experiments, participants would rate their motion sickness. The neurostimulation experiments are still ongoing, explained Brewster, but the initial results show that “people last longer” when mitigations are provided, indicating “that it has an effect.” As for the peripheral visual cues, the results show that the effects vary from person to person. Having presented the current state of the research, Brewster concluded that eventually, all of the effective methods will be combined in such a way that there is no conflict with immersion.

Q&A Session

A participant pointed out that motion sickness is commonly reported when first starting to play VR games. Since motion sickness becomes less of an issue for those who play games in VR more frequently than those that do not, Brewster was asked whether one of the studies compared these two groups of people. “There is actually quite a debate about whether increased exposure reduces it in general or whether exposure to a particular type of thing reduces it for that type of thing,” responded Brewster. There is some evidence that playing the same game for long periods decreases motion sickness for that particular game. Since passenger experiences differ in the mode of transportation, the duration, and the type of tasks performed in XR, “whether you would build up enough immunity to it is an interesting question,” said Brewster.

Another participant asked whether one of the studies tested for motion sickness in AR? Brewster gave the example of the AR zombie game, during which there was no motion sickness reported. There are only a few virtual elements, so there is no motion sickness if enough of the outside environment is in view. However, when crammed in a plane seat, a more suitable scenario is in VR.

Concerning the VR experiments that utilized peripheral visual cues to mitigate motion sickness, Brewster was asked if only the “fields and trees” had been tested or if abstract shapes had also been tested. A few different backgrounds had been tested, responded Brewster, such as white dots resembling stars on a black background, adding that there is an “interesting impact depending on the type of background. The VR video with naturalistic peripheral cues “is a bit disturbing because it is quite visually attention-grabbing.” The peripheral cues in the background should mitigate motion sickness and, at the same time, should not “break immersion with the task that you are doing in the center,” explained Brewster.