Elucidating brain processing systems adapted to the real world via VR technology enabling free construction of virtual worlds

SHIGEMASU Hiroaki

Specialized field: Visual psychophysics, Cognitive Neuroscience, Cognitive ergonomics

Human beings see a world with three-dimensional depth based on two-dimensional images projected onto the retina. We don't just see the retinal images per se; we perceive three-dimensional imagery not by directly observing what is projected onto the retina, but rather through the brain's processing to infer and reconstruct a three-dimensional world. The complex information processing systems of the brain are crucial for receiving information from the outer world via the eyes, ears, and other sensory organs, linking these inputs with appropriate action, and adapting to the real-world environment. Professor Hiroaki Shigemasu is elucidating the nature of this information processing in the brain by using cutting-edge technologies, like virtual reality (VR) and functional magnetic resonance imaging (fMRI). He is trying to grasp the universal characteristics of human beings with these approaches.

Clarifying the functions of each brain region through experiments using MRI

The brain's information processing plays a major role in human perception. However, we still lack a complete picture of how brain areas respond to different perceptions. Prof. Shigemasu uses fMRI, a technique for imaging brain activity, to elucidate the brain mechanisms involved in three-dimensional perception. In previous experiments, he has discovered that brain regions show different activation patterns when looking at differences in the depth position between objects versus looking at depth-direction structure inside objects, and he has elucidated the functions of each brain region.

The brain's information processing does more than just perceive the three-dimensional world from vision and other inputs. It must also output appropriate actions based on that information. Prof. Shigemasu realized that "to elucidate the brain's processing system, we must also consider action," and he broadened his research to regions involved in motor output. Which specific areas within the brain handle the processing of input information relevant to motor output? To elucidate this question, Prof. Shigemasu conducted experiments where participants were first shown an object, and then immediately afterwards the object was hidden, and they were asked to grasp the object with their hand by inferring its orientation under conditions with no visual information. Brain activity during the experiment was measured with MRI. The results showed that the orientation of the object can be decoded from the activity pattern of the visual region, even when there is no visual input in real-time. He is currently engaged in empirical studies showing that various types of active processing are carried out relating to motor output based on visual brain areas which have previously been thought to process mainly passive visual input.

"While many questions remain regarding how brain regions relating to motor output work, we've discovered that the processing in visual brain areas is closely connected to motor output for manipulating objects. And that finding has helped develop my research."

This elucidation of brain regions is likely to be useful for realizing brain-machine interfaces that allow operation of a machine or computer based on brain activity.

SHIGEMASUsensei3

Cutting-edge technology helps zero in on the universal characteristics of human beings

Thanks to the appearance of inexpensive, high-performance head-mounted displays, we are entering an era where VR and other artificially-constructed three-dimensional spaces are commonplace. Prof. Shigemasu: "If artificial worlds are to provide perceptual effects on a par with the real world, we need to understand how people achieve three-dimensional perception." In recent years, his focus has broadened to research using VR technology.

The distinguishing feature of VR is the ability to construct worlds free from the constraints of the real world. But what is the impact, on an individual's sensations of their own body, of a space that does not follow the physical laws of the real world, or a body different from their own? Such questions are still largely shrouded in mystery. "The anticipated widespread adoption of three-dimensional virtual spaces highlights the importance of understanding how effectively humans can adapt to alterations in space and body." Prof. Shigemasu aims to elucidate the features of human perception by manipulating VR environments and examining the resulting effects.

To investigate whether somatic sensations or motor output vary when the position of an individual's legs are visually manipulated in VR, Dr. Shigemasu conducted an experiment where subjects trained with their avatar projected frontally. The results showed that, when the avatar's legs were presented lower than in reality, the subjects raised their legs higher than in reality after training, and the changed perception of leg position persisted for as long as 10 minutes afterward. In questionnaire results, none of the experiment participants noticed that the presented body position was different from reality, and thus it was discovered that these effects occur unconsciously.

"We found that people may unconsciously increase their movement when training is carried out with their body position lowered in VR. This may have applications like exercise to boost insufficient movement, and rehabilitation of the elderly."

In experiments exploring techniques of operating extended body parts such as robot arms, with the same sensations as moving one's own body, it was found that, when a subject's hand is changed into a visually split state in VR, subjects move some sensation of their own body into the split hand. People can smoothly operate an extended body part by establishing a correspondence of the part with that position.

Prof. Shigemasu is also engaged in joint research on VR with researchers inside and outside Japan. Through experiments, he is empirically clarifying not only low-level processing like sensation and perception, but also high-level psychological effects such as motivation and empathy.

Among his other activities, Prof. Shigemasu is a participant in the R&D organization "Healthcare x VR" established by the Kochi Medical School. He conducts joint research with corporations aimed at treatment and remote medicine employing VR. He collaborates with e-Jan Networks Co. in support of corporate telework, and he has also started research investigating the psychological effects of communication in three-dimensional spaces. He is engaged in diverse interdisciplinary projects, such as technical cooperation with an event offering simulated experience of the Yosakoi Festival in a VR environment. He says that "the scope of my research has been steadily broadening due to the evolution of VR," but his objective lies, in the end, in elucidating the information processing systems of the human brain that have been optimized through the course of evolution.

Prof. Shigemasu has high expectations: "By using VR to examine spaces, or perceptions/reactions relating to one's own state, that cannot exist in reality, we can--paradoxically--uncover the characteristics of brain processing systems which have adapted to the real world." VR is a cutting-edge technology, and seems to have significant potential, even as a tool for investigating the nature of human beings. "By freely manipulating the world and people's bodies in VR, and examining what determines oneself, we can pursue the fundamental question: 'what is the self?' Explication of the brain's processing systems and a clearer understanding of the nature of the mind and self may bring into focus the tools and interfaces people truly need."

SHIGEMASUsensei4

Date of posting: February 2024