Technology Uses Mouth Gestures to Interact in Virtual Reality

A virtual reality device reads your Mouth Gestures: how could this technology improve?

New technology allows users to interact with each other in the virtual reality environment using mouth gestures.

Facebook, HTC, Microsoft and PlayStation are some of the top players in the game of virtual reality device marketing with competitive pricing and designs for the mass market. But what Oculus, Vive, HoloLens and PlayStation VR have in common is that by offering immersive visual experiences in their viewers, they hide the upper half of a user’s face and prevent facial recognition of the entire face.

Lijun Yin, leader of the Binghamton team, points out that to solve this aspect, the new framework that construe mouth gestures as a means of physical phenomenon within virtual reality in real time.

The device has been tested by graduate students, who played with it a game whose aim was to guide an avatar around a forest and eat as many cakes as possible. The players had to select their direction of movement using the head rotation, move with mouth gestures and could only eat a cake smiling.

The system was able to describe and classify the movements of the user’s mouth gestures, and achieved high recognition rates.

Yes, we know that it may seem like a technology without much innovative potential to change the gaming and immersive experience of virtual reality devices, but the truth is that these not only have applications in the world of video games. “The virtual world is not just for entertainment. For example, health care uses VR to help patients with disabilities,” said Yin. “Medical professionals or even military personnel can undergo training exercises that can not be experienced in real life. This technology allows the experience to be more realistic.”

Even communication tools such as Skype can make the experience more real with these advances: “Imagine feeling as if you were in the same geometric space, face to face with your Skype partner, and the computer program can effectively represent your facial expressions, mouth gestures and replicate them to make it look real.”

The Yin people tested the application on a group of third graders. Once a user put a viewer in the form of a viewer, presented them with a simplistic game; the goal of the game was to lead the player avatar around a forest and eat as many cakes as possible. The players had to select their direction of movement using the main rotation, move using mouth gestures and they could only eat cake smiling. The system could describe and classify movements of the user’s mouth, and achieved high types of correct recognition. The system has also been proven and validated with a real-time virtual reality application.

Although the (mouth gestures) technology remains at the prototype stage, Yin believes its technology is applicable to a plethora of fields.

Tags : technologyVirtual Reality

The author FileEdge