Extended Reality R&D department: several examples of experiments and research

December 9, 2021
R&D department in extended reality: several examples of experiments and research

The field of augmented and virtual reality development is developing by leaps and bounds: Facebook announced its plans to build a metaverse, major market players release new AR / VR / MR devices, new technologies and tools appear regularly. These factors represent both opportunities and challenges. It is very easy to lag behind the development of new directions, to concentrate on the familiar and convenient.

Research & Development in VR/AR

In order to stay at the peak of new solutions, we at Qualium Systems made a decision about a year ago to create an R&D department in virtual and augmented reality. As part of this department, we master and test new features, tools and capabilities. We use everything that has a perspective to solve the business problems of our clients.
 

Passthrough API

For example, we were one of the first to test a then new feature for Oculus Quest 2 – Passthrough API. This technology enables mixed reality experiences that combine real and virtual worlds into one. Immediately after an update became available, we started testing the API. It is an exciting promise by Oculus and has a lot of perspective. We are already brainstorming the ideas of using Passthrough API to solve our clients’ business issues. To test Passthrough API, we used the following Technology stack: Unity SDK, Oculus SDK 31-34, Passthrough API.

Our experiment with the Passthrough API was for V31 when the technology was in experimental mode. In the further V34 version of Oculus, the Passthrough API has ceased to be an experimental function and now it does not need to be activated by a user via the console.

You can watch our experiment in the video.

 

Comparison of the Oculus 2 with Passthrough API and HoloLens 2

Also, in continuation of our experiments with the Passthrough API technology, we decided to test the hypothesis – Can the Oculus 2 with Passthrough API take the place of HoloLens 2? After the MRTK library has been adapted to the Oculus Quest 2 and in the ver. 30 of the Operating system we received a pass-through feature, we decided to check if Oculus 2 could be used as an alternative to expensive HoloLens 2. To validate the idea, we’ve made two applications with the same features on the Oculus Quest 2 and on the HoloLens 2. You can see the result in the article https://www.qualium-systems.com/blog/ar-vr/can-the-oculus-2-with-passthrough-api-take-the -place-of-hololens-2-checking-the-hypothesis.

As part of our experiments, we are trying to highlight the industries that can most benefit from XR technologies, especially during a pandemic and its aftermath.
 

The standalone application with computer vision base on Barracuda and TF model

We understood that during the COVID-19 pandemic, many people lost the opportunity to train in the gym with a trainer. Lack of coaching support can lead to improper exercise performing, which in turn can lead to injury. But thanks to modern technology, it has become possible to train correctly. We are creating a standalone application with computer vision based on Barracuda and TF model. You view a recording of an exercise with a trainer and repeat it in real time. The computer vision based on Barracuda and TF model technology compares your movements and displays information on how well you perform the exercise. You can see the result in the video.

 

Tensorflow lite body recognition

As part of testing our hypotheses that can be applied to sports and fitness, we compare Tensorflow lite body recognition based on PoseNet model and Unity Barracuda using tfjs-to-tf and tf2onnx.

 

Face tracking on PC with Unity SDK and UltraFaceBarracuda

One industry that can benefit from XR adoption is Retail. As part of this direction, we tested Face tracking on PC with Unity SDK and UltraFaceBarracuda: depending on the face position, the picture that a person is looking at moves. As a result of this, the effect of a hologram is created. It is an interesting solution for marketing purposes with the ability to track people’s actions and test various marketing activities. To do so, we used the Technology stack: Unity SDK, UltraFaceBarracuda. See the result in the video.

This is a small part of our experiments. We continue to research, hypothesize, and help our clients.

AR/VR Apps development