hand tracking

Developments in VR Hand Tracking

AA logo white
White Avatar Academy Logo


Article by: Imran

Publishing Date: 7th March 2022


In the world of virtual reality, hand tracking allows the user to interact with the virtual world without the use of controller devices and instead use a virtual representation of their physical hands within the virtual world. This allows for more immersion and “realism” within the virtual world. 

In the real world your hands are not bound to controller devices as you interact with the real world, and this allows for a lot of freedom for your hands. The current and further developments of hand tracking within virtual reality will allow for a similar level of freedom and realism when it comes to interactions in the virtual world.

Developments in VR Hand Tracking:

Hand tracking in virtual reality has come a long way since one of the first developments in hand tracking in 1982. This hand tracking was achieved using Sayre gloves, which are wired gloves. These gloves achieved hand tracking by using light emitters that hit the photocells in different ways depending on the finger movements of the wired gloves, these photocells would then convert these movements into electrical signals [1].

One of the many current developments in VR hand tracking allows for hands-free hand tracking without controllers or gloves. One of the ways this is achieved is through the combined effort of VR hand tracking hardware, VR tracking software, and VR tracking tools and applications. 

The tracking hardware uses sensors that are built into the virtual reality headset that collect hand data like positions, rotations, and movements of your hands. The VR tracking software and engine will then process the gathered data (positions, rotations, and movements of the hands) to generate a pair of hands within the virtual world that are a representation of your pair of hands in the real world.

When the real hands have been tracked and a pair of virtual hands have been created within the virtual world, the virtual reality applications and tools will then use the virtual hands to allow for interactions in the virtual world using hand gestures that are similar to the interactions that real hands will have in the real world. 

The user will be able to grab a specific object the same way you would grab it in the real world, for example grabbing a cup by its handle in the virtual world like you would in the real world. The user could poke, touch, and scroll specific objects in the virtual world similar to how you poke and scroll on the screen of a smartphone, or press on the keys of a keyboard with your fingers, and touch a button [2].


Virtual reality and hand tracking in virtual reality has clearly come a long way since the 1980s, we went from wired gloves using photocells that sent electrical signals as a way to track hand movements towards having a virtual reality headset with the tracking sensors built into the headset that allowed for tracking of the positions, rotations and movements of the hands and creating virtual representation of these hands in the virtual world. This then allowed for virtual hand tracking and interactions that are similar to the hand interactions, we have in the real world without mostly needing controllers or wired gloves.

White Avatar Academy Logo
AA is an Oculus ISV Partner

Hand-Tracking Developments

White Avatar Academy Logo


Article by: Jordan Murphy

Publishing Date: 21st March 2022

Virtual Reality has pushed the boundaries in the immersive field since it started, and one of the newest features of this is Hand-Tracking. Hand-Tracking for the Oculus Quest is still in the beta version at the current time; however, Facebook have revealed their new iteration that they are working on. Their high-fidelity hand-tracking is on Facebook’s research page, a publication called Constraining Dense Hand Surface Tracking with Elasticity

Hand-Tracking works by using the inside-out cameras that are a part of the Oculus Quest’s hardware. The headset detects the position and orientation of the user’s hands and tracks the configuration of the users’ fingers. Once the headset detects them, the computer vision algorithms are used to track the movement and orientation of the user’s hands. 

Facebook Reality Labs

Facebook Reality Labs (FRL) journey to create more natural devices started with the Touch controllers. The Touch controllers for the Oculus Quest are comfortable to hold and contain sophisticated sensors. They deliver a life like hand presence and make the most basic interactions in VR feel like the user is doing that action. As close to the real thing that the Touch controllers can get, they cannot replicate the expressiveness of hand signs or the act of typing. 

On the 1st of December 2020, FRL published a paper with an accompanying video that showed their new research and technology allowing them to track the human hand in real time. With Facebook’s interest in VR through Oculus, it is not surprising to see them working on a better VR environment that is closer to human interaction. The efforts made to create the hand-tracking system for the Oculus Quest had an issue with hand overlap, which is the core of the goal of FRL’s latest research. The research focused on solving the big problem of losing tracking when hands overlap, touch, or fingers are self-occluded (Hexus, 2020).

Self-contact and hand-tracking technology

Many of the gestures and actions that people make with their hands involve self-contact and occlusion: shaking hands, making fists, or interlacing their fingers while thinking or idle. These uses illustrate why hand-tracking technology today is not designed to treat the extreme amounts of self-contact and self-occlusion exhibited by the common hand gestures.

FRL’s paper states that by extending recent advances in vision-based tracking and physically based animation, they can present the first algorithm capable of tracking high-fidelity hand deformations through highly self-contacting and self-occluding hand gestures. In the paper, they describe that by constraining a vision-based tracking algorithm with a physically based deformable model, they have obtained an algorithm that is robust enough for use in self interactions and massive self-occlusion that are displayed by the common hand gestures, allowing them to have two hands interacting with each other and some of the most difficult gestures that the human hands can make. 

Despite the improvements that this paper has proven to make on the already publicly available hand-tracking technology, the method that FRL has developed has its limitations. It is computationally expensive. The method is also limited by the ability to capture high frequency folds and wrinkles. In Figure 4 of the paper, they recorded that sometimes the captured images do not portray the wrinkles on the tracked mesh. These are issues that can be worked out in future iterations, but this research shows a promising future for hand-tracking in the immersive industry.


At Mersus Technologies, we use Hand-Tracking on our Avatar Academy Platform. This allows us to create as close to real world experiences as possible which benefits the learner in the transfer of the knowledge and skill they gain from Avatar Academy to the real world procedures. 

White Avatar Academy Logo
AA is an Oculus ISV Partner