Data-driven Markerless Multi-camera Motion Capture System

Project IDRFP-19002
Ability Data
Project PI
Research Fellow, Rehabilitation Research Institute of Singapore
ResearcherLIM Wee Sen, Project Officer
SELVARAJ Bharatha, Project Officer


The challenge

To capture human motion, the most common way for almost all human movement research labs is to use a marker-based motion capture system. Markers attached to the subject body allow all the cameras to know the locations of body parts to reconstruct the movement in 3D. However, the markers are preventing this kind of system from being used in many applications as it requires a lot of subject preparation time, manual post-processing time, and a few humans in the loop. Therefore, this project aims to build a human motion capture system without attaching any markers or sensors on the human subject while maintaining the tracking accuracy that is comparable to the marker-based motion capture system.

The proposed solution

This can be done in a data-driven way using modern neural-network-based machine learning techniques. Basically, we teach a computation model to predict the 2D location of markers on the synchronized videos of a markerless subject and strategically triangulate them to get 3D position of those virtual markers. To create the training dataset, our annotations on the video are done with high precision using data from a marker-based motion capture system. In addition, the scale of our training data is already bigger than any public datasets of the same kind and it is continuously growing to enhance the accuracy and robustness of the trained model.