Motion Capture 3D Wayang Character with Pose Landmarks using MediaPipe

Abstract

Motion Capture (MoCap) is vital for the digital tracking and recording of motions across many applications, enhancing sectors from entertainment to industrial processes.  This work developed a web-based markerless motion capture system utilizing MediaPipe, capable of recognizing and replicating real-time human movements. The movements are then displayed on a 3D Wayang character, Semar, rendered in WebGL. The process was to record live webcam footage, process it with MediaPipe to find landmarks for 33-point poses, project these 3D coordinates onto the Blender-rigged Wayang model, and then use WebGL to render the animated result. The results showed that accuracy was greatly affected by ambient lighting and that the system worked best between 0.5 and 3 meters from the camera. Although hand landmark detection produced accurate results, the intricacy of the rigging and the absence of specific WebGL 3D animation references made it difficult to accurately integrate leg and body movements with the 3D model. Regardless the challenges, this research provides a more practical and cost-effective alternative to conventional motion capture methods, opening up new possibilities for entertainment, gaming, and cultural heritage preservation. Character rigging will be improved and MediaPipe will be further integrated with other 3D models in the future.