XR Animator is probably the best video-to-mocap tool that is 100% free and open-source. There is no paid tier or locked features whatsoever.
XR Animator can detect and record full body movement including body, face and hands from any footage or web cam input. It is based on the open-source Machine Learning/AI libraries MediaPipe and TensorFlow.js and it just gives these technologies a user interface that anyone can use.
XR Animator UI/UX
Speaking of user interface, this is what XR Animator looks like. It’s … quite special. It is not very intuitive at first but once you know where to click, you’ll be able to record a mocap with ease. I’ll show you the steps in this article.
VTubing and beyond
As you can see from the image, the app seems to be aimed at VTubers and anime fans. However, mocap is mocap. You can export the animation as a BVH and GLTF skeleton and use it for whatever purpose you want.
Getting Started
To start using XR Animator, you can either use the Web app or download a compiled software for Windows, Mac or Linux.
XR Animator Web App
XR Animator installers for Windows, Mac and Linux
The web and local app are largely identical in functionality but I found the UI of the Windows one difficult to work with. One of the additional features of the local installation is a “transparent background” but I found that rather confusing.
For my own workflow, I like using the web app.
However, note that I’m more interested in exporting mocap data than life recording. A VTuber may find the transparent background of the local app ideal to overlay over other footage.
Quick-start Guide
Notice the areas marked as 1, 2, 3 and 4 in the image below. The instructions that follow are based on it.
- If you want to use pre-recorded footage instead of live mocap from a webcam, click the button marked with ① (File Input). In the middle of the screen, you’ll see a menu which allows you to Choose File. Select your file from the file browser, then press OK.
- Double-click the button marked with ② (Webcam/Media). Click “Yes”, then either select your Webcam for live mocap or “Local Media file” for pre-recorded footage.
- Double-click the button marked with ③ (Motion capture) and choose what you’d like to capture. You can choose between different combinations of body, hands and face tracking. The options are self-explanatory (Body only, Face only, Body + Face, etc.). After making your choice, the mocap systems will take a second to load and your character will now be performing the motion capture from your webcam or footage.
- To Record a mocap, Double-click the button marked with ③ (yes, again) and choose ●Record Mocap. Choose the recording speed: most likely x1.
- Finally press the Play/Pause button from the area marked with ④. Now the mocap recording is in progress and you’ll have to wait until it reaches the end of the footage or finalize it manually with the ■Stop button.
- To download your mocap, Double-click the button marked with ③ (yes, again) and choose Export motion to file. Then choose the format you want to save from the available options (VMD, gLTF, BVH, VRMA). BVH worked best for me. I was interested in gLTF but it would simply not bring up the save file screen at all so I could not test it.
Once you have your BVH, you can import it in Blender through File > Import > BVH.