At EnginXr, we needed to animate people in 3D for our projects. One of the solutions would have been to make the animation frame by frame on a 3D software such as Blender. But it takes a lot of time and it does not necessarily offer a natural look.
Another solution would have been to rent a motion capture studio.
The quality of the animation renderings is excellent but comes with a significant cost in terms of operator time and material costs. In addition, if changes are required in the animation, these can easily generate significant costs, by asking to re-prepare a recording set, with the costs that this implies.
At EnginXr, we needed a simple, fast and efficient solution to record our animations. The total synthesis under Blender as well as the rental of a motion capture studio were not adapted for our needs so we created our own solution: using the Oculus Quest virtual reality headset with its 6-dimensional controllers (6DoF) to make motion capture in combination with headphones to record voice.
And guess what? Results are pretty good !
And here is the solution
General logic
Our “home-made” motion capture studio is separated into 3 parts.
- A client application on board the Oculus Quest which, when clicking on a joystick, records the position / rotation of the joysticks and helmet and sends this telemetry data to a server in NodeJS.
- A server in NodeJS with socket.io who takes care of transmitting telemetric data between our Oculus Quest and the editor application.
- An “Editor” application, which receives the telemetry data from the Oculus Quest and records animations in combination with the audio spectrum of the presenter’s voice.
For the occasion, we acquired a good wireless headset (HS70-Corsair) which is directly connected to the computer running the editor. It is also possible to record sounds from the Oculus Quest but an audio signal can be quite heavy and we also avoid synchronization problems by directly connecting the wireless microphone to the editor.
As we used a paid package for server communications, we cannot make the code available on GitHub. But here is how we did it and the code that will allow you to do the same!
The client on the Oculus Quest
First, of all the communication part between the client, server and the editor we used socket.io which allows instant communication.
To implement socket.io in the Unity editor and client, we preferred to use an excellent paid package: BestHTTP Pro.
https://assetstore.unity.com/packages/tools/network/best-http-10872
Using this package allows for a very simple implementation of socket.io and allows you to focus on other tasks. There is far enough to develop (;
Regarding the code on our client application, namely the application embedded on the Oculus Quest, it is very simple. A simple Unity project where we record the position / rotation of the controllers and the head and send it all via socket.io to our NodeJSserver through a single script.
The code “socketClient.cs” :
using BestHTTP.SocketIO;
using System;
using System.Collections;
using System.Collections.Generic;
using UnityEngine;
[Serializable]
public class telemetry_packet
{
public TF left_hand;
public TF right_hand;
public TF head;
}
[Serializable]
public class TF
{
public Vector3 position;
public Quaternion rotation;
public TF(Transform t)
{
this.position = t.localPosition;
this.rotation = t.localRotation;
}
}
public class socketClient : MonoBehaviour
{
SocketManager Manager;
public const string localSocketUrl = "http://localhost:7000/socket.io/";
public const string webSocketUrl = "http://yourInternetEndPoint:7000/socket.io/";
telemetry_packet tp;
public GameObject right_hand;
public GameObject left_hand;
public GameObject head;
// Start is called before the first frame update
void Start()
{
timeFrame = 1f / fps;
timeTelemetry = timeFrame;
SocketOptions options = new SocketOptions();
options.AutoConnect = false;
options.ConnectWith = BestHTTP.SocketIO.Transports.TransportTypes.WebSocket;
Manager = new SocketManager(new Uri(webSocketUrl), options);
Manager.Socket.On("connect", OnConnect);
Manager.Socket.On("disconnect", OnDisconnect);
Manager.Open();
}
bool connect = false;
bool record = false;
void OnConnect(Socket socket, Packet packet, params object[] args)
{
tp = new telemetry_packet();
connect = true;
Debug.Log("connected");
}
void OnDisconnect(Socket socket, Packet packet, params object[] args)
{
connect = false;
// args[0] is the nick of the sender
// args[1] is the message
Debug.Log("disconnected");
}
bool click = false;
float time = 0f;
float fps = 60;
float timeFrame = 0f;
float timeTelemetry = 0f;
public Material skybox;
IEnumerator Tele ()
{
tp.head = new TF(head.transform);
tp.right_hand = new TF(right_hand.transform);
tp.left_hand = new TF(left_hand.transform);
string json = JsonUtility.ToJson(tp);
yield return null;
Manager.Socket.Emit("telemetry", json);
yield return null;
}
// Update is called once per frame
void FixedUpdate()
{
if (Input.GetKeyUp(KeyCode.R) || OVRInput.Get(OVRInput.Button.One))
{
if (!click)
{
time = 0f;
click = true;
if (record)
{
skybox.SetColor("_SkyTint", Color.white);
record = false;
}
else
{
skybox.SetColor("_SkyTint", Color.red);
record = true;
}
if (connect)
{
Manager.Socket.Emit("setrecording", record);
}
}
}
else
{
time += Time.deltaTime;
if (time > 1f)
{
click = false;
}
}
if (connect)
{
timeTelemetry -= Time.deltaTime;
if (timeTelemetry < 0f)
{
timeTelemetry = timeFrame;
StartCoroutine("Tele");
}
}
}
void OnDestroy()
{
skybox.SetColor("_SkyTint", Color.white);
// Leaving this sample, close the socket
if (Manager != null)
{
Manager.Close();
}
}
}
The server
Here it is also quite simple. We just want to receive the telemetry data from our client and transmit them to our editor. This was achieved using NodeJS with the “express”, “fs” and “socket.io” libraries which you can simply install with NPM.
Use the command npm install + express, fs, socket.io.
For hosting our server, we use Heroku. This service allows free hosting of small node.jsservices.
The code “server.js” :
var express = require("express");
var fs = require('fs');
var app = new express();
var http = require("http").Server(app);
var io = require("socket.io")(http);
app.use(express.static(__dirname + "/public" ));
app.get('/',function(req,res){
res.redirect('index.html');
});
io.sockets.on('connection', function (socket) {
console.log('Someone is connected');
socket.on('setrecording',function(value){
console.log('Recording change to '+value);
io.emit('recording', value);
});
socket.on('telemetry',function(json){
io.emit('telemetry', json);
});
socket.on('disconnect', function() {
console.log('Disconnected');
io.emit('clientdisconnected', true);
});
});
console.log('Server is correctly running for the moment :-)');
http.listen(7000,function(){
console.log("Server running at port "+ 7000);
});
The editor
This is the most complicated part. First, you need to create a new Unity project.
To record the audio coming from our headset, we used the “DarkTable” project called “SavWav”.
Why redo the wheel when it already exists!
https://gist.github.com/darktable/2317063
Then the first thing we need to do is get the telemetry data from our server.
The code “socketMaster.cs” :
using BestHTTP.SocketIO;
using System;
using System.Collections;
using System.Collections.Generic;
using UnityEngine;
[Serializable]
public class telemetry_packet
{
public TF left_hand;
public TF right_hand;
public TF head;
}
[Serializable]
public class TF
{
public Vector3 position;
public Quaternion rotation;
public TF(Transform t)
{
this.position = t.position;
this.rotation = t.rotation;
}
}
public class socketMaster : MonoBehaviour
{
SocketManager Manager;
Follower follow;
AnimMicroRecorder amr;
public const string localSocketUrl = "http://localhost:7000/socket.io/";
public const string webSocketUrl = "http://yourEndPoint:7000/socket.io/";
// Start is called before the first frame update
void Start()
{
SocketOptions options = new SocketOptions();
options.AutoConnect = false;
options.ConnectWith = BestHTTP.SocketIO.Transports.TransportTypes.WebSocket;
Debug.Log(new Uri(webSocketUrl));
Manager = new SocketManager(new Uri(webSocketUrl), options);
Manager.Socket.On("connect", OnConnect);
Manager.Socket.On("disconnect", OnDisconnect);
Manager.Socket.On("telemetry", Telemetry);
Manager.Socket.On("recording", OnStatusChange);
Manager.Open();
follow = GameObject.FindObjectOfType<Follower>();
amr = GameObject.FindObjectOfType<AnimMicroRecorder>();
}
void OnConnect(Socket socket, Packet packet, params object[] args)
{
Debug.Log("connected");
}
void OnDisconnect(Socket socket, Packet packet, params object[] args)
{
// args[0] is the nick of the sender
// args[1] is the message
Debug.Log("disconnected");
}
void Telemetry(Socket socket, Packet packet, params object[] args)
{
// args[0] is the nick of the sender
// args[1] is the message
string data = (string)args[0];
telemetry_packet tp = JsonUtility.FromJson<telemetry_packet>(data);
if (follow.tp == null)
{
follow.tp = tp;
follow.setHeadOriginal();
}
else
{
follow.tp = tp;
}
}
void OnStatusChange(Socket socket, Packet packet, params object[] args)
{
bool record = (bool)args[0];
// args[0] is the nick of the sender
// args[1] is the message
Debug.Log("OnStatusChange "+ record);
if (record)
{
follow.record = true;
follow.setHeadOriginal();
amr.startRecording();
}
else
{
amr.stopRecording();
follow.record = false;
}
}
void OnDestroy()
{
// Leaving this sample, close the socket
if (Manager != null)
{
Manager.Close();
}
}
// Update is called once per frame
void Update()
{
}
}
When we receive JSON data from the server, we decompress it in a classand send it to the “Follower” object which will imitate the movements of the “Client”.
When we receive JSON data from the server, we decompress them in a class and send it to the “Follower” object which will imitate the movements of the “Client”. The Follower object will imitate exactly the same position and rotation movements that we will record on the “Client” side.
The code “Follower.cs” :
using System.Collections;
using System.Collections.Generic;
using UnityEngine;
public class Follower : MonoBehaviour
{
[HideInInspector]
public telemetry_packet tp;
public Transform root;
TF original_pos;
// Start is called before the first frame update
void Start()
{
original_pos = new TF(root);
}
public GameObject right_hand;
public GameObject left_hand;
public GameObject head;
public bool record = false;
Vector3 originalHeadPosition = Vector3.zero;
public void setHeadOriginal ()
{
if (tp != null)
{
originalHeadPosition = tp.head.position;
}
}
// Update is called once per frame
void FixedUpdate()
{
if (tp != null)
{
Vector3 right = tp.right_hand.position - originalHeadPosition;
//right.y = tp.right_hand.position.y;
right_hand.transform.localPosition = right;
Vector3 left = tp.left_hand.position - originalHeadPosition;
//left.y = tp.left_hand.position.y;
left_hand.transform.localPosition = left;
head.transform.localPosition = tp.head.position - originalHeadPosition;
right_hand.transform.rotation = tp.right_hand.rotation;
left_hand.transform.rotation = tp.left_hand.rotation;
head.transform.rotation = tp.head.rotation;
}
}
}
Now when a controller moves on our client, it also moves in our editor, we just have to save the animation and the sound. To record the animation, we use an editor function available in Unity called “GameObjectRecorder” which allows the recording of an animation natively. To record the sound, we use the “DarkTable” project called “SavWav” which you can find by the link above.
The code “AnimMicroRecorder.cs” :
using System.Collections;
using System.Collections.Generic;
using UnityEditor;
using UnityEditor.Animations;
using UnityEngine;
public class AnimMicroRecorder : MonoBehaviour
{
private AnimationClip clip;
private GameObjectRecorder m_Recorder;
public GameObject root_target;
AudioClip _RecordingClip;
int _SampleRate = 44100; // Audio sample rate
int _MaxClipLength = 300; // Maximum length of audio recording
public float _TrimCutoff = .01f; // Minimum volume of the clip before it gets trimmed
public Material mat_skybox;
// Start is called before the first frame update
void Start()
{
}
float timeRecord = 0f;
bool record = false;
float timefps = 0f;
public void startRecording ()
{
mat_skybox.SetColor("_SkyTint", Color.red);
Debug.Log("Start Recording");
clip = new AnimationClip();
m_Recorder = new GameObjectRecorder(root_target);
m_Recorder.BindComponentsOfType<Transform>(root_target, true);
timeRecord = 0f;
timefps = 1f / fps;
timeFrame = 0f;
record = true;
if (Microphone.devices.Length > 0)
{
_RecordingClip = Microphone.Start("", true, _MaxClipLength, _SampleRate);
}
}
int number = 1;
public void stopRecording ()
{
mat_skybox.SetColor("_SkyTint", Color.white);
Debug.Log("Stop Recording");
record = false;
m_Recorder.SaveToClip(clip, 30f);
string fileName = System.DateTime.Now.ToString("dd-hh-mm-ss");
string name = fileName + ".anim";
AssetDatabase.CreateAsset(clip, "Assets/Resources/Recordings/Animations/"+ name);
AssetDatabase.SaveAssets();
if (Microphone.devices.Length > 0)
{
Microphone.End("");
var samples = new float[_RecordingClip.samples];
_RecordingClip.GetData(samples, 0);
List<float> list_samples = new List<float>(samples);
int numberOfSamples = (int)(timeRecord * (float)_SampleRate);
list_samples.RemoveRange(numberOfSamples, list_samples.Count - numberOfSamples);
var tempclip = AudioClip.Create("TempClip", list_samples.Count, _RecordingClip.channels, _RecordingClip.frequency, false, false);
tempclip.SetData(list_samples.ToArray(), 0);
string path = Application.dataPath + "\\Resources\\Recordings\\Animations\\" + fileName;
SavWav.Save(path, tempclip);
}
}
float fps = 30f;
float timeFrame = 0f;
// Update is called once per frame
void FixedUpdate()
{
if (Input.GetKeyDown(KeyCode.R))
{
if (record)
{
stopRecording();
}
else
{
startRecording();
}
}
if (record)
{
timeRecord += Time.deltaTime;
timeFrame += Time.deltaTime;
timefps -= Time.deltaTime;
if (timefps < 0f)
{
//Debug.Log("fps");
m_Recorder.TakeSnapshot(timeFrame);
timefps = 1f / fps;
timeFrame = 0f;
}
}
// Take a snapshot and record all the bindings values for this frame.
}
void OnDestroy()
{
mat_skybox.SetColor("_SkyTint", Color.white);
}
}
Conclusion
In conclusion, you now have all the parts you need to make your own motion capture studio with an Oculus Quest and headphones, that’s the magic. Our only limit is our imagination !
To go further, you can animate your character’s mouth with a package called “Salsa Sync” also available in the Unity Asset Store. Easy to use and visually realistic! To record the demo video, we used the Unity package called “Unity Recorder”. You can find it in Unity’s package manager.
Do not hesitate to ask us questions. If you add improvements to this project and / or record animations, don’t hesitate to show us your results.
The EnginXr team