Some usecases of clmtrackr
In the previous post, I explained how CLMtrackr was put together. Since then, my examples of face substitution and emotion detection has received a fair amount of attention, so in this post I’m going to explain a bit about how these are put together as well, plus comment on some fixes that I’ve done to CLMtrackr recently.
This demo was inspired by a face substitution demo by Arturo Castro & Kyle McDonald. Basically it substitutes, or overlays, another persons face over your face, and does some fancy tricks to make it look natural. To do this with CLMtrackr, we first have to annotate the face in the image we want to substitute, and we can then deform this face (using face_deformer.js) to the same shape as your face, and overlay it in the exact same pose and position.
But in order to make it look natural (or creepy, as some would say), we also have to use a method called poisson blending. Usually, when you paste one image onto another, it’s easy to tell that there’s been a copy-paste operation, since the colors of the edges of the pasted image won’t quite match up with the background.
For the emotion detection demo, I used a pretty basic classification method called logistic regression. We already have a parametric model of the face, so we can use the parameters of the model as features. For training, we annotate images of people expressing the emotions we are interested in and project these annotations onto our PCA decomposition (as described in the previous post) to get the closest parametrization. These parameters are then input as training data for the regression. The classification works relatively OK, but a better method would be to first establish some neutral “baseline” for each person before classifying, since there is some variation from person to person which throws off the classification.
A fun side effect of the emotion classifier is that we can illustrate the learned emotions by using the regression coefficients as parameters for our facial model:
Some of these learned emotions look very similar, which caused the classifier to have a hard time distinguishing them. Interestingly, we can also negate the coefficients to see what the opposites of the learned emotions look like:
Play with the visualizations of the learned emotion model here
The classification method is not only restricted to emotions, so we could also try to classify whether a person is male or female. Try out a demo of this here, though note that it’s not really that accurate. Below are the resulting faces from the learned gender classifier:
Deforming the face
Some other toy examples I’ve added is live face deformation and live “caricatures”.
Both of these demos are based on capturing your face, deforming the face in some way, and pasting it back over your original face. The caricature demo was fairly easy to put together - the parameters in our parametric model of face describe the “offsets” from a mean face, meaning that these offsets distinguish any face from an “average face”. We can use this to create very simple “caricatures”, where we exaggerate the difference from the mean face by multiplying the parameters, and then overlay the deformed face with the new parameters over the original video. We can of course also modify (add constant offsets to) the parameters manually, i.e. deform your own face in realtime, which gives rise to the face deformation demo.
As I discussed doing in my previous blog post, I’ve also added local binary patterns and sobel gradients as preprocessing for responses. Especially local binary patterns seem to be more precise than raw responses, at the cost of some slowdown (due to need to preprocess patches). Since they’re slower, they’re not used by default, so you’ll have to enable them on initialization if you want to use them. Check out the reference for documentation on how to enable the different types of responses. There’s also the possibility to blend or cycle through different types of responses, which in theory might improve precision, a la ensemble models. Try out the different responses and combinations here.
In other news, CLMtrackr was used in this years april fools on reddit : “headdit”. For an april fools, the gesture recognition worked surprisingly well, though I’ll admit to not throwing away my mouse and keyboard just yet.
If you liked this blogpost, you should follow me on twitter!