Why my 3D faces look dead: the physics of skin
Our app gets the shape of your face right and still looks like a wax museum. Most of the problem turned out to be motion.
The first time we showed the app to people outside the team, a friend took a selfie, watched her 3D face smile back at her, and said, very kindly, “That’s so cool. Please never show me that again.”
The landmarks were in the right place and the texture was her own photo, and it still looked like a wax figure that had just woken up. I’ve spent a lot of the time since trying to work out why.
Animation in my thesis pipeline works by moving a few dozen control points on a template (mouth corners up, brows up) and letting a smooth interpolation drag the rest of the mesh along, like pins in a rubber sheet. Real skin slides over fat and muscle and bone, and the rubber sheet model is wrong in a few specific places.
Raise your eyebrows and the forehead skin moves while the brow ridge under it stays put, so the skin bunches in one place and glides in another. Smile, and the cheek muscle shortens and the cheek pushes outward because the tissue keeps its volume. Our interpolation doesn’t know about volume, so the smiles come out flat, a bit like a mask being pulled from behind. The fold from the nose to the mouth corner should deepen, and we get a shallow dent. The eyes are a separate machine: the eyeball rotates, the lids slide over it, and it’s wet, so the highlight moves differently from the skin. Ours were aligned perfectly and looked empty.
None of this is news to film people. Benjamin Button and Avatar had head-mounted cameras on the actors and artists fixing shots by hand, and USC’s Digital Emily needed a light stage for one convincing face. We’re doing it from one selfie in under a second on a phone, so of course we’re worse. I just didn’t expect so much of it to be about timing.
Ekman’s FACS splits an expression into onset, apex and offset. A real smile and a polite one can look almost the same at the apex and differ in how fast the mouth moves, whether the eyes join, how it fades. Our first version interpolated linearly from neutral to target, so every part of the face started and stopped at the same instant at the same speed, which no living face does. Mori wrote in 1970 that the uncanny valley is deeper for things that move than for still ones, and that’s exactly what we saw. People were fine with screenshots and hated the animation.
The fix came from undergrad mechanics. Each region of the face now follows its target through a mass-spring-damper instead of snapping to it, so it accelerates, sometimes overshoots slightly, and settles. Stiffness varies by region. Lips and eyelids are stiff with light damping so they move fast and arrive first, cheeks are softer and trail the mouth with a small overshoot, and the area under the jaw is softer still and lags the head. We also added the usual animator tricks, irregular blinks, small eye movements a few times a second, and a slight drift so the face is never fully still (a completely still face looks either dead or like it’s about to do something bad).
It runs per vertex on the GPU and costs almost nothing on the new 64-bit iPhones. Around thirty lines of shader code did more for the “please never show me that again” problem than six months of geometry work, which is a little painful to admit.
We still can’t get the lighting on skin right. Light enters skin, scatters under the surface and comes out somewhere else (it’s why your ear glows red with the sun behind it), and without that subsurface scattering skin looks like plastic. Our cheap approximation looks less like plastic, but not really like skin yet. Wrinkles are faked too. The mesh is too coarse for real folds, so we fade in wrinkle textures for some expressions and hope nobody looks closely.
I went into this thinking realism was about resolution, more triangles and sharper textures, and so far most of our improvement has come from the motion.