What a light field actually is
Magic Leap raised $542 million without showing anyone a product. A look at the optics they're promising, and why it's so hard to wear on your head.
Magic Leap raised $542 million this week in a round led by Google, and as far as I can tell almost nobody outside the company has seen what they’re building. The press coverage talks about “cinematic reality” and a “digitized light field signal,” which sent me back to my optics notes to figure out what they might actually mean.
Start with what every current display does, whether it’s a phone or an Oculus Rift. Each eye gets one flat image, and that image sits at one focal distance. In a stereo headset your eyes still converge on objects at different depths, because the left and right images are different, but your lenses keep focusing at the fixed distance of the screen. In the real world those two signals always agree. In a headset they don’t. People call this the vergence-accommodation conflict, and it’s one reason close-up objects in VR feel slightly off and some people get headaches after twenty minutes.
A light field is the full description of the light in a region: at every point, how much light is traveling in every direction. A normal photo is a slice of it, since the sensor records how much light lands at each pixel and throws away which direction it came from. Lytro’s camera keeps some of the direction information by putting an array of microlenses in front of the sensor, which is why you can refocus a Lytro photo after taking it. The cost is resolution. The sensor’s pixels get split between “where” and “which direction,” so the final image is small.
If a display could reproduce the light field that a real object would send to your eye, your eye’s lens would focus on it the way it focuses on real things, because the rays would really be diverging from the right depth. A virtual cup on a real table would sit at the table’s focal distance, and you could shift focus between the cup and the wall behind it.
Building that as something you wear is the hard part. As far as I understand the options, you either trade a very high pixel density for angular resolution (the Lytro trade in reverse), or you stack several focal planes and switch between them, or you do something clever with scanning. The patents people have dug up mention waveguides and a scanning fiber. Every one of these approaches is fighting resolution, brightness, field of view and power at the same time, and the head-mounted version also has to be light, stay cool and run on a battery. I don’t know of any lab that has shown all of that working together.
My interest is partly selfish. I work on 3D faces, and the difference between a virtual person painted on a pane of glass a meter away and one who appears to actually sit across the table from you is probably the difference between a gimmick and something people use every day.
For the record, my bet: they ship something impressive to consumers within two years. $542 million buys a lot of optics PhDs, and Google doesn’t usually write checks this size for slideware. What I’m less sure about is whether it will be a light field display in the full sense or a smart approximation with a few focal planes. Even the approximation would be a big step up from anything you can buy right now.