Notes from
Latent Space

A line-by-line estimate for one 47-minute episode using this year's public video generation prices. Generating the footage is only part of the budget; performance and continuity remain much harder to get right.

Coding agents somehow learned to build good-looking websites, although nothing checks whether a design is beautiful. What video editing can borrow from that: preference models, critics and loops.

Google DeepMind's Gemini Robotics ER 2 gives robots a better high-level brain. The part I still think nobody has built is the body-first learning underneath. Here's the research plan I'd run.

Generative recommenders are going into production. How a video becomes an embedding, then a semantic ID, and how a Transformer uses those IDs to choose what you'll watch next.

At I/O this week, YouTube brought Gemini Omni to Shorts Remix: take a Short, change the characters, setting or style, and keep the motion and story. Why remix is the right first experience for video-to-video.

OpenAI closed the Sora app this week, seven months after launching it as a social network, and the API ends in September. What went wrong, and what it says about where generative video belongs.

Yann LeCun's AMI Labs raised $1.03 billion to build world models on the JEPA framework. Revisiting what I wrote in 2022, and the one thing I still think the plan is missing.

ByteDance's Seedance 2.0 generates multi-shot sequences with references for characters, props and sound, and triggered cease-and-desist letters within a day. What's left for the editor, and for rights holders.

World Labs opened an API for Marble. I spent the weekend building a small workbench for rendering its worlds as Gaussian splats and authoring camera paths through them. What broke: scale, coordinates and drift.

I built an agent that edits my social videos end to end using knowledge of my personal taste. What that asks of the timeline, the feedback loop and the product.

In one week World Labs launched Marble, Yann LeCun confirmed he's leaving Meta to start a world-model company, and Google shipped Gemini 3. The world-model bet went public from three directions.

OpenAI launched Sora 2 as a TikTok-style app where every video is generated. The model is impressive. I'm less sure a feed of generations gives people a reason to keep watching.

Google's new image model, known everywhere as Nano Banana, keeps a person looking like themselves across edits and scenes. For AI storytelling, identity preservation matters more than image quality.

DeepMind's Genie 3 generates interactive worlds in real time at 720p that stay consistent for minutes. Persistence is the test for a world model, and it just got much better.

Microdramas are one of the fastest-growing forms of entertainment, and the format exists because of production cost. What happens when AI changes the cost?

Meta's V-JEPA 2 learns a world model from a million hours of video, then learns to plan robot actions from 62 hours of robot data. Promising, and still missing touch.

Google's Veo 3 generates video with synchronized dialogue, sound effects and ambient audio. Sound turns clips into scenes. Also at I/O: a language model that writes by denoising.

Real per-second prices for Veo 2, Kling 2.0 and Jimeng this month, and why the retake rate matters more than the sticker price.

GPT-4o's image generation turned the internet into Studio Ghibli for a week. When every style is one prompt away, the scarce thing is judgment.

Figure's Helix splits robot control into a slow model that understands and a fast one that moves. A golf swing works the same way, which is why thinking during it ruins it.

A Chinese lab released a reasoning model that matches OpenAI's o1 and published how it did it. Nvidia lost about $600 billion in a day. Where the efficiency actually came from.

DeepMind's Genie 2 turns one image into a playable 3D world, and Fei-Fei Li's World Labs turns one image into a 3D scene you can walk through. Worlds are the next medium after text, images and video.

"Sad songs for a rainy drive" beats any genre taxonomy. Natural language is becoming the way people ask for music, and it changes what a music recommender has to understand.

Hopfield and Hinton won the physics prize, and Hassabis and Jumper shared chemistry for AlphaFold. A mechanical engineer's reaction to physics and AI converging.

Microsoft signed a 20-year deal to restart a reactor at Three Mile Island to power its data centers. The constraint on AI is moving from chips to electricity, and baseload nuclear is suddenly back.

Google Research's GameNGen simulates Doom in real time with a diffusion model: no game engine, just next-frame prediction conditioned on your inputs. Interactive video is a world model.

AlphaProof and AlphaGeometry 2 together solved four of six problems from this year's Math Olympiad, reaching silver-medal level. What makes a checkable answer so useful for training, and what happens when there isn't one.

Kuaishou, TikTok's main rival in China, released a video model that rivals Sora's samples, and ordinary users in China can already try it. Video models get built by whoever has the video.

OpenAI and Google both showed real-time multimodal assistants this week. Speech as a native modality changes what a conversation with a model is, and latency turns out to be the feature.

Suno v3 and Udio generate full songs with vocals and lyrics from a sentence, and some of them are good. What that means for artists, platforms and listeners.

DeepMind's Genie learned a controllable world model from 2D platformer videos with no action labels, by inferring eight latent actions on its own. The model learns the controls as well as the game.

OpenAI's Sora generates minute-long videos that look astonishing. Its technical report calls it a world simulator. The samples show both why that's tempting and why it isn't true yet.

Stanford's two-armed robot on a wheeled base learned to cook, wipe spills and call an elevator from about fifty demonstrations per task. Cheap hardware plus teleoperation is changing robot data.

Google announced Gemini, trained from the start on text, images, audio and video together. Why training on mixed modalities from day one matters more than bolting vision onto a language model.

Pika 1.0, Stable Video Diffusion and Runway's latest make beautiful short clips. What's missing is the grammar of film: continuity, screen direction and eyelines.

DALL·E 3 in ChatGPT doesn't use what you typed. It has a language model write a long, detailed prompt for you. Creative tools are turning into agents that interpret intent.

Tesla's new video shows Optimus sorting colored blocks with a neural network trained end to end. Curated robot demos and messy child learning, and which one scales.

3D Gaussian Splatting represents a scene as millions of fuzzy, colored blobs and renders it in real time at NeRF quality. It does this without a neural network.

Meta's I-JEPA is the first concrete result from LeCun's world model agenda. It learns image representations by predicting hidden regions in latent space, with no augmentations and no pixel reconstruction.

Apple's headset builds a 3D 'Persona' of your face so you can appear on video calls while wearing it. Ten years after my thesis, 3D face capture ships inside a headset.

A new paper turns every item into a short code of semantic tokens, then has a Transformer generate the code of what you'll want next. The item vocabulary finally describes what things are.

An AI-generated song imitating Drake and The Weeknd got millions of plays before it was pulled. Voice is identity, and the industry needs consent and attribution systems, fast.

OpenAI's GPT-4 scores near the top of the bar exam and can explain a joke in a photo. Describing a scene is useful, but I'm still unsure how far that gets it toward understanding the physical world.

ControlNet lets you steer Stable Diffusion with a pose skeleton, a depth map or an edge sketch. Creators want to set the structure directly, and this gives them a way to.

Google Research's MusicLM generates music from text descriptions. The interesting part is its stack of tokens: one for meaning, one for sound, one shared between music and words.

A million people signed up in five days. The underlying model isn't new. What's new is the interface and the training to follow instructions, and that changes every software team.

A year after the dancer in the suit, Tesla showed a real humanoid prototype. It walked slowly and waved. The legs get the attention. The hands are the harder and more important problem.

Meta's Make-A-Video generates short clips from a sentence. They're five seconds long, low resolution and physically wrong in instructive ways. The hard parts are consistency, continuity and physics.

Stability AI released the weights of a text-to-image model anyone can run on a consumer GPU. Open models change who gets to build, and what gets built.

Yann LeCun's position paper argues that intelligence needs world models that predict in representation space, not pixels. Where I agree, and where I'd push back: bodies and hands.

OpenAI's VPT labeled 70,000 hours of Minecraft videos with the actions players took, using a small model trained on a little labeled data. Passive video became interaction data.

DeepMind's Gato plays Atari, captions images, chats and stacks blocks with a real robot arm, all with the same weights. Turning actions into tokens is the interesting part.

DALL·E 2 generates images that look like real photos and paintings from a sentence. Why the jump to video is much harder than the jump from GANs to this.

DeepMind's Chinchilla paper says large language models have far too many parameters for the data they see. The fix is more data, and that raises a question about where it comes from.

Meta lost over $230 billion in market value in one day after Facebook's first ever drop in daily users. The interest graph beat the social graph.

Nvidia's Instant NGP trains a neural radiance field in seconds with a multiresolution hash table. The representation mattered more than the network.

Two papers this week, GLIDE and latent diffusion, make text-to-image generation with diffusion models look practical. What denoising actually learns, and why it beats the adversarial game.

Meta is shutting down Facebook's face recognition system and deleting more than a billion face templates. An industry chapter closing, from someone who lived in it.

Facebook is now Meta and says it's building the metaverse. From someone who spent years on 3D faces: avatars that can't emote are just chat rooms with legs.

Treat each song as a token and each listening session as a sentence, and recommendation starts to look like language modeling. What that framing gets right, and what it misses.

Tesla's AI Day announced a humanoid robot and then showed a person dancing in a robot costume. The demo was a joke. The idea is right.

Why music recommendation is a different problem from video or news recommendation: sequences, repetition, passive listening, and the difference between a skip and a mood.

GitHub's Copilot preview suggests whole functions inside the editor. Two days of using it on side projects, and what changes when the IDE becomes a conversation.

I've started as an engineering manager at YouTube Music. Why music, and why I think the next few years of music discovery will look very different.

I'm leaving TTT Studios after seven and a half years and Amanda AI after three. What I learned growing an engineering team from ten to fifty, and co-founding a company through a pandemic.

DeepMind's Perceiver handles images, audio, video and point clouds with the same network, by cross-attending to a small latent array. Modalities stop needing their own models.

万能青年旅店 released their second album after ten years. On listening to a band that sings about industrial decline in Hebei, and why art gets better once you can see the structure under it.

OpenAI's CLIP learns from 400 million image and caption pairs to put images and text in one embedding space. Zero-shot classification is the demo. Shared embeddings are the real story.

DeepMind's AlphaFold 2 predicts protein structures about as accurately as experiments do. A fifty-year-old physics problem, solved mostly by learning from examples.

A paper under review at ICLR cuts images into patches and feeds them to a plain Transformer. With enough data, it beats convolutional networks. One architecture for everything is getting closer.

NeRF represents a 3D scene as a small neural network you can render from any viewpoint. No mesh, no triangles. My old thesis problem, answered from a different direction.

YouTube opened a beta of Shorts in India, where TikTok has been banned since June. Launch where the gap is, with the world's biggest music catalog behind you.

Instagram launched Reels the day before the TikTok executive order. The format is a commodity now, and the question is whose recommender learns fastest.

The demos flooding Twitter this week come from the same next-word objective as GPT-2, scaled a hundred times. Few-shot learning appeared on its own. Coding changes first.

In one week, three of the biggest companies in the field paused or ended police sales. Where I agree, and why a pause isn't a policy.

Three months into the pandemic, with no conferences on the calendar. What we're trying, what we decided not to build, and how it feels to run a startup whose market disappeared.

Fortnite's Astronomical event drew 12 million concurrent players. With every venue closed, a game became the biggest concert stage in the world.

Two weeks into working from home, every interaction is a video call. What that does to expectations for video, and what's missing from the tools.

The world's biggest mobile conference was cancelled over the new coronavirus. A founder's worries, a few weeks in, with friends in Wuhan under lockdown.

The New York Times revealed a startup that matches any photo against billions of faces scraped from social media, and sells it to police. The worst version of my industry is now public.

Looking back at 2010 to 2019 from the end of it: the moments that mattered to me, what I got right and wrong on this blog, and one bet for the 2020s.

DeepMind's MuZero plays Go, chess, shogi and Atari at top level without being told the rules. It plans inside a model it learned, and the model only predicts what matters.

Google claims quantum supremacy with a 53-qubit chip. What the chip actually computed, why IBM disputes the number, and why there's still no workload.

A Chinese face-swap app went viral over the weekend and hit a privacy wall within days. China just ran the consumer deepfake experiment at scale.

A year of live events taught us that benchmark accuracy barely predicts real-world accuracy. What we changed in how we collect data and evaluate.

Facebook open-sourced DLRM, its deep learning recommendation model. It shows what big recommenders actually look like: mostly memory, with a small network on top.

Microsoft quietly took down MS-Celeb-1M, ten million photos of a hundred thousand people. A lot of the face recognition industry was built on data nobody agreed to give.

The ban covers city agencies, not private companies like ours. It still changed a lot of sales conversations this week. Where I think the line should be.

The Event Horizon Telescope didn't take a picture in the ordinary sense. It filled in a mostly empty measurement with priors, and was careful about which ones.

Rich Sutton says 70 years of AI research show that general methods plus computation beat human knowledge every time. He's right, and it stings. My one objection is about data.

OpenAI trained a much bigger language model and is holding back the full version. The unicorn story is impressive. What's actually dangerous is cheap, plausible text at scale.

DeepMind's StarCraft agent beat two pros 10-0, then lost the one game where it had to move a camera like a person. What counts as intelligence when the body is different.

Nvidia's StyleGAN generates photographic faces with control over pose, identity and freckles. A face company's view of faces becoming free.

A curious agent gets hypnotized by random static. OpenAI's Random Network Distillation fixes it and beats humans at Montezuma's Revenge. What curiosity should actually reward.

Google's BERT beat almost every language benchmark by predicting hidden words using context on both sides. It's a representation model, which is a different thing from a text generator.

Our first live conference check-ins. Wait times dropped a lot. The model was the part that worked from day one. Lighting, queues and printers were not.

ByteDance merged Musical.ly into TikTok this week. A short follow-up to what I wrote in November, and what the feed tells you about the product.

OpenAI's Dactyl learned to rotate a block with a human-like robot hand, entirely in simulation. Why hands, and why randomizing the simulator is the clever part.

OpenAI trained a Transformer to predict the next word on thousands of books, then fine-tuned it on small tasks. Language is getting its ImageNet moment.

The NTSB's preliminary report on the Tempe crash shows the sensors worked. Classification flickered, braking was disabled, and the human was supposed to be the backup.

Google's Looking to Listen separates one voice from a crowd by watching the speaker's face. Modalities work better when they explain each other.

Ha and Schmidhuber's World Models compresses what an agent sees, learns to predict what happens next, and trains a tiny controller inside its own dream. The most important paper I've read this year.

I've co-founded a company that does face recognition on device. Why on device, why conference check-in first, and the lines we won't cross.

Google's NIMA predicts how people would rate a photo, as a distribution, not a single score. Modeling disagreement turns out to be the useful part.

Someone on Reddit is putting celebrities' faces into porn with a home GPU and open-source tools. How it works, and why consent has to be designed in now.

ByteDance is buying Musical.ly. It looks like a Chinese news company buying an American teen app. It's really a recommendation engine buying a global audience.

Starting from random play, with no human data, it beat the version that beat Lee Sedol 100 games to 0 after three days. Human knowledge was a ceiling.

The iPhone X projects 30,000 infrared dots onto your face to unlock the phone. Four years ago I was faking this in software from a single selfie.

DeepMind's locomotion agents learned to run, jump and duck from a reward for moving forward and a varied course. The flailing arms are the point.

A Google paper throws out the RNN and translates with attention alone. What attention actually computes, and why I think it goes beyond translation.

A Berkeley paper gives agents an internal reward for being surprised by their own predictions. It learns Mario with no score at all, and it answers a question I asked two years ago.

The world's top Go player lost 3-0 to AlphaGo in China this week, and Chinese viewers mostly couldn't watch it live. Why this match will matter more in China than the Lee Sedol one did.

Elon Musk's new company wants a high-bandwidth link between brains and computers. The argument is about output speed. I think it's partly right, and too optimistic on timing.

The new Zelda replaced scripted puzzles with a world that follows consistent rules. It's the best example I've seen of what a training environment for intelligence could look like.

Snap filed to go public and calls itself a camera company. The phrase sounds like marketing, but it describes a real shift in how people put things into computers.

CMU's Libratus beat four top professionals at heads-up no-limit hold'em over twenty days. Hidden information was supposed to protect humans a while longer.

DeepMind open-sourced its 3D lab and OpenAI released Universe in the same week. Environments are becoming the new datasets.

Neural machine translation replaced phrase tables, starting with Chinese to English. Notes from someone who reads both, and why the zero-shot result is the bigger story.

Twitter is shutting down Vine. It didn't lose because six-second videos were a bad idea.

DeepMind's WaveNet generates raw audio one sample at a time, and its speech sounds far more human than anything before. How, and why it's so slow.

Uber will put self-driving cars on Pittsburgh streets next month, six weeks after the first Autopilot fatality became public. The hard part of driving is the rare case.

The biggest augmented reality hit so far barely uses augmented reality. It's a map, built on years of data from another game.

My last paper from grad school is out. Why I still think geometric priors for faces matter when every result now comes from deep learning.

The Tensor Processing Unit has been running in Google's data centers for a year. When a company designs custom silicon for a workload, the workload is here to stay.

Facebook opened Messenger to bots at F8, two weeks after Microsoft's Tay learned racism from Twitter in a day. Why I'm telling clients to wait.

AlphaGo beat Lee Sedol four games to one. The move everyone is talking about, how the system found it, and why self-play is the part that matters.

Boston Dynamics' new Atlas gets shoved, loses its box and gets knocked flat, and keeps going. Why recovering is harder than walking.

Marvin Minsky died this week, the same week DeepMind announced a Go program that beat a professional. Two very different ideas of how minds get built.

Microsoft Research's residual networks won ImageNet with a network eight times deeper than last year's. The idea behind it is almost too simple.

Google released its internal machine learning library. What changes for a thirty-person app studio that builds software for other companies.

Angus Young is sixty and still runs across the stage in a school uniform. Some thoughts after the show.

3D Touch on the iPhone 6s, one year after I guessed pressure-sensitive phones were a couple of years away. How it measures force, and what it's for.

A new paper separates the content of an image from its style using a network trained for classification. How it works, and what it suggests about taste.

Apple Music launched with Beats 1 and human curators. Three weeks later, Spotify ships Discover Weekly. Two theories of music discovery.

Running a network in reverse to see what it learned. Why everything turns into dogs, and what that says about training data.

Unlimited storage and automatic face grouping, free. What happens to small companies when a platform ships their feature as a default.

Leaving the selfie-to-3D-face app I helped build out of my thesis. The technology mostly worked. The rest is the interesting part.

Half the internet sees white and gold, half sees blue and black. The disagreement is about lighting, and it says something about how vision works.

DeepMind's DQN paper is in Nature. What the network actually learns, and the game where it learns nothing at all.

Microsoft's headset is mostly a sensing problem with a display attached. Why AR needs a model of the space before it can put anything in it.

The GAN paper from NIPS this year: a generator, a discriminator, and a learned idea of what counts as real.

Google and Stanford both have networks that write sentences about images. How they work, and what they're actually learning.

Magic Leap raised $542 million without showing anyone a product. A look at the optics they're promising, and why it's so hard to wear on your head.

The part of the Apple Watch announcement I keep thinking about is a small mass on a spring, and what it says about how little computers use touch.

I presented a facial animation paper this week. What it's about, explained with a rubber sheet, some pins and a steel plate.

Our app gets the shape of your face right and still looks like a wax museum. Most of the problem turned out to be motion.

Two new papers on video classification, and why the network that only sees motion beat the one that sees the frames.

What my master's thesis on 3D face meshes does, which part of it I think neural networks will replace soon, and which part I think they won't.