Google Translate got better overnight, and Chinese speakers noticed first
Neural machine translation replaced phrase tables, starting with Chinese to English. Notes from someone who reads both, and why the zero-shot result is the bigger story.
At the end of September Google switched Chinese to English in Google Translate to a neural system. This week it rolled the same approach out to seven more language pairs. If you read both Chinese and English, you noticed the first change before any announcement. My WeChat friends were sending each other screenshots of Google Translate producing sentences that sounded like a person wrote them.
I tested it on the kind of Chinese that used to break everything. Old-fashioned four-character idioms, long sentences with no explicit subject, news articles that stack clauses. It’s not perfect. It still misses idioms, and when it’s wrong it’s confidently fluent and wrong, which is a new kind of danger. But the gap from a few months ago is enormous. The old system gave you English words in roughly Chinese order. The new one gives you English.
The old system was phrase-based. It learned from large parallel corpora which short sequences of words tend to translate to which, then stitched those phrases together and used a language model to pick an arrangement that sounded least bad. Chinese and English differ a lot in word order and in what’s left implicit, so the stitching was where it fell apart.
The new one is an encoder-decoder network: the encoder reads the whole Chinese sentence into a sequence of vectors, and the decoder writes the English one word at a time, using attention to look back at the relevant parts of the source. There’s no phrase table and no rules about reordering. It learns all of that from pairs of sentences.
The paper that interests me more came out this week too: Google’s multilingual version. They trained one model on many language pairs at once, with a token at the start telling it which language to output. Then they asked it to translate between pairs it never saw in training, say Japanese to Korean, when it had only been trained on Japanese to English and English to Korean. And it could do it, reasonably well. The researchers looked at the internal representations and found that sentences with the same meaning in different languages end up close together, which they cautiously describe as evidence of a kind of interlingua.
That’s a big claim, but if it holds up it means the network isn’t just mapping words to words. It’s building an internal space where meaning is somewhat independent of language, as a side effect of being trained to translate. Nobody told it to.
As someone who spends a lot of time living between two languages, I find that moving. When I think, I’m often not sure which language I’m thinking in, and the meaning seems to sit somewhere underneath both. It’s strange to see a machine arrive at something similar just from reading.