Why are LLMs so good at translation?
If you used machine translation fifteen years ago, you remember the output: readable in places, funny in others, and full of words that had no business sitting next to each other. Today a large language model can hand you a translation with the right register and idioms, one that sounds like someone wrote it in the target language. Three developments got us here, and the same history explains why these models still make mistakes you can miss.
A base language model learns by predicting the next token in a huge pile of text. It gets no vocabulary drills or grammar lessons, and it picks up translation as a side effect of that training. The earlier machine translation systems show which problems that training ended up solving.


The memory bottleneck
Google Translate launched in 2006 on a rule-based engine and moved to its own statistical system in 2007. Phrase-based statistical machine translation, which ran the service until 2016, cut sentences into fragments and looked each one up in tables learned from human translations. A sentence close to the training data came out decent. A rare word or a long, nested clause broke it, because the system had no model of the sentence as a whole. It knew how fragments tended to translate and had no representation of what the sentence meant.
In 2014 Ilya Sutskever, Oriol Vinyals and Quoc Le at Google published sequence-to-sequence learning, building on encoder-decoder work by Kalchbrenner and Blunsom and by Cho and colleagues. One recurrent network, the encoder, reads the source sentence and compresses it into a single fixed-size vector. A second network, the decoder, generates the translation from that vector one word at a time. The whole system learns end to end from sentence pairs, with no phrase tables.
The design had two weaknesses. Squeezing a sentence of any length into one vector of fixed size lost information, and translation quality fell as sentences got longer. Training was also slow: a recurrent network processes a sentence one position after another, so you cannot spread the work within a sentence across many processors. Dzmitry Bahdanau, Kyunghyun Cho and Yoshua Bengio addressed the first problem in 2014 with attention. At each output step the decoder computes a weighted look at all of the encoder's states, focusing on the source words relevant to the word it is about to produce. When Google moved Translate to neural MT in 2016, its system (GNMT) combined recurrent networks with attention of this kind.
The 2017 breakthrough
In 2017 eight researchers, most of them at Google, published Attention Is All You Need. If attention does the heavy lifting, they asked, why keep the recurrence at all? Their architecture, the transformer, drops it. In each layer, every token attends to all the other tokens in the sequence at once (in the decoder, to the earlier ones) and builds its new representation from the tokens it weights most.
Take the sentence "The animal didn't cross the street because it was too tired." To translate "it" into German or French you need to know whether it refers to the animal or the street, because the two nouns have different grammatical genders. A recurrent model has to carry "animal" forward through every intervening step and hope the information survives. In a transformer, "it" can attend to "animal" within a single layer, however far apart the words sit. The model runs several attention heads in parallel, and researchers who inspect trained models find that some heads specialize, for example in syntactic relations or in linking pronouns to nouns. Many heads have no clean interpretation. Because attention itself ignores word order, the transformer adds positional encodings so the model knows where each token sits.


Removing recurrence made training parallel. During training the model sees the full target sentence and computes every position at once. At inference time it still generates output one token at a time. That parallelism let labs train on thousands of accelerators, and model sizes grew from the original transformer's 65 to 213 million parameters to GPT-3's 175 billion in three years. On the WMT 2014 English-to-German benchmark, the large transformer scored 28.4 BLEU, more than two points above the best previous results, ensembles included. On English-to-French it set a new single-model record at a fraction of the training cost. BERT, GPT and T5 all use the transformer, as do nearly all of today's large language models.
Try Fink for your next translation
See the translation, every change, and the why on one screen.
Open AppTranslation as a side effect of pretraining
A base language model is pretrained on one objective: predict the next token. It reads hundreds of billions of words of books, articles and web pages. For the classic base models, nobody assembled a parallel corpus or defined translation as a training task. The chat models you use today go through further training on instructions afterwards, and that data often includes translation requests, but the ability shows up before that step.
The pretraining data is mostly English. In GPT-3's training set about 93 percent of the words were English, which leaves around 7 percent for every other language combined. That slice contains a lot of incidental bilingual text: a French quote followed by its English rendering, a product manual printed in two languages, a forum post that switches from Spanish to English halfway through. Eleftheria Briakou, Colin Cherry and George Foster measured this for Google's PaLM in 2023 ("Searching for Needles in a Haystack") and found over 30 million translation pairs across at least 44 languages, none of them put there on purpose. In their small-scale ablations, removing that material hurt translation, though the effect shrank as models grew. The GPT-3 paper had already shown that few-shot translation improves steadily with model size, and at 175 billion parameters GPT-3 beat earlier unsupervised translation systems when translating into English. Surveys of multilingual models (Zhu et al., Huang et al.) cover this ability across many language pairs.
Interpretability work gives a partial answer to the question of mechanism. Models trained on several languages develop shared representations: the concept "dog" activates similar internal features whether it arrives in English, German or Japanese, and language-specific features take over in the last layers, where the model picks its output words. That study found the same shared structure in models trained without any English. In models trained mostly on English, such as Llama 2, Wendler and colleagues ("Do Llamas Work in English?", 2024) found that the intermediate layers sit closer to English than to the input or output language. Once a model represents meaning in a space that several languages share, translation needs one step less: the model reads the input into that space and writes the output from it.


Why the output reads better
Attention lets a model keep a whole passage in view, so a pronoun can resolve to a name from two paragraphs earlier, a verb agrees with a distant subject, and an ambiguous word takes its sense from the surrounding sentences. Older systems translated each sentence on its own and got these cases wrong.
A language model also learned the target language from a vast amount of text written in it, so its output follows that language's idioms and register instead of mirroring the source sentence's structure. It has seen text from most domains too, from manuals and contracts to medical papers and novels. He and colleagues (MAPS, TACL) prompted models to go through a translator's preparation first, extracting keywords, identifying the topic and finding similar example translations, and then translate. This reduced errors such as mistranslation and omission. For specialized work, a few example translations in the prompt or a round of fine-tuning can teach a general model a company's terminology without training it from scratch.
Where LLMs still stumble
Quality depends on how much text a language has. Languages with little web presence get little training data, and translation involving them is measurably weaker, as Gain and colleagues' survey documents. Models do best when English is one side of the pair and worse between two non-English languages. Even for major languages, a multi-domain evaluation found that quality drops as texts get longer, heavier in terminology or further from everyday prose.
The failures are hard to see. A phrase-based system produced word salad you could spot at a glance. Neural systems, from 2016 on, write fluent sentences even when they get the meaning wrong, and LLMs continue that pattern. A mistranslated technical term, a lost nuance or a register that slides from formal to chummy reads as smoothly as a correct translation. You only catch these errors if you compare the output with the source or have a second reference. Comparing source and output on purpose is also how you turn a translator into a study tool, which we cover in how to use a translator to learn a language.
Where Fink fits
Fink uses LLMs for translation and shows you what it changed and why, so you can check a rendering against the source instead of taking it on trust. The page on how Fink works has the details.
Sources and further reading
The history above rests on three papers: Sutskever, Vinyals and Le on sequence-to-sequence learning (2014), Bahdanau, Cho and Bengio on the attention mechanism (2014, published at ICLR 2015), and Vaswani and colleagues' Attention Is All You Need (2017), which introduced the transformer. On multilingual models, two surveys map the field, Multilingual Large Language Models: A Systematic Survey and A Survey on Large Language Models with Multilingualism. Tracing Multilingual Representations in LLMs with Cross-Layer Transcoders looks at shared internal features. Briakou, Cherry and Foster's "Searching for Needles in a Haystack" (ACL 2023) measures incidental bilingual data in PaLM, and Brown and colleagues' GPT-3 paper (2020) reports the language mix of its training data and its translation results by model size. On translation quality, Exploring Human-Like Translation Strategy with Large Language Models (TACL) describes the preparation steps above, How Well Do Large Reasoning Models Translate? supplies the multi-domain evaluation, and Bridging the Linguistic Divide surveys the open problems for low-resource languages. For a single readable introduction, IBM's What is a Transformer Model? is a good place to start. If you want to know how many people translate with these models, we looked at the usage numbers in a separate post.