Mistral vs GLM: a French model and a Chinese model, split by writing system
Neither model is better at translation overall. That held across 375 translation pairs, 25 target languages and 1,500 API calls. Split by writing system, though, the two are not close at all: Mistral Medium 3.5 is stronger translating into Latin-script languages, and GLM 5.2 is stronger translating into everything else. A 23-point gap, statistically significant, and in exactly the direction you would guess from a map. Mistral AI builds in Paris. GLM comes from Zhipu AI, in Beijing.
The direction is not the interesting part. What is interesting is the size, and the fact that it is visible at all: the same 162 comparisons that show no winner overall come apart cleanly the moment you sort them by script. A model's training data leaves a fingerprint in what it translates well, and that fingerprint is legible enough to change which model you should pick.
Fink runs on Mistral models, on EU infrastructure, because we do not want customer text leaving the EU. That is a constraint we accepted on privacy grounds, and the honest worry about any constraint is that it costs you quality. This benchmark was us checking.
Mistral vs GLM: the short answer
| Mistral Medium 3.5 | GLM 5.2 | |
|---|---|---|
| Wins, of 162 decisive | 73 | 89 |
| Win rate | 45.1% (37.6-52.7) | 54.9% (47.3-62.4) |
| Into Latin scripts | 55.8% | 44.2% |
| Into non-Latin scripts | 32.9% | 67.1% |
| chrF++ | 52.9 | 54.5 |
| BLEU | 32.0 | 33.6 |
| Judge score, overall | 8.28 | 8.28 |
| Latency, median | 987 ms | 1,082 ms |
| Latency, 95th percentile | 2,113 ms | 34,191 ms |
Read across the top and the two models are indistinguishable. Read the two bold rows and they are not remotely the same model. That is the whole story, and the rest of this post is how we got there and how much to trust it.
Mistral wins Latin scripts, GLM wins non-Latin scripts
Here is the part you can act on. Across 86 decisive comparisons into Latin-script languages, Mistral won 55.8%. Across 76 decisive comparisons into non-Latin scripts, it won 32.9%. A two-proportion test puts that 23-point swing at p = 0.003, which means a gap this size turning up by chance is about a three-in-a-thousand event.
None of which should be shocking, and we are not going to pretend otherwise. Tokenisation and training-data distribution both track writing system, and a lab's own languages are the ones it has the most text for and the most reason to test against. A model that has seen proportionally more Devanagari, Thai and Bengali text, and tokenises it more efficiently, has a real advantage there that has nothing to do with whether it is a better translator in general.
What is worth noticing is that the effect is large enough to see straight through a benchmark that finds nothing else. Overall, these two models are a coin flip. Group the same 162 comparisons by script and a 23-point gap falls out at p = 0.003. If a model's origin is that legible in its output, then "which model is best at translation" is the wrong question to be asking. The useful question is "best into what".
The per-language view says the same thing at finer grain. Some languages are lopsided: GLM won every decisive comparison into Bengali, Mistral won every one into Indonesian. Others are dead even. Mistral's wins cluster in European Latin-script languages such as German, Czech, Polish and Romanian, plus the Latin-script Asian languages Indonesian, Vietnamese and Turkish. GLM's cluster in Thai, Hindi, Chinese, Korean and Persian.
Two caveats, and they matter. We grouped by script after seeing the data rather than before, so read p = 0.003 as a strong lead rather than a clean test. That the direction was predictable helps; that the specific cut was not pre-registered does not. And with fifteen sentences per language, any single row of that chart could move a lot on a rerun. The grouped result is the one to trust; the individual languages are texture. If you are choosing a model for one specific language pair, this chart tells you where to start looking, not what to conclude. Scripts also carry their own difficulties beyond tokenisation, which is part of why we built pronunciation support for languages most people cannot sound out.
Overall, the two models tied
GLM 5.2 won 89 comparisons. Mistral Medium 3.5 won 73. Of 375 total, 162 were decisive and 213 were not. A sign test on those 162 gives p = 0.21: you would see a gap this size from a fair coin about one time in five.
An earlier run of the same benchmark had Mistral ahead, 107 to 98. The direction of the result flipped between runs. That is exactly what you expect when the difference is noise: run the same coin toss twice and the count moves, even though the coin has not changed. It is also the reason we do not claim GLM is 10% better, and would not have claimed the reverse a week earlier.
What chrF++ and BLEU say
Pairwise judging is not the only way to score a translation. Every output was also compared against the FLORES-200 reference, the translation a professional human produced, using chrF++ and BLEU. These are automatic similarity metrics: they cannot tell you whether a translation is good, only how closely it resembles a known-good one.
chrF++ came in at 52.9 for Mistral and 54.5 for GLM. BLEU at 32.0 and 33.6. And the judge's own averaged score, on a one-to-ten scale across accuracy, fluency and appropriateness, landed at 8.28 for both models, identical to three decimal places.
That agreement is the point. Three measurements that work in completely different ways all say "too close to call", and all lean fractionally the same direction. A tie that shows up once can be an underpowered test. A tie that shows up in the win count, the reference metrics and the absolute scores is a tie.
Try Fink for your next translation
See the translation, every change, and the why on one screen.
Open AppWhat the disagreements actually look like
Percentages hide what a win actually is. So here is one sentence, translated into four languages by both models, with what the judge made of each. It is the same English sentence every time:
This sediment was necessary for creating sandbars and beaches, which served as wildlife habitats.
Four languages, two wins each way, and in nearly every case the whole disagreement comes down to one content word: sandbar. It is a specific thing, a ridge of sand in a river or off a coast. It is not a beach, not a dune, and not a sandy rod.
| Indonesian | Mistral wins |
|---|---|
| Mistral | Sedimen ini diperlukan untuk membentuk gosong pasir dan pantai, yang berfungsi sebagai habitat satwa liar. |
| GLM | Sedimen ini diperlukan untuk menciptakan sandbar dan pantai, yang berfungsi sebagai habitat satwa liar. |
| Reference | Sedimen ini diperlukan untuk menciptakan gundukan pasir dan pantai, yang berfungsi sebagai habitat satwa liar. |
GLM gave up and left sandbar in English. Mistral used gosong pasir, which is the actual Indonesian term. Everything else is near-identical: the second clause is word for word the same in both.
| Hindi | Mistral wins |
|---|---|
| Mistral | यह तलछट रेत के टीलों और समुद्र तटों के निर्माण के लिए आवश्यक थी, जो वन्यजीवों के आवास के रूप में काम आते थे। |
| GLM | यह अवसाद रेतीली छड़ें और समुद्र तटों के निर्माण के लिए आवश्यक था, जो वन्यजीव आवासों के रूप में कार्य करते थे। |
| Reference | वाइल्डलाइफ़ हैबिटेंट्स के रूप में काम करने वाली रेती और तटों को बनाने के लिए गाद ज़रूरी थी. |
Two words wrong in one sentence. GLM's अवसाद is a real word that mostly means depression in the mood sense rather than the geological one, and its रेतीली छड़ें is literally "sandy rods". Mistral's रेत के टीलों is not exactly right either, it means sand dunes, but a dune is at least a thing that exists on a shoreline.
| Thai | GLM wins |
|---|---|
| Mistral | ตะกอนนี้จำเป็นต่อการก่อตัวของหาดทรายและชายหาด ซึ่งทำหน้าที่เป็นแหล่งที่อยู่อาศัยของสัตว์ป่า |
| GLM | ตะกอนนี้จำเป็นสำหรับการสร้างสันทรายและหาดทราย ซึ่งทำหน้าที่เป็นแหล่งที่อยู่อาศัยของสัตว์ป่า |
| Reference | การสร้างตลิ่งทรายและหาดทรายซึ่งทำหน้าที่เป็นถิ่นที่อยู่ของสัตว์ป่านั้นจำเป็นต้องใช้ตะกอนนี้ |
GLM used สันทราย for sandbar and หาดทราย for beach: two landforms, two words. Mistral used หาดทราย and ชายหาด, which in Thai are near-synonyms for beach. The source names two different things. One translation names two things, the other names the same thing twice.
| Bengali | GLM wins |
|---|---|
| Mistral | এই পলি বালিয়াড়ি এবং সমুদ্রসৈকত সৃষ্টির জন্য প্রয়োজনীয় ছিল, যা বন্যপ্রাণীর আবাসস্থল হিসেবে কাজ করত। |
| GLM | এই পলিটি বালুচর এবং সৈকত তৈরির জন্য প্রয়োজনীয় ছিল, যা বন্যপ্রাণীদের বাসস্থান হিসেবে কাজ করত। |
| Reference | বালুচর এবং সমুদ্রসৈকত তৈরি করার জন্য এই অধঃক্ষেপের প্রয়োজন ছিল, যা বন্যজীবনের বাসস্থান হিসাবে কাজ করে। |
GLM picked বালুচর, the same word the professional translator reached for. Mistral picked বালিয়াড়ি, a sand dune again. chrF++ agrees with the judge here: 48.4 against 40.9.
Two of those four went the way the script split predicts and two did not, which is what a 23-point edge looks like from close up. It is a tendency, not a rule. Hindi is the clearest illustration: Mistral lost Hindi 2 to 6 across the whole run, and this sentence happens to be one of the two it won.
How we benchmarked two translation models
Fifteen English sentences from FLORES-200, a public benchmark set of professionally translated text, each translated into 25 languages by both models using the exact prompt our app uses in production. That is 750 translations. Then a third model, Claude Sonnet 5, acting as judge, compared each pair on accuracy, fluency and appropriateness, and every comparison was run a second time with the two candidates swapped. Another 750 calls. Fifteen hundred in total, about an hour of wall time.
Everything is on the LLM benchmark page, updated automatically whenever we publish a new run: win rates with confidence intervals, breakdowns by difficulty and language, and every one of the 375 comparisons with the source text, both translations, the professional reference, and the judge's reasoning. If you think the judge got one wrong, you can go and look.
What this means if you translate with an LLM
Look at what those four disagreements have in common. Every one is a single content word, chosen wrongly, sitting inside a sentence that is otherwise fluent, grammatical and completely confident. Nothing about रेतीली छड़ें or ชายหาด looks wrong on the page. You cannot hear it, you cannot see it in the shape of the sentence, and unless you read the language you have no way of knowing it happened. LLMs produce fluent, confident prose every time, including when they are wrong. A mistranslated technical term, a nuance that evaporated, a register that drifted from formal to chummy: each reads just as smoothly as a correct translation. Those are exactly the errors a reader cannot see without something to compare against, which is part of why LLMs are so good at translation and also why that fluency is dangerous.
This is the problem Fink was built to solve. Rather than asking you to trust a black box, Fink shows the translation, every change along the way, and the reasoning behind it. The MQM quality check runs a second pass that flags specific issues, in accuracy, fluency, terminology and style, and you accept or reject each suggestion. The insights panel gives you a back-translation so you can verify meaning, meaning notes for nuance, and pronunciation for languages you cannot sound out. The benchmark told us our model choice does not cost us quality. The product tells you what the translation actually does. If you want the wider context on how people ended up translating with these models at all, we looked at the usage numbers separately.
Questions people ask
Which LLM is best for translation? It depends on the target language. On this benchmark neither Mistral Medium 3.5 nor GLM 5.2 was significantly better overall, splitting 73 to 89 wins out of 162 decisive comparisons (p = 0.21). But Mistral won 55.8% into Latin-script languages and GLM won 67.1% into non-Latin scripts, and that difference is significant at p = 0.003.
Is Mistral or GLM better for translation? Neither, overall. Mistral is the stronger choice into Latin-script languages such as German, French, Polish and Indonesian. GLM is the stronger choice into Thai, Hindi, Bengali, Chinese, Korean and Arabic. GLM is also about six times slower on average, because it runs with reasoning enabled by default.
Is Mistral good at translation? Yes. It scored 8.28 out of 10 on the judge's overall rating, 52.9 chrF++ and 32.0 BLEU against professional reference translations, statistically level with a reasoning-enabled competitor, and it did so with a median latency under a second. It is the model Fink uses in production, on EU infrastructure.
Which model is better for Chinese, Japanese or Arabic translation? GLM led in Chinese, Korean, Thai, Hindi, Bengali, Arabic and Persian in this run. Japanese was the exception among non-Latin scripts, where Mistral led. With fewer than ten decisive comparisons per language, treat any single language as a lead to follow rather than a settled result.
Why is GLM 5.2 slower than Mistral? It runs with reasoning enabled by default, which generates hidden tokens before the answer. That barely moves the median, 1,082 ms against 987 ms, but it moves the tail enormously: 34,191 ms against 2,113 ms at the 95th percentile.
What is LLM-as-judge for translation evaluation? A third LLM compares two translations of the same source text and picks the better one. We ran each comparison twice with the order swapped to check consistency. The judge contradicted itself 25% of the time, which is a useful reminder that LLM judgements are noisy, not ground truth.
How do you benchmark a translation model? Translate a fixed set of source sentences into many target languages, then compare outputs. We used FLORES-200 sentences, 25 target languages, pairwise LLM judging with an order-swap control, and chrF++ and BLEU scoring against professional reference translations.
What is chrF++? A metric that compares a translation to a reference by looking at character n-grams. Unlike BLEU, which splits on spaces, chrF++ works for languages such as Chinese, Japanese and Thai that do not separate words with spaces.
Method and limits
The benchmark compares mistral-medium-3.5 and zai-glm-5-2 (GLM 5.2), judged by Claude Sonnet 5. Both candidates are served from the same EU endpoint, so infrastructure is held constant. The full method, and the code, are described on the benchmark page, and every figure in this post is generated directly from that run's results.
What this does not establish: fifteen sentences per language is a small sample, and the per-language intervals are wide enough that the by-language chart should be read as texture rather than a ranking. The script split was found in the data rather than pre-registered, so its p-value should be read as a strong lead rather than a confirmed effect. Every source sentence is English and Wikipedia-derived, so nothing here speaks to translating into English, or to the email, interface and document text most people actually translate. Difficulty is defined purely by sentence length. FLORES-200 is public, so we cannot rule out that these exact sentences appeared in either model's training data. A single judge model rated every pair, with no human validation. Running every comparison a second time with the order swapped, the judge changed its answer on 92 of 375, so a quarter of its verdicts did not survive the control; that rate is the best available evidence for how much weight one such judgement carries. Finally, the two models were not run in the same inference mode, so their latencies are reported separately and never averaged together.