Mistral vs GLM: a French model and a Chinese model, split by writing system
Across 375 translation pairs in 25 target languages, neither model came out ahead overall. Sorted by writing system, the two separate: Mistral Medium 3.5 won more often translating into Latin-script languages, and GLM 5.2 won more often translating into everything else. Mistral's win rate drops by 23 points between the two groups (p = 0.003), in the direction you would guess from where the models come from. Mistral AI is based in Paris, and GLM comes from Zhipu AI in Beijing.
The size of the gap surprised us more than its direction. The same 162 decisive comparisons that show no overall winner come apart as soon as you sort them by script, and the gap is large enough to change which model you would pick for a given language. We only found the split after looking at the data, so treat it as a strong lead rather than a confirmed effect.
Mistral vs GLM: results at a glance
| Mistral Medium 3.5 | GLM 5.2 | |
|---|---|---|
| Wins, of 162 decisive | 73 | 89 |
| Win rate | 45.1% (37.6-52.7) | 54.9% (47.3-62.4) |
| Into Latin scripts | 55.8% | 44.2% |
| Into non-Latin scripts | 32.9% | 67.1% |
| chrF++ | 52.9 | 54.5 |
| BLEU | 32.0 | 33.6 |
| Judge score, overall | 8.28 | 8.28 |
| Latency, median | 987 ms | 1,082 ms |
| Latency, 95th percentile | 2,113 ms | 34,191 ms |
In the overall rows the two models are close enough to swap without noticing. The two bold rows differ: Mistral won most comparisons into Latin scripts, and GLM won two thirds of those into non-Latin scripts. GLM's 95th-percentile latency is also about 16 times Mistral's, which you will feel if you translate while you type.
Mistral wins Latin scripts, GLM wins non-Latin scripts
Mistral won 55.8% of the 86 decisive comparisons into Latin-script languages and 32.9% of the 76 into non-Latin scripts. A two-proportion test gives p = 0.003 for that 23-point difference: if script made no difference, you would see a gap at least this large about three times in a thousand.
We expected the direction. Tokenisation and training data both vary by writing system, and a lab has the most text in, and the most reason to test, the languages closest to home. A model that has seen more Devanagari, Thai and Bengali text, and tokenises it more efficiently, has an advantage there even if it is no better a translator in general.
The size is the useful finding. Overall, these two models are a coin flip. Sort the same 162 comparisons by script and a 23-point gap appears at p = 0.003. So if you are choosing a model for translation, ask which one is best into the language you need; the overall ranking hides the split.
Per language, the picture is noisier. GLM won all five decisive comparisons into Bengali, and Mistral won all five into Indonesian, while Dutch and Portuguese split evenly. Mistral led in German, Czech, Polish and Romanian, and in the Latin-script Asian languages Indonesian, Vietnamese and Turkish. GLM led in Thai, Hindi, Chinese, Korean and Persian, and it also took Italian, Spanish and Swedish despite their Latin script. Japanese went to Mistral, 5 to 3.
Two caveats. We grouped by script after seeing the data, not before, so read p = 0.003 as a strong lead rather than a clean test. The predictable direction helps, but we did not pre-register the cut. And with fifteen sentences per language, any single row of that chart could move a lot on a rerun. Trust the grouped result more than any one language. If you are picking a model for one language pair, use the chart to decide where to start testing.
Overall, the two models tied
GLM 5.2 won 89 comparisons and Mistral Medium 3.5 won 73. The other 213 of the 375 ended without a winner. A sign test on the 162 decisive ones gives p = 0.21: a fair coin would produce a gap at least this large about one time in five.
A run of the same benchmark the day before had Mistral ahead, 107 to 98. When the leader flips between runs, the difference is noise: toss the same coin twice and the count moves. For that reason we do not claim GLM is 10% better, and we would not have claimed the reverse after the first run.
What chrF++ and BLEU say
We also scored every output against the FLORES-200 reference, a translation made by a professional translator, using chrF++ and BLEU. Both metrics measure how closely a translation matches the reference. They cannot tell you whether a translation is good on its own terms.
Mistral scored 52.9 chrF++ and 32.0 BLEU, GLM 54.5 and 33.6. The judge also gave each translation a score from one to ten for accuracy, fluency and appropriateness, and the average came out at 8.28 for both models, identical to three decimal places.
These measurements work in different ways and agree. The win count and both reference metrics lean slightly toward GLM, the judge's scores don't lean at all, and none of the gaps is big enough to call. One near-tie can come from an underpowered test; we see it in the win count, the reference metrics and the absolute scores at once.
Try Fink for your next translation
See the translation, every change, and the why on one screen.
Open AppWhat the disagreements look like
A win percentage hides what a single win looks like. We picked one source sentence and four of its target languages, two won by each model, with the judge's verdict for each. The English source is:
This sediment was necessary for creating sandbars and beaches, which served as wildlife habitats.
In nearly every case the disagreement comes down to one content word, sandbar: a ridge of sand in a river or off a coast. Some of the translations below turn it into a dune or a "sandy rod".
| Indonesian | Mistral wins |
|---|---|
| Mistral | Sedimen ini diperlukan untuk membentuk gosong pasir dan pantai, yang berfungsi sebagai habitat satwa liar. |
| GLM | Sedimen ini diperlukan untuk menciptakan sandbar dan pantai, yang berfungsi sebagai habitat satwa liar. |
| Reference | Sedimen ini diperlukan untuk menciptakan gundukan pasir dan pantai, yang berfungsi sebagai habitat satwa liar. |
GLM left sandbar in English. Mistral used gosong pasir, the Indonesian term. The rest of the two translations barely differ, and the second clause is word for word the same.
| Hindi | Mistral wins |
|---|---|
| Mistral | यह तलछट रेत के टीलों और समुद्र तटों के निर्माण के लिए आवश्यक थी, जो वन्यजीवों के आवास के रूप में काम आते थे। |
| GLM | यह अवसाद रेतीली छड़ें और समुद्र तटों के निर्माण के लिए आवश्यक था, जो वन्यजीव आवासों के रूप में कार्य करते थे। |
| Reference | वाइल्डलाइफ़ हैबिटेंट्स के रूप में काम करने वाली रेती और तटों को बनाने के लिए गाद ज़रूरी थी. |
GLM's अवसाद can mean sediment in technical Hindi, but most readers know it as depression in the mood sense, and the reference uses the everyday गाद (silt). GLM's bigger miss is रेतीली छड़ें, literally "sandy rods". Mistral's रेत के टीलों means sand dunes, which is wrong too, though a dune at least belongs on a shoreline.
| Thai | GLM wins |
|---|---|
| Mistral | ตะกอนนี้จำเป็นต่อการก่อตัวของหาดทรายและชายหาด ซึ่งทำหน้าที่เป็นแหล่งที่อยู่อาศัยของสัตว์ป่า |
| GLM | ตะกอนนี้จำเป็นสำหรับการสร้างสันทรายและหาดทราย ซึ่งทำหน้าที่เป็นแหล่งที่อยู่อาศัยของสัตว์ป่า |
| Reference | การสร้างตลิ่งทรายและหาดทรายซึ่งทำหน้าที่เป็นถิ่นที่อยู่ของสัตว์ป่านั้นจำเป็นต้องใช้ตะกอนนี้ |
GLM used สันทราย for sandbar and หาดทราย for beach, so it names two landforms. Mistral used หาดทราย and ชายหาด, which in Thai both mean beach, so its sentence names the same landform twice where the source names two.
| Bengali | GLM wins |
|---|---|
| Mistral | এই পলি বালিয়াড়ি এবং সমুদ্রসৈকত সৃষ্টির জন্য প্রয়োজনীয় ছিল, যা বন্যপ্রাণীর আবাসস্থল হিসেবে কাজ করত। |
| GLM | এই পলিটি বালুচর এবং সৈকত তৈরির জন্য প্রয়োজনীয় ছিল, যা বন্যপ্রাণীদের বাসস্থান হিসেবে কাজ করত। |
| Reference | বালুচর এবং সমুদ্রসৈকত তৈরি করার জন্য এই অধঃক্ষেপের প্রয়োজন ছিল, যা বন্যজীবনের বাসস্থান হিসাবে কাজ করে। |
GLM picked বালুচর, the same word the professional translator used. Mistral picked বালিয়াড়ি, a sand dune again. chrF++ agrees with the judge here, 48.4 against 40.9.
Three of the four went the way the script split predicts. Hindi did not: Mistral lost Hindi 2 to 6 across the whole run, and this sentence is one of its two wins there. A 23-point edge shifts the odds for each sentence without deciding any of them.
How we benchmarked two translation models
Fink, our translation app, runs on Mistral models served from EU infrastructure, so customer text stays in the EU. We ran this benchmark to check whether that choice costs translation quality against a strong model from outside Europe.
We took fifteen English sentences from FLORES-200, a public benchmark set with professional translations, and had both models translate each one into 25 languages with the prompt Fink uses in production. That gave 750 translations. Claude Sonnet 5 then judged each pair on accuracy, fluency and appropriateness, and we ran every comparison a second time with the two candidates swapped. The judging added another 750 calls, for 1,500 in total.
The LLM benchmark page has the full results and updates whenever we publish a new run. It shows win rates with confidence intervals, breakdowns by difficulty and language, and all 375 comparisons with the source text, both translations, the professional reference and the judge's reasoning. If you think the judge got one wrong, you can check.
What this means if you translate with an LLM
In each of the four examples above, the error is one content word inside a sentence that is otherwise fluent and grammatical. रेतीली छड़ें and ชายหาด look fine on the page, and unless you read the language you have no way to notice them. LLM output reads as smoothly when a technical term is mistranslated or the register drifts from formal to chummy. That fluency is part of why LLMs are so good at translation, and it is also why you need something to check the output against, such as a back-translation or a second model reviewing the first. We looked at how people ended up translating with these models in a separate post.
In Fink, an MQM quality check flags issues in accuracy, fluency, terminology and style, and you accept or reject each suggestion. The insights panel adds a back-translation, notes on meaning, and pronunciation for languages you cannot sound out.
Questions people ask
Which LLM is best for translation? It depends on the language you translate into. In this benchmark neither Mistral Medium 3.5 nor GLM 5.2 was significantly better overall, with 73 to 89 wins out of 162 decisive comparisons (p = 0.21). Mistral won 55.8% of comparisons into Latin-script languages and GLM won 67.1% into non-Latin scripts. That split is significant at p = 0.003, though we found it after looking at the data.
Is Mistral or GLM better for translation? Neither, overall. Mistral led in most Latin-script languages, including German, Czech and Indonesian. GLM led in Thai, Hindi, Bengali, Chinese, Korean and Arabic, and also in Italian, Spanish and Swedish. GLM's mean latency was about six times Mistral's, because it ran with reasoning enabled, its default.
Is Mistral good at translation? In this run Mistral Medium 3.5 scored 8.28 out of 10 from the judge, 52.9 chrF++ and 32.0 BLEU against professional reference translations. That is statistically level with GLM 5.2 running with reasoning, at a median latency under one second.
Which model is better for Chinese, Japanese or Arabic translation? GLM led in Chinese, Korean, Thai, Hindi, Bengali, Arabic and Persian in this run. Japanese was the one non-Latin language where Mistral led, 5 to 3. No language had more than eleven decisive comparisons, so treat any single language as a lead to test, not a settled result.
Why is GLM 5.2 slower than Mistral? We ran GLM with reasoning on, its default, so it generates hidden reasoning tokens before the answer. Mistral Medium 3.5 ran with reasoning off. The median barely moves, 1,082 ms against 987 ms, but at the 95th percentile GLM took 34,191 ms against Mistral's 2,113 ms.
What is LLM-as-judge for translation evaluation? A third LLM compares two translations of the same source text and picks the better one. We ran each comparison twice with the order swapped, and the judge contradicted itself on 92 of 375 (25%). A single verdict from an LLM judge is noisy evidence, and you should not treat it as ground truth.
How do you benchmark a translation model? Pick a fixed set of source sentences that have professional reference translations, have each model translate them into your target languages, and compare the outputs. We used 15 FLORES-200 sentences, 25 target languages, pairwise LLM judging with an order swap, and chrF++ and BLEU scores against the references.
What is chrF++? A metric that scores a translation by the character n-grams it shares with a reference, plus word unigrams and bigrams (the "++"). Most of its signal comes from characters, so it works for Chinese, Japanese and Thai, which do not put spaces between words. BLEU counts words, which is why this run has no BLEU score for some languages.
Method and limits
The benchmark compares mistral-medium-3.5 (reasoning off) and zai-glm-5-2 (GLM 5.2, reasoning on by default), judged by Claude Sonnet 5. We served both candidates from the same EU endpoint, so infrastructure is the same for both. The benchmark page describes the full method and the code, and our chart script reads every number in this post's figures from that run's results.
Limits: fifteen sentences per language is a small sample, and the per-language intervals are wide, so read the by-language chart as texture rather than a ranking. We found the script split in the data instead of pre-registering it, so its p-value is a strong lead, not a confirmed effect. Every source sentence is English, drawn from Wikinews, Wikijunior and Wikivoyage, so the results say nothing about translating out of other languages, or about the emails, interface text and documents most people translate. We defined difficulty by sentence length alone. FLORES-200 is public, so these exact sentences may be in either model's training data. One judge model rated every pair, with no human check, and when we swapped the order it changed its answer on 92 of 375 comparisons. That rate is the best evidence we have for how much weight one of its verdicts carries. Finally, we ran the two models in different inference modes, so we report their latencies separately and never average them together.