Prompt effects shift across AI judges in multilingual tests
Across 27 languages, a study of four AI judges and nine prompting strategies found that the judge mattered more than the instructions it received.
Edited by Boris Djordjevic · Paperfoot
An empirical study tested nine prompting strategies for four language models acting as judges across 27 languages and four tasks. The tests covered 13 datasets and produced more than one million individual scores for outputs used as reference standards. The researchers validated the evaluation against human judgments.
The choice of judge mattered more than the prompting strategy. Strategies’ effects reversed direction across judges, and stability worsened in languages with fewer resources for language technology. Overall summaries concealed these effects. The authors concluded that prompting strategies needed validation for the intended judge, task, and language. They also said comparisons between studies should account for both the judge and its instructions.
Why it matters
These benchmark results challenge the assumption that instructions that help one AI evaluator will help another. They support checking evaluation instructions in the model, language, and task where they will be used.
Caveats
The abstract does not quantify how much more the judge mattered or give results for individual tasks and languages. The work is a preprint, not yet peer-reviewed.
From the paper
- Limitation
Gold-reference evaluation isolates judge behaviour from output variation but introduces ceiling effects in QA and reference-based MT, limiting strategy discrimination for those tasks (Section 4.1); we confirm this directly using AfroBench real system outputs but only for a subset of judges, tasks …
Quoted word for word from the full text on arXiv.
In this paper
multilingual LLM-as-a-judge evaluation
- Methods
- Aya Expanse 32B (evaluated)Gemma 3 27B (evaluated)Qwen2.5 32B (evaluated)Qwen2.5 72B (evaluated)Zero-shot (evaluated)Few-shot anchored (evaluated)Prometheus-style prompting (evaluated)Chain-of-Thought (evaluated)Role prompting (evaluated)Cross-Lingual prompting (evaluated)Monolingual prompting (evaluated)Trans-Eval (evaluated)
- Datasets
- SSA-MTE (evaluated)SummEval (evaluated)WMT MQM (evaluated)XQuAD (evaluated)PersianQA (evaluated)CatalanQA (evaluated)Belebele (evaluated)AfriQA (evaluated)AfroBench (evaluated)
Results
- Role prompting, Spearman correlation with human judgements: \rho=0.483
We evaluate agreement with human judgements on SSA-MTE, SummEval, and WMT MQM, where Role Prompting attains the highest aggregate correlation ( \rho=0.483 ).
1 Introduction
- Trans-Eval, cross-dataset mean Spearman correlation against human quality annotations: \rho=0.321
The cross-dataset mean is highest for Role Prompting ( \rho=0.483 ) and lowest for Trans-Eval ( \rho=0.321 ).
4.4 Agreement with Human Judgments
What it shows
- Prompting strategies do not transfer across judges because judge identity dominates prompting strategy and their interaction exceeds the strategy main effect.
We find that prompting strategies are not transferable: the choice of the judge model dominates the choice of prompting strategy, strategy effects reverse direction across judges, and stability degrades in low-resource languages, where these effects are concealed by aggregate reporting.
Abstract
- Target-language prompting yields opposite effects across judges on identical content.
Target-language prompting raises the mean score of Gemma 3 by 0.83 and lowers that of Aya Expanse by 0.32 on identical inputs, with a per-language maximum degradation of 2.16 on Yoruba.
1 Introduction
- Chain-of-Thought minimizes cross-lingual variability across evaluated prompting strategies.
Chain-of-Thought shows the lowest cross-lingual variability (mean CV =0.036 ), consistent with prior work, suggesting structured reasoning reduces variance by making intermediate steps explicit, though it does not consistently achieve the highest human alignment, indicating consistency does not guarantee quality …
Strategy effect.
- Role Prompting achieves the highest aggregate alignment with human judgments across evaluated datasets.
Among evaluated strategies, Role Prompting offers the best overall trade-off, achieving the highest human alignment ( \rho=0.483 ) while remaining relatively stable across languages.
Upshot.
Quoted word for word from the full text on arXiv.
The paper
Prompting Strategy Effects Depend on the Judge: A Multilingual Study of LLM-as-a-Judge Robustness
arXiv, Oct 22, 2026, Open access, CC BY-SA, Preprint, not peer-reviewed
doi.org/10.48550/arxiv.2610.31172
arXiv:2610.31172v1 in cs.CL, also cs.AI, cs.LG
Comments: 23 pages,
Cite as BibTeX
@misc{mehrabi2026prompting,
title = {Prompting Strategy Effects Depend on the Judge: A Multilingual Study of LLM-as-a-Judge Robustness},
author = {Mehrabi, Asal and Aliannejadi, Mohammad and Siro, Clemencia},
year = {2026},
eprint = {2610.31172},
archivePrefix = {arXiv},
primaryClass = {cs.CL},
doi = {10.48550/arxiv.2610.31172},
url = {https://arxiv.org/abs/2610.31172}
}