Better ranking scores can hide collapse in text generation
Under the per-token ranking rule tested, generation collapsed in 14 of 17 base-model lineages, producing output dominated by one text fragment.
Edited by Boris Djordjevic · Paperfoot
ECHO, an empirical study of language models trained on response preferences, traced generation collapse to a shared shift in their internal states. The shift moved preferred and rejected responses together, so the ranking-based training penalty barely reacted. As it grew, it could raise the probability of one token—a piece of text—until that token dominated output.
Injecting the learned shift into an unmodified base model recreated collapse; equally large random shifts did not. Masking the shared component from the training penalty prevented the probability buildup in the flagship test. Across 17 runs using UltraFeedback response data, better rankings of held-out test responses went with generated samples that two public reward models—automated judges—preferred less often over the base model’s samples.
Why it matters
The experiments exposed a blind spot in ranking scores and identified an internal signal that could warn of generation collapse before it occurs.
Caveats
The 14-of-17 result concerned the per-token ranking rule studied; textbook direct preference optimization runs did not develop the one-token probability buildup. The work is an arXiv preprint.
From the paper
- Limitation
Our evidence spans 17 base-model lineages, two datasets, and adapter and full-parameter training, with two further seeds for the flagship settings and one seed elsewhere (Appendix B.16).
Quoted word for word from the full text on arXiv.
In this paper
preventing vocabulary collapse in preference optimization
- Methods
- centering (this paper)norm penalty (this paper)loudness penalty (this paper)residual penalty (this paper)both (this paper)per-token ranking (baseline)SimPO (baseline)textbook DPO (baseline)N-DPO (baseline)DPO-Positive (baseline)Llama-2-7BLlama-3.1-8B
- Datasets
- driving-preference lists (training)UltraFeedback (training)WikiText-103 (evaluated)
Results
- SimPO, stopping demand: about 2.4 nat
SimPO stops at about 2.4 nat.
Different losses push the offset by very different amounts.
- textbook DPO, stopping demand: about 0.3 nat per token
Textbook DPO sums the score over the whole response, which spreads the same margin across hundreds of tokens and stops it at about 0.3 nat per token.
Different losses push the offset by very different amounts.
- SimPO, offset norm, at the base budget: 33
The offset nonetheless grows from a norm of 33 at the base budget to 77 at three times the budget.
Different losses push the offset by very different amounts.
- SimPO, maximum token lift, from public SFT checkpoint at three times the budget: 4.3 nat
It stays quiet, lifting no token by more than 4.3 nat, against about 18 nat for SimPO from the base models at the same budget.
Different losses push the offset by very different amounts.
- SimPO, preference share over SFT model by two public reward models, one-fold run on held-out prompts: 63 to 65%
Two public reward models agree, preferring the one-fold run’s samples to the SFT model’s on 63 to 65% of held-out prompts but the three-fold run’s on only 33 to 34%.
Different losses push the offset by very different amounts.
- both, held-out pairwise accuracy, up to three times the base budget: between 0.94 and 0.96
The cost to ranking performance is small, since held-out pairwise accuracy settles between 0.94 and 0.96.
Cross-family validation up to three times the base budget.
What it shows
- A common-mode offset, shared across positions, prompts, and chosen and rejected responses, causes vocabulary well formation and generation collapse in preference optimization while being barely penalized by relative ranking losses.
Preference optimization can improve every ranking metric while building a hidden, shared representation shift that the relative loss barely sees, one that can eventually destroy generation.
Abstract
- Centering and adding penalty terms acting on the common-mode offset prevents collapse up to three times the base budget with minimal cost to ranking performance.
Controlling the offset is therefore enough to prevent collapse up to three times the base budget, at almost no cost to ranking.
Cross-family validation up to three times the base budget.
- Higher held-out pairwise ranking accuracy on UltraFeedback negatively correlates with sample preferences from external reward models compared to the base model.
Across these runs, the better a run’s held-out pairwise accuracy, the less often the external judge prefers its sample to the base model’s, with a Spearman correlation of -0.78 for Skywork-Reward-V2 and -0.79 for ArmoRM.
An external judge disagrees with the ranking metrics.
- A vocabulary well forms only after the dose, the ratio of the offset's logit advantage to the base model's top-token gap, exceeds one.
In the offset-only state, the well overtakes the base model’s top token where the dose D_{t}=g_{t}/m_{t} exceeds one (Figure 3a).
A well forms only after the offset’s gain passes the base gap.
Quoted word for word from the full text on arXiv.
The paper
ECHO: Emergent Common-Mode Hidden Offsets Drive Vocabulary Collapse in Preference Optimization
arXiv, Oct 21, 2026, Open access, CC BY, Preprint, not peer-reviewed
doi.org/10.48550/arxiv.2610.29410
arXiv:2610.29410v1 in cs.CL
Comments: 28 pages, 13 figures
Cite as BibTeX
@misc{li2026echo,
title = {ECHO: Emergent Common-Mode Hidden Offsets Drive Vocabulary Collapse in Preference Optimization},
author = {Li, Yun and Javanmardi, Ehsan and Thompson, Simon and Gong, Jiachen and Zhang, Qunli and Zeng, Zifan and Liu, Shiming and Wang, Peng and Guo, Zixuan and Tsukada, Manabu},
year = {2026},
eprint = {2610.29410},
archivePrefix = {arXiv},
primaryClass = {cs.CL},
doi = {10.48550/arxiv.2610.29410},
url = {https://arxiv.org/abs/2610.29410}
}