Larger vision-language models can perform worse on specific tasks
A study of 19 vision-language models across eight multimodal benchmarks identified 375 settings where larger models performed worse than smaller counterparts.
Edited by Boris Djordjevic · Paperfoot
The authors conducted an empirical study of 19 existing vision-language models, systems that process both images and text, from four distinct families across eight multimodal benchmarks and nine evaluation splits. Each model answered 29,386 questions, totaling 558,334 evaluations.
Across pairs within each family, the authors identified 375 subtask combinations where larger models performed worse than smaller ones. These spanned 4,342 unique questions where a smaller model answered correctly but a larger model failed. The patterns replicated across independently developed architectures. Furthermore, 13 subtask-dataset pairs showed inverse scaling across multiple families, which the authors described as consistent with a task-driven phenomenon. The most affected domains included multi-image spatial reasoning, visual perception, and university-level science, technology, engineering, and mathematics knowledge. Accuracy gaps exceeded 20 percentage points in several cases, while diagram understanding appeared comparatively resistant.
Why it matters
The results challenge the assumption that increasing model size uniformly improves multimodal capabilities. Based on evidence across eight multimodal benchmarks, the findings suggest that model selection should be task-aware.
Caveats
The abstract does not report whether results varied across repeated runs, nor does it give the parameter sizes or training compute for the models. In addition, the work is a preprint that has not yet undergone peer review.
From the paper
- Limitation
First, we evaluate only open-weight models, since the unknown architectures and parameter counts of proprietary models make scaling analysis unverifiable.
Quoted word for word from the full text on arXiv.
In this paper
inverse scaling in vision-language models
- Methods
- Qwen3-VL (evaluated)InternVL3.5 (evaluated)Gemma 3 (evaluated)Molmo 2 (evaluated)GPT-5.4Claude Opus 4.6
- Datasets
- MME (evaluated)AI2D (evaluated)ScienceQA (evaluated)BLINK (evaluated)MMT-Bench (evaluated)MMBench (evaluated)MMMU (evaluated)MathVista (evaluated)
Results
- Qwen3-VL, accuracy, on BLINK, Jigsaw: 84.0%
On Jigsaw, the 4B model achieves 84.0% accuracy compared to 30B-A3B’s 60.0% (+24.0pp, Net = 36).
5.3 Per-Family Analysis
- Gemma 3, accuracy, on ScienceQA, Particle motion and energy: 95.1%
ScienceQA contributes 10 cases, including the striking “Particle motion and energy” where 12B achieves 95.1% vs. 27B’s 51.2% (+43.9pp, Net = 18).
5.3 Per-Family Analysis
- Molmo 2, accuracy, on BLINK, Jigsaw: 79.3%
BLINK contributes 8 cases, with Jigsaw showing Net = 31 (4B: 79.3% vs. 8B: 58.7%).
5.3 Per-Family Analysis
- Molmo 2, accuracy, on BLINK: 63.5%; Molmo 2, accuracy, on MMBench: 88.3%
The 4B model outperforms 8B at the aggregate level on both BLINK (63.5% vs. 59.3%) and MMBench (88.3% vs. 87.9%).
5.3 Per-Family Analysis
- Gemma 3, accuracy gap, on ScienceQA, Basic Economic Principles: 39.4pp
The only non-BLINK entries are MME Landmark (InternVL 1B vs. 2B) and ScienceQA Basic Economic Principles, where Gemma 3 4B outperforms 27B by 39.4pp.
Strongest cases.
What it shows
- Vision-language models exhibit inverse scaling at the subtask level, with smaller models outperforming larger counterparts within the same family on specific question categories across multiple independently developed architectures.
While this often holds at the aggregate level, we present systematic evidence that vision-language models (VLMs) exhibit inverse scaling at the subtask level: specific question categories on which larger models perform worse than smaller counterparts within the same model family.
Abstract
- Subtask-level inverse scaling is task-driven rather than model-specific, replicating across independently developed model families and persisting even when restricting comparisons to dense-only architectures.
We identify 13 subtask-dataset pairs that show inverse scaling across multiple families, consistent with the phenomenon being task-driven rather than model-specific.
Abstract
- Chain-of-thought prompting attenuates subtask-level inverse scaling but leaves a quarter of identified cases intact and corrects fewer than half of individual errors.
CoT attenuates inverse scaling but does not remove it: the median Net falls from +8 to 0, yet 96 of 375 cases (25.6%) retain Net ≥ 5, 101 reverse, and the larger model recovers 43.5% of the questions it had lost under direct prompting.
Can chain-of-thought prompting recover the errors?
- At least 84% of inverse-scaling flip errors reflect real model failures rather than formatting issues or annotation defects.
We use the human rate as our conservative estimate: at least 84% of inverse scaling flips reflect real model failures.
5.6 Error Taxonomy
- The findings reflect regressions among released model variants rather than isolated causal effects of parameter count alone due to potential differences in vision encoders, training data, and alignment.
Fourth, models released under one family name may differ in vision encoder (Appendix K), training data, alignment, and active parameters, so our results describe subtask-level regressions among released variants rather than causal effects of parameter count alone; a controlled study training VLMs of several sizes under one fully released recipe would isolate parameter count and make such analyses reproducible, which we leave to future work …
Limitations.
Quoted word for word from the full text on arXiv.
The paper
On the Limits of Scale: Inverse Scaling in Vision-Language Models
arXiv, Oct 19, 2026, Open access, CC BY, Preprint, not peer-reviewed
doi.org/10.48550/arxiv.2610.27360
arXiv:2610.27360v1 in cs.CV, also cs.AI
Cite as BibTeX
@misc{saxena2026limits,
title = {On the Limits of Scale: Inverse Scaling in Vision-Language Models},
author = {Saxena, Rohit and Suglia, Alessandro and Rei, Marek and Minervini, Pasquale},
year = {2026},
eprint = {2610.27360},
archivePrefix = {arXiv},
primaryClass = {cs.CV},
doi = {10.48550/arxiv.2610.27360},
url = {https://arxiv.org/abs/2610.27360}
}