Visual transition training improves vision and language models
Subsets of about 2,000 training examples collectively improved scores on 22 of 26 external tests by up to 27.3 percentage points.
Edited by Boris Djordjevic · Paperfoot
In a preprint, researchers developed WOVEN, a training source and benchmark of 36,076 video examples, and evaluated it across 38 frontier multimodal models—systems that process text and images. The benchmark organizes examples across 20 scene types, 5 action types, and 8 reasoning types to evaluate how models track visual changes. Even the strongest models, including GPT-5.4 and Qwen3-VL-235B-A22B, fell far below human performance, showing deficits that persisted across model families and scales.
The authors then trained multimodal models at multiple scales on WOVEN data. Small subsets of about 2,000 training items each collectively improved performance on 22 of 26 external benchmarks by up to 27.3 percentage points. Furthermore, WOVEN data replaced 30% to 50% of a task's own training data with comparable accuracy. These comparisons yielded a training recipe that selects data by reasoning operations rather than visual domains.
Why it matters
The findings show that visual transition reasoning can serve as a reusable building block to improve how vision models understand physical and temporal changes. Evidence from 26 external benchmarks indicates that developers can systematically train these capabilities using compact, targeted datasets rather than task-specific data alone.
Caveats
The training data relies on rollouts from generative video models rather than real-world physical interactions, and the abstract does not report score variation across runs or the compute used. The paper is also a preprint that has not yet undergone peer review.
In this paper
visual transition reasoning
- Methods
- Orca (baseline)GPT-5.4 (evaluated)Qwen3-VL-235B-A22B (evaluated)Llama-4-Maverick (evaluated)Qwen2.5-VL-3B (evaluated)Qwen2.5-VL-3B-Instruct (evaluated)V-JEPA 2 (evaluated)passive physical events (evaluated)four temporal subsets (evaluated)camera motion (evaluated)object manipulation (evaluated)Wan2.2-I2V-A14B
- Datasets
- WOVEN (this paper)SAT (evaluated)ActionEQA (evaluated)CLEVRER (evaluated)WorldPrediction (evaluated)MMSI-Bench (evaluated)VLM4D (evaluated)PhysBench (evaluated)SPAR-Bench (evaluated)MVBench (evaluated)
Results
- GPT-5.4, accuracy, in-distribution: 65.8%
The best model, GPT-5.4, reaches 65.8% in-distribution accuracy against 92.3% for humans (the mean of three annotators), and the next three fall between 57.8% and 63.2%, a gap that persists across model scales and cannot be closed by scaling alone (Appendix F.2).
3.1 Current MLLMs Show a Systematic Deficit
- Qwen2.5-VL-3B-Instruct, in-distribution accuracy, on WOVEN, in-distribution: 89.3%; Qwen2.5-VL-3B-Instruct, accuracy, on WOVEN, held-out scenes: 88.2%
As shown in Table 3, supervised fine-tuning of Qwen2.5-VL-3B-Instruct on the full training set raises in-distribution accuracy from 26.4% to 89.3%, with 88.2% on held-out scenes.
3.2 Visual Transition Reasoning Is Learnable
- passive physical events, accuracy, on WorldPrediction: -10.2; passive physical events, accuracy, on ActionEQA: -1.7
… b) Endogenous (agent-driven) transitions are not learned from exogenous supervision: the subset of passive physical events contains no agent action to learn from, and it lowers accuracy on the two benchmarks built on an agent’s action and its outcome, WorldPrediction ( -10.2 ) and ActionEQA ( -1.7; Table 6; Appendix G.13 …
5.3 Where Does Transfer Break?
- four temporal subsets, temporal-adjacency accuracy, temporal-adjacency: 55.6 – 58.7%
… a) Temporal localization is learned only from temporal supervision: the four temporal subsets raise temporal-adjacency accuracy from 25.7% to 55.6 – 58.7%, whereas the seven other subsets leave it near the baseline, and supervision on temporal adjacency and on temporal ordering do not transfer to each other (Table 3; Appendix G.9 …
5.3 Where Does Transfer Break?
- camera motion, robustness gains, among the causal subsets: +1.7; object manipulation, robustness gains, among the causal subsets: +23.2; passive physical events, gain: +29.9
Among the causal subsets, robustness gains increase from camera motion ( +1.7 ) and object inspection ( +3.0 ) to navigation ( +13.6 ) and object manipulation ( +23.2 ), and passive physical events give the largest gain of all subsets ( +29.9; Table 3; Appendix G.17).
5.2 What Makes the Learned Capability Robust?
What it shows
- Visual transition reasoning serves as a shared training primitive in MLLMs across diverse task domains, with a systematic training recipe derived from controlled comparisons of its downstream transfer.
To the best of our knowledge, we are the first to establish visual transition reasoning as a shared training primitive in MLLMs across diverse task domains and derive a systematic training recipe from controlled comparisons of its downstream transfer.
7 Conclusion
- Frontier MLLMs exhibit a substantial deficit in visual transition reasoning, with even the top-performing model achieving 65.8% accuracy compared to a 92.3% human baseline.
These deficits are substantial: even the strongest model achieves only 65.8% accuracy against a 92.3% human baseline.
1 Introduction
- Subsets teaching the same reasoning family share similar residual transfer profiles, whereas subsets sharing only an action type are no more similar than those sharing neither.
Surprisingly, results show that subsets teaching the same reasoning family have similar residual profiles, with a mean pairwise correlation of +0.32, whereas subsets sharing only an action type are no more similar than subsets sharing neither ( -0.22 versus -0.23 ), so the action shown in a transition does not predict where its supervision transfers …
5.1 What Determines Where Supervision Transfers?
- Substituting up to 30 to 50 percent of task-specific training items with WOVEN data yields accuracy comparable to training on task-specific items alone.
Results show that substituting up to 30 – 50% (depending on the benchmark) of the task-specific items leaves accuracy comparable to training on these items alone, and above a control that simply deletes the replaced items (Fig. 3; Appendix G.14).
4 Does Visual Transition Reasoning Serve as a Shared Primitive?
Quoted word for word from the full text on arXiv.
The paper
WOVEN: Weaving Visual World Modeling into Multimodal LLMs
arXiv, Oct 8, 2026, Open access, Preprint, not peer-reviewed
doi.org/10.48550/arxiv.2610.12417
arXiv:2610.12417v1 in cs.CV, also cs.CL, cs.LG
Cite as BibTeX
@misc{fan2026woven,
title = {WOVEN: Weaving Visual World Modeling into Multimodal LLMs},
author = {Fan, Zheyu and Zhang, Yue and Deng, Mingkai and Wang, Kangrui and Wang, Qineng and Chen, Canyu and Hao, Jie and Fan, Xing and Guo, Chenlei and Xing, Eric P. and Bansal, Mohit and Li, Manling},
year = {2026},
eprint = {2610.12417},
archivePrefix = {arXiv},
primaryClass = {cs.CV},
doi = {10.48550/arxiv.2610.12417},
url = {https://arxiv.org/abs/2610.12417}
}