Attackers approached full-video scores without seeing frames
On 35 benchmarks, attackers that saw no video frames approached the accuracy achieved with full videos.
Edited by Boris Djordjevic · Paperfoot
The researchers built Video-Index, a curated benchmark of 840 video question-answer items from 76 sources, after auditing 115 video benchmarks for shortcuts. Their “attack pyramid” tested five levels of access to test items. Attackers that never saw a frame approached full-video accuracy on 35 benchmarks. On 51 benchmarks with tests of information about events over time, shuffled frames retained a median 96% of full-video accuracy.
The team screened 505,518 question-answer pairs from 112 benchmarks. They released the 210 hardest verified items under their attacks in each of four capability groups. On Video-Index, with the same fixed input, Claude Opus 5, an AI model, outscored every open-source model tested by over 37 percentage points. Agent tools added about 20 more percentage points.
Why it matters
The audit showed that high scores on some video benchmarks could come without seeing the video. Video-Index gives researchers a released test set screened for these shortcuts.
Caveats
The items were verified against the attacks tested, not every possible shortcut. The abstract does not name the open-source comparison models; the work is a preprint, not yet peer-reviewed.
In this paper
video understanding benchmark auditing and curation
- Methods
- attack pyramid (this paper)Claude Opus 5 (evaluated)
- Datasets
- Video-Index (this paper)
- Links
- Project pageCodeData
Results
- Claude Opus 5, score, on Video-Index, With the same fixed input: over 37 percentage points
With the same fixed input, Claude Opus 5 outscores every open-source model by over 37 percentage points, and agent tools add about 20 more, yet all systems leave room to improve efficiency and accuracy.
Abstract
What it shows
- On 51 video benchmarks with temporal probes, shuffled frames keep a median 96% of full-video accuracy.
On 51 benchmarks with temporal probes, shuffled frames keep a median 96% of full-video accuracy.
Abstract
- Near-duplicate questions account for at least half of the items across 63 video benchmarks.
Near-duplicate questions make up at least half the items in 63 benchmarks.
Abstract
- On 35 video benchmarks, attackers that never see any video frame approach full-video accuracy.
On 35 benchmarks, attackers that never see a frame approach full-video accuracy.
Abstract
- With the same fixed input, Claude Opus 5 outperforms every open-source model by more than 37 percentage points, with agent tools providing approximately 20 additional points.
With the same fixed input, Claude Opus 5 outscores every open-source model by over 37 percentage points, and agent tools add about 20 more, yet all systems leave room to improve efficiency and accuracy.
Abstract
Quoted word for word from the abstract on arXiv.
The paper
Video-Index: A Curated Meta-Benchmark for Video Understanding
arXiv, Oct 1, 2026, Open access, Preprint, not peer-reviewed
doi.org/10.48550/arxiv.2610.00960
arXiv:2610.00960v1 in cs.CV
Comments: Blog: https://www.enxinsong.com/blog/video-index/ GitHub: https://github.com/Espere-1119-Song/Video-Index Hugging Face: https://huggingface.co/datasets/Video-Index/Video-Index
Cite as BibTeX
@misc{song2026videoindex,
title = {Video-Index: A Curated Meta-Benchmark for Video Understanding},
author = {Song, Enxin and Xu, Yinuo and Yang, Shusheng and Chai, Wenhao and Gu, Jiatao},
year = {2026},
eprint = {2610.00960},
archivePrefix = {arXiv},
primaryClass = {cs.CV},
doi = {10.48550/arxiv.2610.00960},
url = {https://arxiv.org/abs/2610.00960}
}