The Agentic Era of Video Understanding

For complex tasks, agents have become the default. Nobody uses chat mode for that work anymore: people hand it to Claude Code, Codex or Gemini CLI, which open files, run commands and check their own results. We think video understanding should benefit from this too.

做复杂任务，agent 已经成了主流，没有人再用 chat mode 来完成复杂任务。大家把任务交给 Claude Code、Codex 或 Gemini CLI，让它们打开文件、运行命令、检查自己的结果。我们认为 video understanding 也应该从中受益。

So we are releasing Video-Index 1.1 with an agentic evaluation setup. The scores are on GitHub, and the questions, videos and agent traces on Hugging Face.

所以我们发布 Video-Index 1.1，加入 agentic 评测设置。分数在 GitHub 上，题目、视频和 agent 轨迹在 Hugging Face 上。

What Video-Index measuresVideo-Index 测什么

Video-Index is a set of 840 multiple-choice questions drawn from 76 existing video benchmarks. They fall into four groups of 210: perception, temporal, spatial and physical, and reasoning and knowledge. Before picking them, we audited 115 benchmarks with five levels of shortcut attacks. On 35 of those benchmarks, a model that never sees a frame comes close to its full-video accuracy. Video-Index keeps the questions that hold up best against these shortcuts.

Video-Index 是 840 道选择题，选自 76 个已有的视频 benchmark，感知、时序、空间与物理、推理与知识四组各 210 道。选题之前，我们用五级捷径攻击审计了 115 个视频 benchmark。其中 35 个，一帧都不看的模型就能接近看完整视频的准确率。Video-Index 留下的是最经得住这些捷径的题。

Every system answers the same questions about the same videos, in one of two setups. A frames-only model, marked non-agentic in Figure 1, gets at most 512 frames, sampled at 1 FPS with a 224-pixel short side, and answers in one pass. An agent gets the video as a file and works in its vendor's own harness: Codex for GPT-6-Astra, Claude Code for the Claude models and Gemini CLI for Gemini.

所有系统回答的是同一批视频上的同一批题，只是设置不同。只给抽帧的模型，即图 1 中的非 agentic，拿到最多 512 帧，按 1 FPS 采样、短边 224 像素，一次作答。agent 拿到的是视频文件，在各家自己的 harness 里运行：GPT-6-Astra 用 Codex，Claude 模型用 Claude Code，Gemini 用 Gemini CLI。

Gemini's agentic modeGemini 的 agentic 模式

Google has made a similar change to Gemini. In September it added an agentic mode to video understanding in the Gemini API, described in its launch post. In the default static mode, Gemini reads a video at a fixed 1 FPS and answers from the frames it was handed. In agentic mode, the model decides what to watch. It calls a tool to load the part of the video it needs, from the frames, the audio or the transcript. It can also rewatch a window at a higher frame rate.

Google 也给 Gemini 做了类似的改动。9 月，它给 Gemini API 的视频理解加了 agentic 模式，见发布文。默认的 static 模式按固定 1 FPS 读视频，用拿到的帧作答。agentic 模式下，模型自己决定看什么：它调用工具加载需要的那一段，可以是画面、音频或字幕，还能用更高的帧率重看某个窗口。

With the same model, the agentic mode uses far fewer tokens and scores higher.

同一个模型，agentic 模式用的 token 少得多，准确率更高。

What is new in Video-Index 1.1Video-Index 1.1 新增了什么

A fixed set of frames decides in advance what the model gets to see. If the answer falls between two sampled frames, or sits in a corner too small to read at 224 pixels, the model cannot go back and look. It cannot step through frames to count, cut out a region and zoom in, or write a few lines of code to measure something.

固定的一组帧事先就决定了模型能看到什么。答案如果落在两帧之间，或者在 224 像素下小到看不清的角落，模型没法回去再看。它不能一帧帧数数，不能裁出一块放大，也不能写几行代码去量一下。

The agentic setup. Each agent gets the video file and a shell, and runs in its vendor's harness.

agentic 评测设置。每个 agent 拿到视频文件和一个 shell，在各家自己的 harness 里运行。

Video research has worked this way for a while. VideoAgent let a language model decide which frames to caption next. A memory-augmented VideoAgent and DoraemonGPT query a structured memory of the video, VCA explores segments by tree search and VideoTree builds a tree of keyframes to walk down. Deep Video Discovery turns a long video into a database an agent can search. T* treats the task as finding a handful of relevant frames among tens of thousands. Newer models learn to call such tools on their own through reinforcement learning, as in Thinking With Videos, LongVT and Ego-R1. VideoLoop has the agent rewrite a bounded working memory as it explores a long video, so the evidence it found does not drown in what it has seen. Most video benchmarks still hand every model the same fixed frames.

视频方向的研究早就在这么做了。VideoAgent 让语言模型决定下一帧去 caption 哪一帧。带 memory 的 VideoAgent 和 DoraemonGPT 去查询视频的结构化 memory，VCA 用树搜索探索片段，VideoTree 建一棵关键帧树逐层往下找。Deep Video Discovery 把长视频变成 agent 可以搜索的数据库。T* 把问题看成从上万帧里找出少数几帧。更新的模型通过强化学习自己学会调用这些工具，比如 Thinking With Videos、LongVT 和 Ego-R1。VideoLoop 让 agent 在探索长视频时不断改写一块有界的工作记忆，找到的证据不会被看过的大量内容淹没。可大多数视频 benchmark 仍然给每个模型同一组固定的帧。

The gain is largest on questions that turn on time, such as causality, duration and counting, where an agent can replay a moment, measure it or step through it.

提升最大的是取决于时间的题，比如因果、时长和计数：agent 可以回放某一刻、量时长，或一帧帧地数。

Where agents still failagent 仍然会错在哪里

Gemini 4 Argon spends about a fifth of Claude Opus 5.5's time on a question, and the cases below show what such a short look misses.

Gemini 4 Argon 每道题花的时间大约只有 Claude Opus 5.5 的五分之一，下面这几道题能看出这样看得短会漏掉什么。

Takeaways核心结论

We release Video-Index 1.1 with agentic evaluation setup.

我们发布 Video-Index 1.1，加入 agentic 评测设置。

For benchmark authors: release the video files along with the questions, run agents on them, and report time and cost next to accuracy.

给 benchmark 作者：题目之外，把视频文件也放出来，用 agent 跑一遍，并把用时、花费与准确率一起报告。

For model teams: publish video scores both from frames and as an agent, and name the harness behind the agent score. Open video models need the tool use that would let them run as agents at all.

给模型团队：视频分数既公布抽帧的，也公布 agent 的，并写明 agent 分数用的是哪个 harness。开源 video model 首先需要能调用工具，才谈得上以 agent 方式运行。

Acknowledgements致谢

We thank Enxin Song for checking and running the new round of evaluations, and David Liu and Sihan Xu for helpful discussions and feedback on this post.

感谢 Enxin Song 检查并运行了新一轮测试，感谢 David Liu 和 Sihan Xu 对本文的有益讨论与反馈。

Citation

@misc{chai2026agenticvideo,
  title        = {The Agentic Era of Video Understanding},
  author       = {Chai, Wenhao},
  year         = {2026},
  howpublished = {Blog post},
  url          = {https://wenhaochai.com/blogs/agentic-video-evaluation.html}
}