Blog›The Agentic Era of Video Understanding
Oct 17, 2026

The Agentic Era of Video Understanding

We release Video-Index 1.1 with agentic evaluation setup.

我们发布 Video-Index 1.1,加入 agentic 评测设置。

By Wenhao Chai作者 Wenhao Chai
Gemini 4 Argon in Gemini CLI on two Video-Index 1.1 questions: where a man put his phone in a 20-minute first-person video, and how many times Melvin checks his locks in a two-hour film. It answers both right, while Claude Opus 5.5 ran out of its hour on both. The frames come from the two videos at the moments Argon looked, and the commands are condensed from the logged ones.Gemini 4 Argon 在 Gemini CLI 里做两道 Video-Index 1.1 的题:一段 20 分钟的第一人称视频里,戴眼镜的人把手机放在了哪里;一部两个多小时的电影里,Melvin 检查了几次门锁。两道它都答对了,Claude Opus 5.5 两道都用完了一小时的时限。画面取自两段视频里 Argon 看过的时刻,命令按日志里的命令压缩改写。

For complex tasks, agents have become the default. Nobody uses chat mode for that work anymore: people hand it to Claude Code, Codex or Gemini CLI, which open files, run commands and check their own results. We think video understanding should benefit from this too.

做复杂任务,agent 已经成了主流,没有人再用 chat mode 来完成复杂任务。大家把任务交给 Claude Code、Codex 或 Gemini CLI,让它们打开文件、运行命令、检查自己的结果。我们认为 video understanding 也应该从中受益。

So we are releasing Video-Index 1.1 with an agentic evaluation setup. The scores are on GitHub, and the questions, videos and agent traces on Hugging Face.

所以我们发布 Video-Index 1.1,加入 agentic 评测设置。分数在 GitHub 上,题目、视频和 agent 轨迹在 Hugging Face 上。

We also ran Gemini 4 Carbon as an agent on Humanity's Sixth Sense, a benchmark of what people see at a glance. It passes 82% of the questions, where the best single model call passes 55%.

我们还以 agent 方式在 Humanity's Sixth Sense 上跑了 Gemini 4 Carbon。这个 benchmark 考的是人一眼就能看出来的东西。Carbon 答对 82% 的题,而单次调用最好的模型只答对 55%。

What Video-Index measuresVideo-Index 测什么

Video-Index is a set of 840 multiple-choice questions drawn from 76 existing video benchmarks. They fall into four groups of 210: perception, temporal, spatial and physical, and reasoning and knowledge. Before picking them, we audited 115 benchmarks with five levels of shortcut attacks. On 35 of those benchmarks, a model that never sees a frame comes close to its full-video accuracy. Video-Index keeps the questions that hold up best against these shortcuts.

Video-Index 是 840 道选择题,选自 76 个已有的视频 benchmark,感知、时序、空间与物理、推理与知识四组各 210 道。选题之前,我们用五级捷径攻击审计了 115 个视频 benchmark。其中 35 个,一帧都不看的模型就能接近看完整视频的准确率。Video-Index 留下的是最经得住这些捷径的题。

Every system answers the same questions about the same videos, in one of two setups. A frames-only model, marked non-agentic in Figure 1, gets at most 512 frames, sampled at 1 FPS with a 224-pixel short side, and answers in one pass. An agent gets the video as a file and works in its vendor's own harness: Codex for GPT-6-Astra, Claude Code for the Claude models and Gemini CLI for Gemini.

所有系统回答的是同一批视频上的同一批题,只是设置不同。只给抽帧的模型,即图 1 中的非 agentic,拿到最多 512 帧,按 1 FPS 采样、短边 224 像素,一次作答。agent 拿到的是视频文件,在各家自己的 harness 里运行:GPT-6-Astra 用 Codex,Claude 模型用 Claude Code,Gemini 用 Gemini CLI。

Gemini's agentic modeGemini 的 agentic 模式

Google has made a similar change to Gemini. In September it added an agentic mode to video understanding in the Gemini API, described in its launch post. In the default static mode, Gemini reads a video at a fixed 1 FPS and answers from the frames it was handed. In agentic mode, the model decides what to watch. It calls a tool to load the part of the video it needs, from the frames, the audio or the transcript. It can also rewatch a window at a higher frame rate.

Google 也给 Gemini 做了类似的改动。9 月,它给 Gemini API 的视频理解加了 agentic 模式,见发布文。默认的 static 模式按固定 1 FPS 读视频,用拿到的帧作答。agentic 模式下,模型自己决定看什么:它调用工具加载需要的那一段,可以是画面、音频或字幕,还能用更高的帧率重看某个窗口。

With the same model, the agentic mode uses far fewer tokens and scores higher.

同一个模型,agentic 模式用的 token 少得多,准确率更高。

What is new in Video-Index 1.1Video-Index 1.1 新增了什么

A fixed set of frames decides in advance what the model gets to see. If the answer falls between two sampled frames, or sits in a corner too small to read at 224 pixels, the model cannot go back and look. It cannot step through frames to count, cut out a region and zoom in, or write a few lines of code to measure something.

固定的一组帧事先就决定了模型能看到什么。答案如果落在两帧之间,或者在 224 像素下小到看不清的角落,模型没法回去再看。它不能一帧帧数数,不能裁出一块放大,也不能写几行代码去量一下。

The agentic setup. Each agent gets the video file and a shell, and runs in its vendor's harness.

agentic 评测设置。每个 agent 拿到视频文件和一个 shell,在各家自己的 harness 里运行。

Video research has worked this way for a while. VideoAgent let a language model decide which frames to caption next. A memory-augmented VideoAgent and DoraemonGPT query a structured memory of the video, VCA explores segments by tree search and VideoTree builds a tree of keyframes to walk down. Deep Video Discovery turns a long video into a database an agent can search. T* treats the task as finding a handful of relevant frames among tens of thousands. Newer models learn to call such tools on their own through reinforcement learning, as in Thinking With Videos, LongVT and Ego-R1. VideoLoop has the agent rewrite a bounded working memory as it explores a long video, so the evidence it found does not drown in what it has seen. Most video benchmarks still hand every model the same fixed frames.

视频方向的研究早就在这么做了。VideoAgent 让语言模型决定下一帧去 caption 哪一帧。带 memory 的 VideoAgent 和 DoraemonGPT 去查询视频的结构化 memory,VCA 用树搜索探索片段,VideoTree 建一棵关键帧树逐层往下找。Deep Video Discovery 把长视频变成 agent 可以搜索的数据库。T* 把问题看成从上万帧里找出少数几帧。更新的模型通过强化学习自己学会调用这些工具,比如 Thinking With Videos、LongVT 和 Ego-R1。VideoLoop 让 agent 在探索长视频时不断改写一块有界的工作记忆,找到的证据不会被看过的大量内容淹没。可大多数视频 benchmark 仍然给每个模型同一组固定的帧。

The gain is largest on questions that turn on time, such as causality, duration and counting, where an agent can replay a moment, measure it or step through it.

提升最大的是取决于时间的题,比如因果、时长和计数:agent 可以回放某一刻、量时长,或一帧帧地数。

Where agents still failagent 仍然会错在哪里

Gemini 4 Argon spends about a fifth of Claude Opus 5.5's time on a question, and the cases below show what such a short look misses.

Gemini 4 Argon 每道题花的时间大约只有 Claude Opus 5.5 的五分之一,下面这几道题能看出这样看得短会漏掉什么。

Humanity's Sixth Sense as an agent以 agent 方式跑 Humanity's Sixth Sense

Humanity's Sixth Sense (HSS) asks about what people take in at a glance: whether a car fits a gap, who acts first, what happens next. It has 522 questions, 288 on images and 234 on videos, each with rubric criteria written by people. A judge model checks every criterion, and a question passes only if all of them are met. People pass 93.1%. The leaderboard calls each model once with the media, and its best, Gemini 4 Argon, passes 55.4%. The HSS paper also ran four agents on 388 of the questions; the best, GPT-6-Astra in Codex, passes 59.3%.

Humanity's Sixth Sense(HSS)问的是人一眼就能看出来的东西:车能不能开进空位,谁先动手,接下来会发生什么。它有 522 道题,288 道配图片,234 道配视频,每道题都有人写的评分细则。裁判模型逐条检查,所有细则都满足,这道题才算通过。人类的通过率是 93.1%。排行榜让每个模型看一次媒体、回答一次,最好的 Gemini 4 Argon 通过 55.4%。HSS 论文还在其中 388 道题上跑了四个 agent,最好的是 Codex 里的 GPT-6-Astra,通过 59.3%。

We ran Gemini 4 Carbon as an agent in Antigravity CLI on all 522 questions, in the paper's agent setup: the image or video is a file in the agent's workspace, and the agent has a shell with ffmpeg and Python but no internet. The judge is the paper's, Claude Opus 5 with its grading prompt, and each question gets one attempt. Carbon passes 82.2%, 27 points above the best single call.

我们在 Antigravity CLI 里以 agent 方式让 Gemini 4 Carbon 做了全部 522 道题,设置与论文的 agent 实验相同:图片或视频是 agent 工作目录里的一个文件,agent 有一个装了 ffmpeg 和 Python 的 shell,但不能上网。裁判用论文的 Claude Opus 5 和它的评分 prompt,每道题只做一次。Carbon 通过 82.2%,比单次调用最好的模型高 27 个百分点。

Antigravity CLI can also hand a video file to Gemini as native video input. In the run above we blocked that: a video opened this way reached the model as a note to extract frames with its tools. In a second run we allowed it and changed nothing else. Carbon then passes 81.2%, against 82.2% when it cut its own frames, a difference within noise (p = 0.59). It took the native video in 97 of the 234 video questions. On those 97 it passes 73.2% with native video and 80.4% when it had to cut frames (p = 0.09). As with Gemini's agentic mode above, an agent that picks its own frames does at least as well as one that takes the whole video in.

Antigravity CLI 也能把视频文件作为原生视频输入交给 Gemini。上面那次运行里我们屏蔽了这一路:用这种方式打开的视频,到模型那里只剩一句提示,让它用工具抽帧。第二次运行我们放开了它,别的都不变。Carbon 的通过率是 81.2%,自己抽帧时是 82.2%,差别在噪声范围内(p = 0.59)。在 234 道视频题里,它有 97 道读了原生视频。在这 97 道题上,读原生视频通过 73.2%,只能自己抽帧时通过 80.4%(p = 0.09)。和前面 Gemini 的 agentic 模式一样,自己挑帧的 agent 至少不比整段读入视频的差。

Claude Opus 5.5 in Claude Code ran 42 of the questions, sampled by domain and media type, before we stopped it. It passes 64.3% of them; Carbon passes 76.2% of the same 42 with frames only and 85.7% with native video. With 42 questions the 95% interval is about ±14 points. Opus spent a median 49 minutes on a question, Carbon about 15.

Claude Code 里的 Claude Opus 5.5 只做了 42 道题(按领域和媒体类型抽样),之后我们就停了它。它通过 64.3%;在同样这 42 道题上,Carbon 只能自己抽帧时通过 76.2%,读原生视频时通过 85.7%。只有 42 道题,95% 区间大约是 ±14 个百分点。Opus 每道题的中位用时是 49 分钟,Carbon 约 15 分钟。

We also checked whether Carbon's 93 failures come from the questions. Claude Opus 5.5 reviewed each with the image or video frames, the reference, the rubric and Carbon's answers, and we checked every item it flagged against the media. In 9 the question, its reference, its rubric or its video file is at fault, and the viewer below shows the 6 that come with an image or a short video. Counting these 9 as passes would raise Carbon from 82.2% to 83.9%.

我们还检查了 Carbon 答错的 93 道题是不是题目本身有问题。Claude Opus 5.5 逐题复查,看图片或视频帧、参考答案、评分细则和 Carbon 的回答;它挑出来的每一道题,我们都又对照媒体核实了一遍。其中 9 道是题目、参考答案、评分细则或视频文件有问题,下面展示其中配图片或短视频的 6 道。把这 9 道算作通过,Carbon 从 82.2% 升到 83.9%。

Takeaways核心结论

We release Video-Index 1.1 with agentic evaluation setup.

我们发布 Video-Index 1.1,加入 agentic 评测设置。

For benchmark authors: release the video files along with the questions, run agents on them, and report time and cost next to accuracy.

给 benchmark 作者:题目之外,把视频文件也放出来,用 agent 跑一遍,并把用时、花费与准确率一起报告。

For model teams: publish video scores both from frames and as an agent, and name the harness behind the agent score. Open video models need the tool use that would let them run as agents at all.

给模型团队:视频分数既公布抽帧的,也公布 agent 的,并写明 agent 分数用的是哪个 harness。开源 video model 首先需要能调用工具,才谈得上以 agent 方式运行。

For harness builders: give the agent the video as a file and the tools to cut its own frames. On Humanity's Sixth Sense, native video input did not raise the score.

给 harness 开发者:把视频作为文件交给 agent,并给它自己抽帧的工具。在 Humanity's Sixth Sense 上,原生视频输入没有提高分数。

Acknowledgements致谢

We thank the Video-Index team: Enxin Song, Yinuo Xu, Shusheng Yang and Jiatao Gu. Enxin Song also checked and ran the new round of evaluations. We thank David Liu and Sihan Xu for helpful discussions and feedback on this post.

感谢 Video-Index 团队:Enxin Song、Yinuo Xu、Shusheng Yang 和 Jiatao Gu。Enxin Song 还检查并运行了新一轮测试。感谢 David Liu 和 Sihan Xu 对本文的有益讨论与反馈。

Citation

@misc{chai2026agenticvideo,
  title        = {The Agentic Era of Video Understanding},
  author       = {Chai, Wenhao},
  year         = {2026},
  howpublished = {Blog post},
  url          = {https://wenhaochai.com/blogs/agentic-video-evaluation.html}
}
All posts全部文章 Back to top回到顶部