Larger Tokenizers Suffer More Multi-Epoch Regret
On a fixed amount of text trained for 8 epochs, larger tokenizers suffer more multi-epoch regret, and our two best setups each remove about half of the 128K regret.
Wenhao Chai
Wenhao Chai is a second-year Ph.D. student in Computer Science at Princeton University, advised by Professor Karthik Narasimhan. He received his master's degree from the University of Washington and his bachelor's from Zhejiang University. He was a student researcher at Google DeepMind. He has interned at Pika Labs, where he worked with Professor Christopher D. Manning, and at Microsoft Research Asia.
He leads MovieChat, one of the first large multimodal models and benchmarks for hour-long video understanding with a memory mechanism. He co-leads LiveCodeBench Pro, a benchmark that Google and Meta use to evaluate Gemini and Muse Spark. He has organized workshops and competitions at CVPR 2024, CVPR 2025, and CVPR 2026. His work has been featured in MIT Technology Review. Sequoia China and MIT Technology Review China named him to AI25, their list of AI innovators under 25.
He is currently working on Recursive Self-Improvement (RSI).
Life
All photos
Blog
The Agentic Era of Video Understanding
We release Video-Index 1.1 with agentic evaluation setup.
Predictable Swarm Scaling
Under suitable conditions, we can use simulation to roughly estimate the best structure for an agent swarm and its scaling law.
What Happened to AutomationBench?
Open models score well on AutomationBench; the scoring rule, unsound string-matching guardrails included, decides much of the ranking, and eighteen of its string guardrails fail runs a task owner would accept.
ODE Perspective on Neural Networks
We cover the neural ODE perspective in terms of optimization and architecture design.
View Transformer Layers from Online Optimization Perspective
We cover mesa-optimization, test-time-training (TTT), and a broad view of fast weight programming in transformer models.
What is the Intrinsic Dimension of Your Data?
We introduce the concept of intrinsic dimension and provide a method to estimate it. It is amazing that ImageNet has only 50 of the intrinsic dimension.
Bridging the Parallel Decoding of LLMs with the Diffusion Process
We introduce Jacobi Decoding, a parallel decoding algorithm for LLMs, and its connection to the diffusion process in terms of high-level concepts.
News
All news- Sep 2026Three papers accepted by NeurIPS 2026, one as a Spotlight.
- Sep 2026One paper accepted by the NeurIPS 2026 Evaluations and Datasets Track as an Oral.
- Aug 2026One paper accepted by EMNLP 2026.
- Jul 2026One paper accepted by COLM 2026.
- Jul 2026Named to AI25, the list of 25 AI innovators under 25 from Sequoia China and MIT Technology Review China.
- Jun 2026Received a Laude Institute Slingshots grant in its third cohort.
- Jun 2026One paper accepted by IJCV.
- Jun 2026Two papers accepted by ECCV 2026.
- Jun 2026Organized the 2nd Workshop on Knowledge-Intensive Multimodal Reasoning at CVPR 2026.
- May 2026Joined Google DeepMind as a student researcher.
- May 2026Berkeley EECS News covered our FrontierCS blog post on CALICO.
- May 2026Two papers accepted by ICML 2026.
- Feb 2026Began hosting interviews for Z Potentials.
- Feb 2026Two papers accepted by CVPR 2026.
- Jan 2026Five papers accepted by ICLR 2026.
- Dec 2025One paper accepted by IEEE TIP.
- Dec 2025Gave an oral presentation at NeurIPS 2025 on RISEBench, our benchmark for reasoning-informed visual editing. Slides.
- Oct 2025Video-MMLU received the Outstanding Paper Award and a travel grant at the ICCV 2025 Workshop on Knowledge-Intensive Multimodal Reasoning.
- Sep 2025One paper accepted by NeurIPS 2025.
- Sep 2025Two papers accepted by the NeurIPS 2025 Datasets and Benchmarks Track, one as an Oral.
- Sep 2025Received a $5,000 Lambda Research Grant.
- Sep 2025Gave an invited talk, “Better and Longer Video Understanding,” at Abaka AI and 2077AI. Slides.
- Sep 2025Joined Princeton University as a CS Ph.D. student.
- Aug 2025One paper accepted by IEEE TPAMI.
- Aug 2025LiveCodeBench Pro presented at the Open AGI Symposium at UC Berkeley. Slides.
- Jul 2025Interviewed by DeepTech and MIT Technology Review China. Post.
- Jun 2025One paper accepted by ICCV 2025.
- Jun 2025Featured in MIT Technology Review as one of the lead authors of LiveCodeBench Pro.
- Jun 2025One paper accepted by IROS 2025 as an Oral.
- May 2025One paper accepted by ACL 2025.
- May 2025Master's thesis nominated for the Distinguished Thesis Award by the ECE Department of the University of Washington.
- Apr 2025Hosted the Lambda-sponsored CVPR 2025 Video Understanding Challenge at the LOVEU workshop.
- Mar 2025One paper accepted by IEEE TVCG.
- Mar 2025Graduated from the University of Washington with a master's thesis on Large Multimodal Models for Video Captioning.
- Feb 2025Three papers accepted by CVPR 2025.
- Jan 2025Two papers accepted by ICLR 2025.
- Jul 2024Two papers accepted by ECCV 2024.
- Jun 2024Joined Pika Labs as a research intern working on video understanding and generation models.
- Apr 2024Hosted the CVPR 2024 Long-form Video Understanding Challenge at the LOVEU workshop.
- Feb 2024Two papers accepted by CVPR 2024, one of them among the 2.8% of submissions selected as Highlights.
- Feb 2024Gave an invited talk at the AAAI 2024 Imageomics workshop.
- Dec 2023One paper accepted by AAAI 2024.
- Jul 2023Two papers accepted by ICCV 2023.
- Feb 2023Joined Microsoft Research Asia as a research intern.
Experiments
All experiments
Generating Distributions
Training Re-evaluation Curves
Adaptive Swarm
How Vocabulary Size Shapes Overfitting
Training a Policy from Scratch against a Frozen Critic
How to design reward function in Multi-Agent RL
Quantifying the Per-layer Contribution of LLMs
Sampled Softmax with Over-Tokenized Transformer
LoopDiT: Loop Transformers for Diffusion Models
Experiment Template
Office hours
FAQ for juniorsUndergraduate and master's students: if you want to talk about research ideas, career plans, or life in AI/ML, book a slot. I set aside at least 30 minutes every week for these meetings, and I encourage students from underrepresented groups to reach out. Open times are on my calendar.