Training Re-evaluation Curves

Setup设置

The runs follow the MuonH Qwen3 recipe of the Marin speedrun, which covers 130m to 1.2B parameters, with the 128K Marin tokenizer on fineweb-edu-10B text. We take two finished runs. The first is the recipe's 130m model with 12 layers instead of 6. Its learning rate stays at 0.02 for the first fifth of its 4959 steps and then falls linearly to zero, a warmup-stable-decay (WSD) schedule with no warmup. The second is the 300m model, which warms up to 0.01 over 1000 steps and then follows a cosine to zero at step 11444. Both runs read one shuffled stream of batches, each 128 sequences of 4096 tokens, so the 130m run's text is the first 43% of the 300m run's. After training, we score each final checkpoint on every batch it trained on, in order, and on 512 later batches of the same stream that neither run read. The final model's loss on the batch of step t, plotted against the training fraction t/T, is the training re-evaluation curve (TREC) of Bergsma, Dey and Hestness (2025). The scoring and the predictions use this code.

两个 run 都按 Marin speedrun 的 MuonH Qwen3 配方训练。这个配方覆盖 130m 到 1.2B 参数，用 128K 个词的 Marin tokenizer 和 fineweb-edu-10B 文本。我们取两个已经训练完的 run。第一个是配方中的 130m 模型，层数从 6 层改为 12 层。它的学习率在 4959 步的前五分之一保持 0.02，之后线性降到零，即不带预热的 warmup-stable-decay（WSD）调度。第二个是 300m 模型，学习率用 1000 步预热到 0.01，之后按 cosine 在第 11444 步降到零。两个 run 读同一个打乱后的批次序列，每批 128 条长 4096 个 token 的序列，所以 130m run 读的文本是 300m run 所读文本的前 43%。训练结束后，我们用每个 run 的最终 checkpoint，按训练顺序给它训练过的每一批打分，再给同一序列中两个 run 都没读到的 512 批打分。最终模型在第 t 步那一批上的损失，对训练进度 t/T 作图，就是 Bergsma、Dey 和 Hestness（2025）提出的训练再评估曲线（TREC）。打分和预测使用这份代码。

--- training (simplified)+++ re-evaluation (simplified; the Code link has the real code) for t in range(T):     batch = stream[t]-    loss[t] = loss_fn(theta, batch)-    theta = optimizer.step(theta, batch)+    # score batch t with the final weights; nothing is updated+    trec[t] = loss_fn(theta_final, batch)

--- 训练（简化）+++ 再评估（简化；真实代码见“代码”链接） for t in range(T):     batch = stream[t]-    loss[t] = loss_fn(theta, batch)-    theta = optimizer.step(theta, batch)+    # 用最终权重给第 t 批打分，不再更新参数+    trec[t] = loss_fn(theta_final, batch)

Experiments实验

Answered已有结论Which training batches does the final model fit best?最终模型对哪些训练批次拟合得最好？

Those read four fifths to nine tenths of the way through training, later under the linear decay than under the cosine. Neither model remembers the first quarter of its training: it scores those batches like text it never read.

训练进度五分之四到十分之九处读到的批次，线性衰减下比 cosine 下更靠后。两个模型都不记得训练前四分之一的批次：它们在这些批次上的损失和没读过的文本一样。

Answered已有结论Does the optimizer's averaging predict the curve?优化器的滑动平均能预测这条曲线吗？

No. With the exponent m that Bergsma et al. fit for AdamW, every reading of the optimizer misplaces the low point, by 0.04 to 0.15 of training. With m fitted to each run, every reading follows the measured shape, and the learning-rate-only control, in which nothing shrinks, does nearly as well, so the shape alone cannot tell the readings apart.

不能。用 Bergsma 等人为 AdamW 拟合的指数 m，优化器的每一种理解预测的最低点都偏离实测，偏差为训练进度的 0.04 到 0.15。如果对每个 run 单独拟合 m，每一种理解都能贴合实测形状；“只看学习率”的对照什么都不缩小，也几乎一样贴合。所以只看形状，分不出哪一种理解是对的。

Bergsma et al. predict the TREC from how much each update still weighs in the final weights. AdamW shrinks the weights by 1 − ηλ at every step, so the update of step i keeps the weight c_i = η_i ∏_j>i (1 − η_jλ). MuonH has no weight decay: after every step it rescales each weight matrix back to its starting norm. If successive steps were unrelated, an update of relative size η would shrink the old weights by 1/√(1 + η²), close to 1 − η²/2, a timescale of 20,000 steps at the 300m peak rate. But momentum makes successive steps point the same way. Each step then also pushes outward along the weights, and the rescaling shrinks the old weights by about 1 − Aη²/2, where A counts how many steps share a direction. Checkpoints of the 300m recipe give A = 73 at its momentum of 0.98, a timescale of about 275 steps; a simulation gives A = 23 at the 130m momentum of 0.95. The recipe's config also lists a weight decay of 0.1, which MuonH never applies; read as AdamW, it would mean 1000 steps.

Bergsma 等人根据每次更新在最终权重里还占多少份额来预测 TREC。AdamW 每一步把权重乘以 1 − ηλ，所以第 i 步的更新最终保留的权重是 c_i = η_i ∏_j>i (1 − η_jλ)。MuonH 没有权重衰减：它在每一步之后把每个权重矩阵缩放回初始范数。如果相邻两步互不相关，一次相对大小为 η 的更新会把旧权重乘以 1/√(1 + η²)，约等于 1 − η²/2；在 300m 的峰值学习率下，时间尺度是 20,000 步。但动量让相邻两步指向同一个方向。于是每一步还会沿着权重本身向外推，缩放回原范数时，旧权重约乘以 1 − Aη²/2，其中 A 表示有多少步共享同一个方向。从 300m 配方的检查点测得，在它 0.98 的动量下 A = 73，时间尺度约为 275 步；模拟给出 130m 的 0.95 动量下 A = 23。配方的配置里还写着 0.1 的权重衰减，但 MuonH 从不使用它；如果按 AdamW 理解，时间尺度会是 1000 步。

Answered已有结论Does a shorter timescale move the low point later?更短的时间尺度会把最低点推后吗？

Yes: the more steps a tokenizer takes over the same text, the later the batches the final model fits best, and the low point moves further than the averaging alone predicts.

是：同样的文本，tokenizer 切出的步数越多，最终模型拟合得最好的批次越靠后；而且最低点后移得比单靠滑动平均预测的更多。

Takeaways核心结论

Re-scored with their final weights, two MuonH runs fit best the batches they read four fifths to nine tenths of the way through training and have forgotten the first quarter. Bergsma et al.’s formula follows this shape only with a fitted exponent, and then a control without any averaging fits nearly as well; more steps over the same text move the low point later than the averaging predicts.

用最终权重重新给训练批次打分，两个 MuonH run 拟合得最好的都是训练进度五分之四到十分之九处读到的批次，训练前四分之一则已完全忘掉。Bergsma 等人的公式只有在拟合指数之后才贴合这个形状，而这时一个不做任何滑动平均的对照也几乎一样贴合；同样的文本切出更多的步，最低点后移得比滑动平均预测的更多。

Related work相关工作

• Predicting Training Re-evaluation Curves Enables Effective Data Curriculums for LLMs, Bergsma, Dey and Hestness 2025: defines the TREC, predicts its shape from AdamW's weight-decay averaging, and places high-quality data at its low point. We measure TRECs of MuonH runs, which have no weight decay, and test that prediction with the timescale of a fixed norm.
• How to set AdamW's weight decay as you scale model and dataset size, Wang and Aitchison 2024: reads AdamW's weights as an average of its updates over a timescale of 1/(ηλ) steps. Our fixed-norm reading carries that timescale over to MuonH.
• Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networks, Kosson, Messmer and Jaggi 2024: shows that weight decay drives each weight vector to a steady rotation per step, and that fixing the norm has a similar effect. MuonH is such a fixed-norm optimizer.
• Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-training, Bergsma et al. 2025: measures the AdamW timescale as a fraction of training and fits how its best value scales. Our runs sit far beyond the range it covers.
• 滑动平均视角下的权重衰减和学习率, Su 2025: derives from the same averaging view how long a model remembers its data, about 1/(ηλ) steps, and the schedules that weigh every batch equally. TRECs measure that memory directly.

• Predicting Training Re-evaluation Curves Enables Effective Data Curriculums for LLMs，Bergsma、Dey 和 Hestness 2025：提出 TREC，用 AdamW 权重衰减的滑动平均预测它的形状，并把高质量数据放在曲线的最低点。我们测量没有权重衰减的 MuonH run 的 TREC，并用固定范数的时间尺度检验这个预测。
• How to set AdamW's weight decay as you scale model and dataset size，Wang 和 Aitchison 2024：把 AdamW 的权重看作它的更新在 1/(ηλ) 步时间尺度上的平均。我们对固定范数的理解把这个时间尺度推广到 MuonH。
• Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networks，Kosson、Messmer 和 Jaggi 2024：证明权重衰减让每个权重向量收敛到每步固定的转角，并指出固定范数有类似效果。MuonH 正是一种固定范数的优化器。
• Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-training，Bergsma 等 2025：把 AdamW 的时间尺度表示为训练的比例，并拟合它的最优值如何随规模变化。我们的 run 远在它覆盖的范围之外。
• 滑动平均视角下的权重衰减和学习率，苏剑林 2025：从同样的滑动平均视角推导模型能记住多久的数据，约为 1/(ηλ) 步，并推导让每一批权重相同的调度。TREC 直接测量这种记忆。

Acknowledgements致谢

The author is pleased to acknowledge that the work reported on in this post was substantially performed using the Princeton Research Computing resources at Princeton University. Princeton Research Computing is a consortium of groups including the Princeton Institute for Computational Science and Engineering (PICSciE) and Research Computing at Princeton University.

本文报告的工作主要使用普林斯顿大学 Princeton Research Computing 的计算资源完成。Princeton Research Computing 是由普林斯顿计算科学与工程研究所（PICSciE）和普林斯顿大学 Research Computing 等团队组成的联合体。

Experiment record实验记录

Citation

@misc{chai2026trec,
  title        = {Training Re-evaluation Curves},
  author       = {Chai, Wenhao},
  year         = {2026},
  howpublished = {Blog post},
  url          = {https://wenhaochai.com/exps/trec.html}
}