Experiments›Frozen Critic
Oct 5, 2026

Training a Policy from Scratch against a Frozen Critic

By Wenhao Chai作者 Wenhao Chai

Setup设置

The critic is Limite, a 1.0B-parameter value model from Paradigma that estimates, from the prompt and the earlier tokens, whether a response will end correct; we keep it frozen. The actor is a dense model of the Qwen3.5 architecture with 0.46B parameters and random weights; it reads the question one character at a time and may write only 27 tokens: digits, arithmetic signs, the pieces of \boxed{}, a few other characters and the stop token. We train only the actor, with PPO on GSM8K: the reward is 1 when the last \boxed{} holds the answer, plus 0.1 when a finished response ends in a boxed number, and a response cut at 1,024 tokens takes the critic's value instead. We compare the GAE λ that turns the critic's values into per-token advantages, at 0.95, 0.5 and 0; at λ = 0 the actor learns from the critic alone. The metric is GSM8K test accuracy under the same strict grader, and the runs use this code.

critic 是 Paradigma 发布的 Limite,一个 1.0B 参数的 value 模型:它根据题目和此前的 token,估计一份回答最后会不会答对;我们把它冻结。actor 是 Qwen3.5 架构的稠密模型,4.6 亿参数,随机初始化;它按字符读题,只能写 27 个 token:数字、运算符号、\boxed{} 的几个片段、少数其他字符和结束 token。我们只训练 actor,在 GSM8K 上用 PPO:最后一个 \boxed{} 里是正确答案时 reward 为 1,结束的回答以 boxed 数字结尾时再加 0.1,在 1,024 个 token 处被截断的回答用 critic 的值代替评分。比较的变量是 GAE 的 λ,取 0.95、0.5 和 0,它决定 critic 的值怎样变成每个 token 的 advantage;λ = 0 时,actor 只从 critic 学习。指标是同一严格评分下的 GSM8K 测试集准确率,所有 run 使用这份代码。

Details详细说明

One view of pretraining holds that next-token prediction mostly teaches a model to judge partial solutions, and that the policy still has to be learned. We test the second half in its starkest form: the value function is given and frozen, and the policy starts from nothing. The critic is Limite, a 1.0B-parameter value model that Paradigma released on 21 September 2026. For every response token it estimates, from the prompt and the earlier tokens, whether the response will end correct. The actor is a dense model of the Qwen3.5 architecture with 0.46B parameters and random weights. Its vocabulary has 100 tokens: the 95 printable ASCII characters, newline, the word boxed, two chat markers and the stop token, each of them one token of the critic's vocabulary, so the critic scores every token the actor writes. The actor reads the question one character at a time but may write only 27 tokens: the ten digits, + − * / = and brackets, $, comma and period, the pieces of \boxed{}, space, newline and the stop token. A random actor that may write any of the critic's 151,667 tokens never samples a boxed number, so it never earns a reward; with 27 tokens a boxed number is within reach.

有一种看法认为,next-token prediction 主要教会模型判断一份写到一半的解答好不好,而策略本身仍然要学。我们用最极端的形式检验后半句:value 函数是现成的,并且冻结;策略从零开始。critic 是 Paradigma 在 2026 年 9 月 21 日发布的 Limite,一个 1.0B 参数的 value 模型。对回答里的每个 token,它根据题目和此前的 token 估计这份回答最后会不会答对。actor 是 Qwen3.5 架构的稠密模型,4.6 亿参数,随机初始化。它的词表有 100 个 token:95 个可打印 ASCII 字符、换行、单词 boxed、两个对话标记和结束 token。每个 token 都是 critic 词表里的一个 token,所以 critic 能给 actor 写的每个 token 打分。actor 按字符读题,但只能写其中 27 个 token:十个数字、+ − * / = 和括号、$、逗号和小数点、\boxed{} 的几个片段、空格、换行和结束 token。能写 critic 全部 151,667 个 token 的随机 actor 永远采样不到 boxed 数字,也就永远拿不到 reward;只有 27 个 token 时,boxed 数字可以采样到。

The actor is the only model we train, with PPO on GSM8K. The reward is 1 when the number in the last \boxed{} is the answer, plus 0.1 when a finished response's last box holds a single number; a response cut at 1,024 tokens takes the critic's value of the whole response in place of the grade. The compared variable is the GAE λ that turns the critic's values into per-token advantages: 0.95, 0.5 and 0. With γ = 1 and λ = 0, a token's advantage is the change in value it causes, so the actor learns from the critic alone; a larger λ gives the reward more weight. The metric is GSM8K test accuracy under the same strict grader. Each setting has one run, with seed 42.

actor 是唯一被训练的模型,在 GSM8K 上用 PPO 训练。最后一个 \boxed{} 里的数是正确答案时 reward 为 1;结束的回答最后一个 box 里是单个数字时,再加 0.1;在 1,024 个 token 处被截断的回答,用 critic 对整份回答的值代替评分。比较的变量是 GAE 的 λ,它决定 critic 的值怎样变成每个 token 的 advantage:取 0.95、0.5 和 0。γ = 1、λ = 0 时,一个 token 的 advantage 就是它带来的 value 变化,actor 只从 critic 学习;λ 越大,reward 占的比重越大。指标是同一严格评分下的 GSM8K 测试集准确率。每个设置只跑一次,种子 42。

Experiments实验

Answered已有结论Does the frozen critic rank GSM8K solutions that other models write?冻结的 critic 能给别的模型写的 GSM8K 解答排序吗?

Partly: it ranks every generator's solutions better than chance. Within a problem it beats response length on the two general Qwen models and only ties it on the math model and on R1-Distill.

部分可以:它对每个生成模型的解答排序都好于随机。在题内,它在两个通用 Qwen 模型上好于按回答长度排序,在数学模型和 R1-Distill 上只与长度持平。

Details详细说明

Partly. It ranks the solutions of every generator better than chance. Within a problem it beats response length on the two general Qwen models and only ties it on the math model and on R1-Distill.

部分可以。它对每个生成模型的解答排序都好于随机。在题内,它在两个通用 Qwen 模型上好于按回答长度排序,在数学模型和 R1-Distill 上只与长度持平。

Answered已有结论Can PPO train a randomly initialised actor that writes only arithmetic tokens against the frozen critic?只能写算术 token 的随机 actor,能用冻结的 critic 通过 PPO 训练出来吗?

No: at every λ the actor learns never to end its responses, and no evaluation solves a problem.

不能:每个 λ 下,actor 都学会永远不结束回答,没有一次评测解对一道题。

Details详细说明

Not in our runs (one per λ). At every λ the actor learns never to end its responses, and no evaluation solves a problem.

在我们的 run 里不能(每个 λ 一次)。每个 λ 下,actor 都学会永远不结束回答,没有一次评测解对一道题。

Answered已有结论Does removing the stop penalty make the actor finish its responses and box answers?去掉结束时的惩罚后,actor 会结束回答并写出 boxed 答案吗?

No: the actor now stops at once, and at every λ each training response soon becomes the stop token alone. Accuracy stays zero.

不会:actor 改为立刻结束,每个 λ 下,训练回答很快都只剩结束 token。准确率一直是零。

Details详细说明

No. It overshoots: the actor stops at once. At every λ, from step 5 to 9 on, every training response is the stop token alone, and accuracy stays zero.

不会。效果过头了:actor 立刻结束。在每个 λ 下,从第 5 到 9 步起,每个训练回答都只有结束 token,准确率一直是 0。

Answered已有结论Does the frozen critic know the answer?冻结的 critic 知道答案吗?

Only its size: the number it values most grows with the answer but is rarely the answer. It does only slightly better than always giving the most common training answer.

只知道大小:它给分最高的数字随答案变大而变大,但很少正好是答案。它只比总是回答训练集里最常见的答案好一点。

Details详细说明

Only its size. Among all numbers from 0 to 999, the critic's top-valued \boxed{k} is the answer on 4.5% of problems; always answering 4 is right on 1.5% of the same problems.

只知道大小。在 0 到 999 的全部数字里,critic 给分最高的 \boxed{k} 在 4.5% 的题上是正确答案;在同样这些题上,总是回答 4 有 1.5% 正确。

Answered已有结论Does a larger critic know the answer?更大的 critic 知道答案吗?

Not as our actor writes it: with the answer written one character per token, every Open-Reasoner-Zero critic picks it about as rarely as Limite. In its own format the largest critic picks it more often, but no critic passes our screen.

按我们 actor 的写法不知道:答案按字符写时,每个 Open-Reasoner-Zero critic 选中答案的比例都和 Limite 一样低。用它自己的格式,最大的 critic 选中得多一些,但没有一个 critic 通过筛查。

Details详细说明

Not as our actor writes it. With the answer written one character per token, no Open-Reasoner-Zero critic is right on more than 5.0% of problems. In its own answer format the 32B critic is right on 19.0%, and every critic values random text far above zero.

按我们 actor 的写法不知道。答案按字符写时,没有一个 Open-Reasoner-Zero critic 的正确率超过 5.0%。用它自己的答案格式,32B critic 在 19.0% 的题上正确;而且每个 critic 给随机文本的值都远高于 0。

Answered已有结论Is 19% low for a critic that sees no steps?对一个看不到解题步骤的 critic,19% 算低吗?

Yes: answering at once with no steps, Qwen3.5-4B already picks the answer more often than the critic, and Qwen3.5-27B far more often.

算低:不写步骤、直接作答时,Qwen3.5-4B 答对的就比这个 critic 多,Qwen3.5-27B 多得多。

Details详细说明

Yes. Answering at once, with no steps, Qwen3.5-4B is right on 23.5% of the same problems and Qwen3.5-27B on 48.5%.

算低。不写步骤、直接作答时,Qwen3.5-4B 在同样的题上对 23.5%,Qwen3.5-27B 对 48.5%。

Answered已有结论Can the actor beat its critic?actor 能超过它的 critic 吗?

Not so far: when stopping costs the critic's value, the actor never stops, and when a finished response also earns that value, it stops at once. Neither run solves a test problem.

目前没有:停止要扣掉 critic 的值时,actor 永不结束;结束的回答也得到这个值时,actor 立刻结束。两个 run 都没有解对一道测试题。

Details详细说明

Not in the two runs so far. When stopping costs the critic's value, the actor never stops; when a finished response also earns that value, it stops at once. Test accuracy is 0 in both.

目前两个 run 都没有。停止要扣掉 critic 的值时,actor 永不结束;结束的回答也得到这个值时,actor 立刻结束。两个 run 的测试准确率都是 0。

Takeaways核心结论

Acknowledgements致谢

The author is pleased to acknowledge that the work reported on in this post was substantially performed using the Princeton Research Computing resources at Princeton University. Princeton Research Computing is a consortium of groups including the Princeton Institute for Computational Science and Engineering (PICSciE) and Research Computing at Princeton University.

本文报告的工作主要使用普林斯顿大学 Princeton Research Computing 的计算资源完成。Princeton Research Computing 是由普林斯顿计算科学与工程研究所(PICSciE)和普林斯顿大学 Research Computing 等团队组成的联合体。

Experiment record实验记录

Citation

@misc{chai2026frozencritic,
  title        = {Training a Policy from Scratch against a Frozen Critic},
  author       = {Chai, Wenhao},
  year         = {2026},
  howpublished = {Blog post},
  url          = {https://wenhaochai.com/exps/frozen-critic.html}
}
Experiments Back to top回到顶部