Training a Policy from Scratch against a Frozen Critic

Setup设置

One view of pretraining holds that next-token prediction mostly teaches a model to judge partial solutions, and that the policy still has to be learned. We test the second half in its starkest form: the value function is given and frozen, and the policy starts from nothing. The critic is Limite, a 1.0B-parameter value model that Paradigma released on 21 September 2026. For every response token it estimates, from the prompt and the earlier tokens, whether the response will end correct. The actor is a dense model of the Qwen3.5 architecture with 0.46B parameters and random weights. Its vocabulary has 100 tokens: the 95 printable ASCII characters, newline, the word boxed, two chat markers and the stop token, each of them one token of the critic's vocabulary, so the critic scores every token the actor writes. The actor reads the question one character at a time but may write only 27 tokens: the ten digits, + − * / = and brackets, $, comma and period, the pieces of \boxed{}, space, newline and the stop token. A random actor that may write any of the critic's 151,667 tokens never samples a boxed number, so it never earns a reward; with 27 tokens a boxed number is within reach.

有一种看法认为，next-token prediction 主要教会模型判断一份写到一半的解答好不好，而策略本身仍然要学。我们用最极端的形式检验后半句：value 函数是现成的，并且冻结；策略从零开始。critic 是 Paradigma 在 2026 年 9 月 21 日发布的 Limite，一个 1.0B 参数的 value 模型。对回答里的每个 token，它根据题目和此前的 token 估计这份回答最后会不会答对。actor 是 Qwen3.5 架构的稠密模型，4.6 亿参数，随机初始化。它的词表有 100 个 token：95 个可打印 ASCII 字符、换行、单词 boxed、两个对话标记和结束 token。每个 token 都是 critic 词表里的一个 token，所以 critic 能给 actor 写的每个 token 打分。actor 按字符读题，但只能写其中 27 个 token：十个数字、+ − * / = 和括号、$、逗号和小数点、\boxed{} 的几个片段、空格、换行和结束 token。能写 critic 全部 151,667 个 token 的随机 actor 永远采样不到 boxed 数字，也就永远拿不到 reward；只有 27 个 token 时，boxed 数字可以采样到。

The actor is the only model we train, with PPO on GSM8K. The reward is 1 when the number in the last \boxed{} is the answer, plus 0.1 when a finished response's last box holds a single number; a response cut at 1,024 tokens takes the critic's value of the whole response in place of the grade. The compared variable is the GAE λ that turns the critic's values into per-token advantages: 0.95, 0.5 and 0. With γ = 1 and λ = 0, a token's advantage is the change in value it causes, so the actor learns from the critic alone; a larger λ gives the reward more weight. The metric is GSM8K test accuracy under the same strict grader. Each setting has one run, with seed 42.

actor 是唯一被训练的模型，在 GSM8K 上用 PPO 训练。最后一个 \boxed{} 里的数是正确答案时 reward 为 1；结束的回答最后一个 box 里是单个数字时，再加 0.1；在 1,024 个 token 处被截断的回答，用 critic 对整份回答的值代替评分。比较的变量是 GAE 的 λ，它决定 critic 的值怎样变成每个 token 的 advantage：取 0.95、0.5 和 0。γ = 1、λ = 0 时，一个 token 的 advantage 就是它带来的 value 变化，actor 只从 critic 学习；λ 越大，reward 占的比重越大。指标是同一严格评分下的 GSM8K 测试集准确率。每个设置只跑一次，种子 42。

Experiments实验

Answered已有结论Does the frozen critic rank GSM8K solutions that other models write?冻结的 critic 能给别的模型写的 GSM8K 解答排序吗？

Partly. It ranks the solutions of every generator better than chance. Within a problem it beats response length on the two general Qwen models and only ties it on the math model and on R1-Distill.

部分可以。它对每个生成模型的解答排序都好于随机。在题内，它在两个通用 Qwen 模型上好于按回答长度排序，在数学模型和 R1-Distill 上只与长度持平。

Answered已有结论Can PPO train a randomly initialised actor that writes only arithmetic tokens against the frozen critic?只能写算术 token 的随机 actor，能用冻结的 critic 通过 PPO 训练出来吗？

Not in our runs (one per λ). At every λ the actor learns never to end its responses, and no evaluation solves a problem.

在我们的 run 里不能（每个 λ 一次）。每个 λ 下，actor 都学会永远不结束回答，没有一次评测解对一道题。

Answered已有结论Does removing the stop penalty make the actor finish its responses and box answers?去掉结束时的惩罚后，actor 会结束回答并写出 boxed 答案吗？

No. It overshoots: the actor stops at once. At every λ, from step 5 to 9 on, every training response is the stop token alone, and accuracy stays zero.

不会。效果过头了：actor 立刻结束。在每个 λ 下，从第 5 到 9 步起，每个训练回答都只有结束 token，准确率一直是 0。

Related work相关工作

• RFPO, Li et al. 2026: a single calibrated, frozen critic serves as the rollout reward, the GAE baseline and a forecaster for unfinished prefixes in LLM post-training. Its critic and actor come from one PPO run and the actor starts pretrained; here the critic comes from another run and the actor starts from random weights.
• DVPO, Huang et al. 2025: a global value model is pretrained offline on preference data and then frozen as the critic for policy learning. It is the same frozen-critic design for RLHF, with a pretrained actor.
• PURE, Cheng et al. 2025: a frozen process reward model used as the RL reward is exploited and training collapses under summed credit, which min-form credit assignment prevents. An actor trained against a frozen critic can fail the same way.
• Motif, Klissarov et al. 2023: an intrinsic reward built from an LLM's preferences over captions trains a NetHack agent from scratch. A frozen language model guides a policy learned from nothing, in a game instead of in text.

• RFPO，Li 等 2026：在大模型后训练中，一个校准过的冻结 critic 同时充当 rollout 的奖励、GAE 的基线和未写完前缀的成功预测。它的 critic 和 actor 来自同一次 PPO 训练，actor 从预训练模型开始；本文的 critic 来自另一次训练，actor 从随机权重开始。
• DVPO，Huang 等 2025：先用偏好数据离线预训练一个全局 value 模型，再冻结它，作为策略学习的 critic。这是 RLHF 里同样的冻结 critic 设计，actor 是预训练模型。
• PURE，Cheng 等 2025：把冻结的过程奖励模型直接当强化学习奖励时，求和式的功劳分配会被钻空子，训练崩溃；取最小值的功劳分配可以避免。用冻结 critic 训练的 actor 也可能这样失败。
• Motif，Klissarov 等 2023：用大模型对描述文字的偏好构造内在奖励，从零训练 NetHack 智能体。冻结的语言模型指导一个从零学起的策略，场景是游戏而不是文本。

Acknowledgements致谢

The author is pleased to acknowledge that the work reported on in this post was substantially performed using the Princeton Research Computing resources at Princeton University. Princeton Research Computing is a consortium of groups including the Princeton Institute for Computational Science and Engineering (PICSciE) and Research Computing at Princeton University.

本文报告的工作主要使用普林斯顿大学 Princeton Research Computing 的计算资源完成。Princeton Research Computing 是由普林斯顿计算科学与工程研究所（PICSciE）和普林斯顿大学 Research Computing 等团队组成的联合体。

Experiment record实验记录

Citation

@misc{chai2026frozencritic,
  title        = {Training a Policy from Scratch against a Frozen Critic},
  author       = {Chai, Wenhao},
  year         = {2026},
  howpublished = {Blog post},
  url          = {https://wenhaochai.com/blogs/frozen-critic.html}
}