How to design reward function in Multi-Agent RL多智能体 RL 中如何设计奖励函数

Setup设置

The baseline is single-agent EasyPPO with its Frontier-CS settings: its Qwen3.5-9B SFT initialization, trained on 200 FrontierSmith problems and validated on 170 of the 172 Frontier-CS algorithmic problems (two prompts exceed the length limit), scored 0–100 as the mean per-test-case score by a local judge that follows the official rules; the last C++ block of an answer is the program judged, and every prompt says so. Ten of the 200 FrontierSmith problems had checkers that did not measure the algorithm: one gave full marks for any valid output, seven gave full marks at the statement's baseline, and two scored every correct program 0. FrontierSmith fixed them on October 17; the runs that ended before then trained on the original problems, and the agentic run trains on the fixed ones. A reply cut off at its token limit, or one that never closes its reasoning, scores 0 without being judged; here we differ from FrontierSmith, whose reward judges the code it finds in such a reply. This rule came in on October 7. Runs that were training then switched to it at their next 24-hour segment, except three that never trained under it: the feedback team with the difference reward and the two single agents with feedback. The first team study (a lead with three teammates) kept FrontierSmith's rule in all its arms; the experiment record says which. We replace the single agent with a team, one lead and four subagents as in DeepSeek-V4.1-Flash's Agent Team mode: the subagents are helpers that may test programs against the judge and report ideas, analysis or code, and only the lead's final submission scores the team. We compare ways to split the team's reward; compute is counted in generated tokens and latency in tokens on the critical path. The first four questions below were asked on competition math with FlashREINFORCE: the first two set up the trainer, and the next two asked the same thing there. Code: the team game, judge and scripts, the miles fork and the math runs; the experiment record links the exact commit of each finished run.

基线是单智能体 EasyPPO，用它的 Frontier-CS 设置：它的 Qwen3.5-9B SFT 初始化，在 200 道 FrontierSmith 题上训练，在 172 道 Frontier-CS 算法题中的 170 道上验证（另 2 道的 prompt 超过长度上限），由按官方规则实现的本地评测器打 0–100 分，即各测试用例得分的平均；评测的是回答里最后一个 C++ 代码块，每个 prompt 都写明了这一点。200 道 FrontierSmith 题里有 10 道的评测器不衡量算法：1 道对任何合法输出都给满分，7 道达到题面的基线就给满分，2 道给每个正确程序都打 0 分。FrontierSmith 在 10 月 17 日修好了这些题；在此之前结束的 run 用的是原题，agentic run 用修好的题训练。被 token 上限截断的回答，或推理没有结束的回答，直接记 0 分，不送评测器；这一点与 FrontierSmith 不同，它的奖励会评测这种回答里找到的代码。这条规则从 10 月 7 日起生效：当时正在训练的 run 从下一个 24 小时段起改用它，只有 3 个 run 从未在这条规则下训练，即差分奖励的带反馈团队和两个带反馈的单智能体；第一个团队实验（1 个 lead 加 3 个 teammate）的各组都保留了 FrontierSmith 的规则；实验记录写明了每个 run 用的规则。我们把单智能体换成团队，按 DeepSeek-V4.1-Flash 的 Agent Team 模式设为 1 个 lead 加 4 个 subagent：subagent 是帮手，可以把程序交给评测器自测，再汇报思路、分析或代码；只有 lead 的最终提交决定团队成绩。我们比较团队奖励的不同分法；算力按生成的 token 数计，延迟按关键路径上的 token 数计。下面前 4 个问题是在竞赛数学上用 FlashREINFORCE 做的：前 2 个搭好训练器，后 2 个问的是同一件事。代码：团队游戏、评测器和脚本、miles 分支和数学实验；实验记录里链接了每个跑完的 run 的准确提交。

--- baseline: one agent, one judged program (simplified)+++ ours: a team game with judge feedback (simplified) def episode(problem):-    return judge(problem, agent.solve(problem)).score+    # the lead writes one task per subagent# lead 给每个 subagent 写一条任务+    tasks = lead.plan(problem, n=4)+    # each subagent may test up to 3 programs, sees each result, then reports# 每个 subagent 最多自测 3 个程序，看到每次结果后写报告+    reports = [sub.explore(problem, t, judge, tests=3) for t in tasks]+    # the lead reads every report and submits the team's program# lead 读完所有报告，提交团队的程序+    final = judge(problem, lead.solve(problem, reports))+    # only the final submission scores the team; the split is what we compare# 只有最终提交决定团队成绩；比较的是奖励怎么分+    return split(final, reports)

Experiments实验

Answered已有结论Does FlashREINFORCE's quick-start recipe reproduce the paper on R1-Distill-1.5B?FlashREINFORCE 的快速入门配方能否在 R1-Distill-1.5B 上复现论文？

Yes: our run stays within about 2 points of the paper's arms at every reported step, and it ends about 2 points below the trust-region arm; each arm has one seed.

能复现：在论文给出的每个步数上，我们的 run 与论文两组相差都在约 2 分以内，结束时比信任域组低约 2 分；每组只有一个 seed。

Answered已有结论Does score centering remove the train/inference mismatch, and does it buy held-out accuracy?Score centering 能否消除训练/推理不一致？它能提高保留集准确率吗？

It removes the mismatch, and held-out accuracy stays the same, so the mismatch does not limit held-out accuracy.

能消除不一致，保留集准确率保持不变，所以不一致并不限制保留集准确率。

Answered已有结论Does talking help a team of agents on math, and which credit makes it help?在数学题上，智能体之间交流有没有帮助？哪种功劳分配让它有帮助？

Untrained talking hurts, because agents copy each other. Paying each turn for its own gain gave the highest held-out accuracy, in a contrast that had no registered bar and two seeds.

未经训练的交流有害，因为智能体会互相照抄。按每一轮自己的增益给奖励时保留集准确率最高；这个对比没有事先登记门槛，只有两个 seed。

Answered已有结论Does DeepSeek-style lead-and-teammates training beat one agent with the same turn and token limits?DeepSeek 式的 lead 加队友训练，能否胜过同预算的单智能体？

No: the collaboration bonus makes the lead delegate more often, but the team does not beat the same lead working alone.

不能：协作奖励让 lead 更常委派，但团队没有胜过同一个 lead 单干。

Answered已有结论Can a local judge stand in for the official Frontier-CS judge?本地评测器能否代替官方的 Frontier-CS 评测器？

Yes: it scores like the official judge on classic and interactive problems, and contestant code cannot read the test answers. Inside training, every judge after the first scored interactive answers 0, so the validation scores of those runs count interactive answers as 0.

能：它在普通题和交互题上的打分都与官方评测器一致，而且选手代码读不到测试答案。在训练中，第一版之后的各版评测器给交互题的回答全部打 0 分，所以这些 run 的验证集得分把交互题回答按 0 分计入。

Answered已有结论Which starting point can RL train from on Frontier-CS?在 Frontier-CS 上，RL 能从哪个起点开始训练？

EasyPPO's SFT initialization: it finishes most of its answers, while the base model with reasoning mostly runs out of tokens.

EasyPPO 的 SFT 初始化：它大部分回答都能写完，而原版模型开启思考时大多用完了 token。

Testing检验中Does our EasyPPO port train a single agent on Frontier-CS?我们移植的 EasyPPO 能否在 Frontier-CS 上训练单智能体？

Testing检验中Does a team without feedback beat one agent?没有反馈的团队能否胜过单智能体？

Testing检验中With judge feedback, which reward split makes a team beat one agent at equal tokens?有评测反馈时，哪种奖励分法能让团队在相同 token 下胜过单智能体？

Answered已有结论Why does each setup with feedback score as it does?带反馈的各做法，得分为什么高、为什么低？

Each setup is held back by its submission: except under the difference reward, every setup judged a better program before it submitted. The bonus wins because it pays subagents to test, which also teaches the one shared model to write programs that compile; the difference reward loses because a report whose tests all fail makes the lead's final worse, so the reward teaches subagents to stop testing. One seed.

各做法都受限于最后的提交：除差分奖励外，每个做法在提交之前都已评测过更好的程序。加分奖励赢，是因为它奖励 subagent 自测，共用的一个模型也因此学会写能编译的程序；差分奖励输，是因为自测全部失败的报告会拖低 lead 的最终程序，这个奖励便教 subagent 不再自测。只有一个 seed。

Testing检验中Does one agent with a shell (mini-swe-agent) score higher than one answer?给单个智能体一个 shell（mini-swe-agent），得分能否超过一次作答？

Testing检验中Does a swarm of four mini-swe-agents that coordinate only through one shared repo beat one agent with four times the steps?4 个 mini-swe-agent 组成的 swarm 只通过一个共享 repo 协作，能否胜过用 4 倍步数的单个智能体？

Testing检验中Why do most programs fail to compile, and does a prompt, the base weights or a lower temperature fix it?为什么大部分程序编译不过？换 prompt、换权重或降低温度能否解决？

Answered已有结论Does the trainer score tokens the way the rollout engine sampled them?训练器给 token 的打分，与 rollout 引擎采样时的打分一致吗？

Up to bf16 rounding: the two differ by as much as either side differs from itself when only the order of a sum changes, and the gap stays flat over training, so we apply no importance correction.

只差 bf16 的舍入：两者之间的差距，和任一侧只改变求和顺序时与自己的差距一样大，而且训练全程不变，所以我们不做重要性校正。

Takeaways核心结论

Related work相关工作

• EasyPPO (arXiv 2609.36802): PPO with a critic and three small changes, with Frontier-CS results. It is our baseline and initialization, extended here from one agent to a team.
• DeepSeek-V4.1-Flash (arXiv 2609.19969), Sec. 5.3.5: an Agent Team mode trained with task reward, a collaboration bonus and a critical-path latency penalty. We replicate its training on math and adopt its team structure on Frontier-CS.
• FlashREINFORCE: asynchronous REINFORCE with a sequence trust region. We reproduce its R1-Distill-1.5B curve and train the math teams with it.
• Score centering (arXiv 2609.20807): a score-function correction for off-policy sampling. We test whether it buys held-out accuracy inside FlashREINFORCE.
• Extending SWE-2's reward: a cost-aware reward whose dual update steers where training lands on the score/cost frontier. For the same reason we report teams at equal tokens and at equal critical-path tokens.

• EasyPPO（arXiv 2609.36802）：带 critic 的 PPO 加三个小改动，含 Frontier-CS 结果。它是我们的基线和初始化，本文把它从单智能体扩展到团队。
• DeepSeek-V4.1-Flash（arXiv 2609.19969）第 5.3.5 节：Agent Team 模式，用任务分、协作奖励和关键路径延迟惩罚训练。我们在数学上复刻它的训练，在 Frontier-CS 上采用它的团队结构。
• FlashREINFORCE：带序列信任域的异步 REINFORCE。我们复现了它在 R1-Distill-1.5B 上的曲线，并用它训练数学上的团队。
• Score centering（arXiv 2609.20807）：针对离策略采样的得分函数修正。我们检验它在 FlashREINFORCE 里能否提高保留集准确率。
• 扩展 SWE-2 的奖励：用对偶更新控制训练落在分数/成本前沿的哪一点。出于同样的考虑，我们按相同 token 和相同关键路径 token 两种口径报告团队结果。

Acknowledgements致谢

The author is pleased to acknowledge that the work reported on in this post was substantially performed using the Princeton Research Computing resources at Princeton University. Princeton Research Computing is a consortium of groups including the Princeton Institute for Computational Science and Engineering (PICSciE) and Research Computing at Princeton University.

本文报告的工作主要使用普林斯顿大学 Princeton Research Computing 的计算资源完成。Princeton Research Computing 是由普林斯顿计算科学与工程研究所（PICSciE）和普林斯顿大学 Research Computing 等团队组成的联合体。

Experiment record实验记录

Citation

@misc{chai2026marl,
  title        = {How to design reward function in Multi-Agent RL},
  author       = {Chai, Wenhao},
  year         = {2026},
  howpublished = {Blog post},
  url          = {https://wenhaochai.com/exps/multi-agent-reward.html}
}