How to design reward function in Multi-Agent RL多智能体 RL 中如何设计奖励函数

Setup设置

The baseline is single-agent EasyPPO with its Frontier-CS settings: its Qwen3.5-9B SFT initialization, trained on 200 FrontierSmith problems and validated on the 172 Frontier-CS algorithmic problems, scored 0–100 as the mean per-test-case score by a local judge that follows the official rules; the last C++ block of an answer is the program judged, and every prompt says so. We replace the single agent with a team, one lead and four subagents as in DeepSeek-V4.1-Flash's Agent Team mode: the subagents are helpers that may test programs against the judge and report ideas, analysis or code, and only the lead's final submission scores the team. We compare ways to split the team's reward; compute is counted in generated tokens and latency in tokens on the critical path. The first four questions below asked the same thing on competition math with FlashREINFORCE. Code: the team game, judge and scripts, the miles fork and the math runs; the experiment record links the exact commit of each finished run.

基线是单智能体 EasyPPO，用它的 Frontier-CS 设置：它的 Qwen3.5-9B SFT 初始化，在 200 道 FrontierSmith 题上训练，在 172 道 Frontier-CS 算法题上验证，由按官方规则实现的本地评测器打 0–100 分，即各测试用例得分的平均；评测的是回答里最后一个 C++ 代码块，每个 prompt 都写明了这一点。我们把单智能体换成团队，按 DeepSeek-V4.1-Flash 的 Agent Team 模式设为 1 个 lead 加 4 个 subagent：subagent 是帮手，可以把程序交给评测器自测，再汇报思路、分析或代码；只有 lead 的最终提交决定团队成绩。我们比较团队奖励的不同分法；算力按生成的 token 数计，延迟按关键路径上的 token 数计。下面前 4 个问题是在竞赛数学上用 FlashREINFORCE 问的同一件事。代码：团队游戏、评测器和脚本、miles 分支和数学实验；实验记录里链接了每个跑完的 run 的准确提交。

--- baseline: one agent, one judged program (simplified)+++ ours: a team game with judge feedback (simplified) def episode(problem):-    return judge(problem, agent.solve(problem)).score+    # the lead writes one task per subagent# lead 给每个 subagent 写一条任务+    tasks = lead.plan(problem, n=4)+    # each subagent may test up to 3 programs, sees each result, then reports# 每个 subagent 最多自测 3 个程序，看到每次结果后写报告+    reports = [sub.explore(problem, t, judge, tests=3) for t in tasks]+    # the lead reads every report and submits the team's program# lead 读完所有报告，提交团队的程序+    final = judge(problem, lead.solve(problem, reports))+    # only the final submission scores the team; the split is what we compare# 只有最终提交决定团队成绩；比较的是奖励怎么分+    return split(final, reports)

Experiments实验

Answered已有结论Does FlashREINFORCE's quick-start recipe reproduce the paper on R1-Distill-1.5B?FlashREINFORCE 的快速入门配方能否在 R1-Distill-1.5B 上复现论文？

Yes: our run follows the paper's trust-region arm through training and ends only slightly below it.

能复现：我们的 run 一路跟住论文的信任域组，结束时只略低一点。

Answered已有结论Does score centering remove the train/inference mismatch, and does it buy held-out accuracy?Score centering 能否消除训练/推理不一致？它能提高保留集准确率吗？

It removes the mismatch, but held-out accuracy does not improve.

能消除不一致，但保留集准确率没有提高。

Answered已有结论Does talking help a team of agents on math, and which credit makes it help?在数学题上，智能体之间交流有没有帮助？哪种功劳分配让它有帮助？

Untrained talking hurts, because agents copy each other; only paying each turn for its own gain raised held-out accuracy.

未经训练的交流有害，因为智能体会互相照抄；只有按每一轮自己的增益给奖励，保留集准确率才提高。

Answered已有结论Does DeepSeek-style lead-and-teammates training beat one agent at the same budget?DeepSeek 式的 lead 加队友训练，能否胜过同预算的单智能体？

No: the collaboration bonus makes the lead delegate more often, but the team does not beat the same lead working alone.

不能：协作奖励让 lead 更常委派，但团队没有胜过同一个 lead 单干。

Answered已有结论Can a local judge stand in for the official Frontier-CS judge?本地评测器能否代替官方的 Frontier-CS 评测器？

Yes: it scores like the official judge on classic and interactive problems, and contestant code cannot read the test answers.

能：它在普通题和交互题上的打分都与官方评测器一致，而且选手代码读不到测试答案。

Answered已有结论Which starting point can RL train from on Frontier-CS?在 Frontier-CS 上，RL 能从哪个起点开始训练？

EasyPPO's SFT initialization: it finishes most of its answers, while the base model with reasoning mostly runs out of tokens.

EasyPPO 的 SFT 初始化：它大部分回答都能写完，而原版模型开启思考时大多用完了 token。

Testing检验中Does our EasyPPO port train a single agent on Frontier-CS?我们移植的 EasyPPO 能否在 Frontier-CS 上训练单智能体？

Testing检验中Does a team without feedback beat one agent?没有反馈的团队能否胜过单智能体？

Testing检验中With judge feedback, which reward split makes a team beat one agent at equal tokens?有评测反馈时，哪种奖励分法能让团队在相同 token 下胜过单智能体？

Testing检验中Why do most programs fail to compile, and does a prompt, the base weights or a lower temperature fix it?为什么大部分程序编译不过？换 prompt、换权重或降低温度能否解决？

Takeaways核心结论

Related work相关工作

• EasyPPO (arXiv 2609.36802): PPO with a critic and three small changes, with Frontier-CS results. It is our baseline and initialization, extended here from one agent to a team.
• DeepSeek-V4.1-Flash (arXiv 2609.19969), Sec. 5.3.5: an Agent Team mode trained with task reward, a collaboration bonus and a critical-path latency penalty. We replicate its training on math and adopt its team structure on Frontier-CS.
• FlashREINFORCE: asynchronous REINFORCE with a sequence trust region. We reproduce its R1-Distill-1.5B curve and train the math teams with it.
• Score centering (arXiv 2609.20807): a score-function correction for off-policy sampling. We test whether it buys held-out accuracy inside FlashREINFORCE.
• Extending SWE-2's reward: a cost-aware reward whose dual update steers where training lands on the score/cost frontier. For the same reason we report teams at equal tokens and at equal critical-path tokens.

• EasyPPO（arXiv 2609.36802）：带 critic 的 PPO 加三个小改动，含 Frontier-CS 结果。它是我们的基线和初始化，本文把它从单智能体扩展到团队。
• DeepSeek-V4.1-Flash（arXiv 2609.19969）第 5.3.5 节：Agent Team 模式，用任务分、协作奖励和关键路径延迟惩罚训练。我们在数学上复刻它的训练，在 Frontier-CS 上采用它的团队结构。
• FlashREINFORCE：带序列信任域的异步 REINFORCE。我们复现了它在 R1-Distill-1.5B 上的曲线，并用它训练数学上的团队。
• Score centering（arXiv 2609.20807）：针对离策略采样的得分函数修正。我们检验它在 FlashREINFORCE 里能否提高保留集准确率。
• 扩展 SWE-2 的奖励：用对偶更新控制训练落在分数/成本前沿的哪一点。出于同样的考虑，我们按相同 token 和相同关键路径 token 两种口径报告团队结果。

Acknowledgements致谢

The author is pleased to acknowledge that the work reported on in this post was substantially performed using the Princeton Research Computing resources at Princeton University. Princeton Research Computing is a consortium of groups including the Princeton Institute for Computational Science and Engineering (PICSciE) and Research Computing at Princeton University.

本文报告的工作主要使用普林斯顿大学 Princeton Research Computing 的计算资源完成。Princeton Research Computing 是由普林斯顿计算科学与工程研究所（PICSciE）和普林斯顿大学 Research Computing 等团队组成的联合体。

Experiment record实验记录

Citation

@misc{chai2026marl,
  title        = {How to design reward function in Multi-Agent RL},
  author       = {Chai, Wenhao},
  year         = {2026},
  howpublished = {Blog post},
  url          = {https://wenhaochai.com/blogs/multi-agent-reward.html}
}