How to design reward function in Multi-Agent RL多智能体 RL 中如何设计奖励函数
Setup设置
The main experiment runs Recursive Flint, a 40B mixture-of-experts model with 2.5B active parameters, on the 40 validation problems (val40) of the classic algorithmic problems of Frontier-CS v1.1; 67 more are kept for training (train67). Each agent plays mini-swe-agent's bash-only long game. It runs one shell command per step in a sandbox, writes and compiles files, runs its own tests, and may submit as often as it likes.
Scoring works like a contest. A submission is judged at once on half of the hidden test cases, and the agent sees this provisional score with the per-case scores and the compiler errors. The answer of an episode is the submission with the best provisional score. Its final score on the other half of the cases, which no agent sees, is the result, from 0 to 100. A local judge that follows the official rules scores both halves.
A team is four Flints in lockstep rounds: in each round every agent runs one command, in a random order. Each agent works in a git repository of its own and can read its teammates' code. Every submission and its provisional score are shown to all, and a post command sends a message to the others. Two rules, both from the test runs, make the agents talk and keep working: after each of its submissions an agent must post a message, and an agent that tries to stop starts a new conversation with its files kept, until its budget is used up. One Flint and the independent Flints play under the same restart rule. We compare a team with one Flint and with four independent Flints at the same total of generated tokens per problem: 256k, 500k, 1M or 5M. A long episode overflows the 65k-token context, so the oldest steps of a conversation are dropped and the agents keep notes in files. The team comparison is inference only; the last main question trains one Flint with GRPO, and training on the team's reward comes after this comparison.
The validation experiments below came first, and each group says what it means for the main experiment. Code: the team game, judge and scripts, the miles fork and the math runs; the experiment record links the exact commit of each finished run.
主实验用 Recursive Flint(40B 的混合专家模型,激活参数 2.5B),在 Frontier-CS v1.1 经典算法题的 40 道验证题(val40)上运行;另有 67 道留作训练(train67)。每个智能体玩 mini-swe-agent 只用 bash 的长局:在沙箱里每步运行一条 shell 命令,写文件、编译、运行自己的测试,并且可以随意多次提交。
打分方式和比赛一样。每次提交立即在一半隐藏测试点上评测,智能体看到这个临时得分,以及每个测试点的分数和编译错误。一局的答案是临时得分最高的那次提交。它在另一半测试点上的最终得分就是结果,范围 0 到 100;这一半测试点任何智能体都看不到。两半都由按官方规则实现的本地评测器打分。
一个团队是 4 个 Flint,按轮同步:每一轮每个智能体运行一条命令,顺序随机。每个智能体在自己的 git 仓库里工作,可以读队友的代码。每次提交和它的临时得分对所有智能体可见,post 命令把消息发给其他智能体。两条规则都来自测试 run,让智能体交流并持续工作:每次提交之后智能体必须发一条消息;智能体想停下时,保留文件、从新的对话重新开始,直到预算用完。1 个 Flint 和独立的 Flint 也按同样的重启规则。我们在每道题生成 token 总数相同的条件下,把团队和 1 个 Flint、4 个独立的 Flint 比较:256k、500k、1M 或 5M。长局会超出 65k token 的上下文,所以对话里最早的步骤会被删掉,智能体把笔记记在文件里。团队比较只做推理;主实验最后一个问题用 GRPO 训练 1 个 Flint;用团队奖励训练放在这个比较之后。
下面的验证性实验都在主实验之前,每一组都写明它对主实验意味着什么。代码:团队游戏、评测器和脚本、miles 分支和数学实验;实验记录里链接了每个跑完的 run 的准确提交。
--- baseline: one agent, one judged program (simplified)+++ ours: a team game with judge feedback (simplified) def episode(problem):- return judge(problem, agent.solve(problem)).score+ # the lead writes one task per subagent# lead 给每个 subagent 写一条任务+ tasks = lead.plan(problem, n=4)+ # each subagent may test up to 3 programs, sees each result, then reports# 每个 subagent 最多自测 3 个程序,看到每次结果后写报告+ reports = [sub.explore(problem, t, judge, tests=3) for t in tasks]+ # the lead reads every report and submits the team's program# lead 读完所有报告,提交团队的程序+ final = judge(problem, lead.solve(problem, reports))+ # only the final submission scores the team; the split is what we compare# 只有最终提交决定团队成绩;比较的是奖励怎么分+ return split(final, reports)
Experiments实验
Testing检验中Does a team of four Flints beat one Flint and four independent Flints at the same tokens?4 个 Flint 组成的团队,能否在相同 token 下胜过 1 个 Flint 和 4 个独立的 Flint?
Four Flint agents (Recursive Flint, a 40B MoE) play mini-swe-agent's bash-only long game, each in a git repository of its own that only it changes. After each command the harness publishes the agent's commits to a shared repository, and before each command it fetches the others' commits as team/agent1 to team/agent4, so an agent takes a teammate's code with git, for example git checkout team/agent2 -- solution.cpp. After each of its submissions an agent must post a message before its next command; the message reaches the others at the top of their next observation. An agent that tries to stop starts a new conversation with its files kept, until its budget is used up. The team's answer is the submission with the best provisional score. Each agent has 64k, 125k, 250k or 1.25M tokens (256k, 500k, 1M or 5M in all).4 个 Flint 智能体(Recursive Flint,40B MoE)玩 mini-swe-agent 只用 bash 的长局,每个智能体在只有它自己能改的 git 仓库里工作。每条命令之后,harness 把它的 commit 发布到一个共享仓库;每条命令之前,把其他智能体的 commit 取回成 team/agent1 到 team/agent4,智能体用 git 拿队友的代码,例如 git checkout team/agent2 -- solution.cpp。每次提交之后,智能体必须先发一条消息,才能运行下一条命令;消息出现在其他智能体下一次观察的开头。智能体想停下时,保留文件、从新的对话重新开始,直到预算用完。团队的答案是临时得分最高的那次提交。每个智能体 64k、125k、250k 或 1.25M token(共 256k、500k、1M 或 5M)。
Four Flints per problem with a quarter of the budget each (64k, 128k, 250k or 1.25M), under the same restart rule, with no shared code and no messages; the submission with the best provisional score counts.每道题 4 个 Flint,各拿四分之一预算(64k、128k、250k 或 1.25M),重启规则相同,不共享代码,也不发消息;取临时得分最高的提交。
One Flint with 250k, 500k, 1M or 5M tokens, under the same restart rule.1 个 Flint,250k、500k、1M 或 5M token,重启规则相同。
Answered已有结论Do Flints talk to each other when they may?允许发消息时,Flint 之间会交流吗?
No, unless the game makes them. In four team test runs (32 agents, 3,196 commands) and in the first 16 episodes of the two 5M teams, no Flint posted a message; two agents looked the post command up and still sent none. When an agent must post after each of its submissions, it does: in a test run all 21 owed posts were sent, and teammates acted on them. One agent rebuilt its solver on the approach a teammate posted and reached the team's best provisional score.
不会,除非游戏强制。在 4 次团队测试 run(32 个智能体,3,196 条命令)里,以及两个 5M 团队的前 16 局里,没有一个 Flint 发过消息;有两个智能体查过 post 命令,仍然没有发。规定每次提交后必须发消息时,智能体就会发:一次测试 run 里应发的 21 条消息全部发出,队友也照着消息行动。有一个智能体按队友消息里的思路重写了求解器,拿到了团队最高的临时得分。
Answered已有结论Does Flint keep working until its budget is used up?Flint 会一直工作到预算用完吗?
Not on its own. When it may end its work, one Flint at 500k ends after 107k tokens on average (train67). Told to go on, every Flint in a test run stopped changing its program after its first try to stop and spent the rest of its budget resubmitting it. Starting its conversation over, with its files kept, makes it work again: 7 of 8 test agents submitted new code after a restart. In the 5M run so far, one Flint restarts about 20 times per problem, and in 20 of 32 problems its best score has not risen in the second half of its rounds.
自己不会。允许自己结束时,500k 的 1 个 Flint 平均用 107k token 就结束了(train67)。只回复“继续”时,测试 run 里每个 Flint 在第一次想停之后都不再改程序,把剩下的预算花在重复提交上。保留文件、让它的对话从头开始,它就会重新工作:8 个测试智能体中有 7 个在重新开始后提交了新代码。在目前的 5M run 里,1 个 Flint 每道题重新开始约 20 次;32 道题中有 20 道,它的最高分在后一半轮数里没有再上升。
Answered已有结论Does GRPO on one problem raise Flint's score there, and elsewhere?在一道题上做 GRPO,能否提高 Flint 在这道题上的得分?在其他题上呢?
There, yes; elsewhere, no. We trained one Flint with GRPO on the packing problem for 20 steps of eight 128k-token episodes. On 32 new packing episodes its score rose from 29.5 to 46.3 (one-sided permutation p = 0.001), mostly because no episode ended without a submission any more (9 of 32 before, none after); the best episode stayed at the reference level, about 60. On val40, one episode per problem, the score went from 19.8 to 18.5, within the noise.
在这道题上能,在其他题上不能。我们在 packing 题上用 GRPO 训练 1 个 Flint,20 步,每步 8 局、每局 128k token。在 32 局新的 packing 上,得分从 29.5 升到 46.3(单侧置换检验 p = 0.001),主要因为不再有局没有提交就结束(之前 32 局中有 9 局,之后没有);最好的一局仍停在参考水平,约 60 分。在 val40 上每题 1 局,得分从 19.8 变为 18.5,在噪声范围内。
Testing检验中Does GRPO on 34 problems raise Flint's score on held-out problems?在 34 道题上做 GRPO,能否提高 Flint 在保留题上的得分?
Testing检验中How much faster can Flint's RL rollouts run without changing what the trainer sees?在不改变训练器所见的前提下,Flint 的 RL rollout 能快多少?
Testing检验中Can an RL run add idle GPUs from other jobs to its rollouts without changing what the trainer sees?一个 RL run 能否把其他作业空闲的 GPU 加进 rollout,而不改变训练器所见?
Answered已有结论Does one document carry an RL step's gradient?一步 RL 的梯度会被一篇文档主导吗?
Sometimes. In replays of run20's twenty rollouts at learning rate zero (w1130), one micro-batch holds over half of the gradient's squared norm in 5 rollouts, and 89 to 95 percent in 3. In those 3, one document holds 93 to 98 percent of the gradient at layer 5, no single token over half, mostly through numbers and symbols in its context and once through code it generated. The router's normalization does not amplify these rows: their routed scores are ordinary. Clipping at 1 gives each step one length, so in those steps the other documents keep 22 to 33 percent of their amplitude. Between identical replays the norm moves by about 2 percent, because Transformer Engine's fused attention with sinks sums the sinks' gradient, and at 32k tokens or more the queries', in no fixed order; the forward pass is identical bit for bit. We have not tested whether clipping per document would help learning.
有时会。用学习率 0 重放 run20 的 20 个 rollout(w1130):有 5 个 rollout 里,一个 micro-batch 占梯度平方和的一半以上,其中 3 个占 89% 到 95%。在这 3 个里,一篇文档占第 5 层梯度的 93% 到 98%,没有哪个 token 超过一半,主要通过它上下文里的数字和符号,有一次通过它自己生成的代码。路由的归一化没有放大这些行:它们的路由分数很普通。按 1 裁剪让每一步长度相同,所以在这些步里,其他文档只保留 22% 到 33% 的幅度。相同重放之间范数约相差 2%,因为 Transformer Engine 带 sink 的融合注意力对 sink 的梯度求和,以及在 32k token 以上时对 query 的梯度求和,求和顺序不固定;前向逐位相同。按文档裁剪是否有助于学习,我们还没有测。
Answered已有结论Can a local judge stand in for the official Frontier-CS judge?本地评测器能否代替官方的 Frontier-CS 评测器?
Yes. It scores like the official judge on classic and interactive problems, and contestant code cannot read the test answers. Inside training, every judge after the first scored interactive answers as 0, and the validation scores of those runs count them as 0.
能。它在普通题和交互题上的打分都与官方评测器一致,而且选手代码读不到测试答案。在训练中,第一版之后的各版评测器给交互题的回答全部打 0 分,这些 run 的验证集得分也按 0 分计入。
The official rules re-implemented, with contestant code sandboxed and every case's output deleted as soon as it is scored.按官方规则重新实现,选手代码在沙箱里运行,每个测试点判完立即删除输出。
Frontier-CS's own engine, run on the same node.Frontier-CS 自己的评测引擎,在同一节点上运行。
Answered已有结论Does the trainer score tokens the way the rollout engine sampled them?训练器给 token 的打分,与 rollout 引擎采样时的打分一致吗?
Yes, up to bf16 rounding. The two scores differ by as much as either side differs from itself when only the order of a sum changes, and the gap stays flat over training. We therefore apply no importance correction.
一致,只差 bf16 的舍入。两者之间的差距,和任一侧只改变求和顺序时与自己的差距一样大,而且训练全程不变。因此我们不做重要性校正。
One rollout's response tokens, scored by the SGLang engine that sampled them, by Megatron's recompute and its replays at two tensor-parallel sizes, and by SGLang prefill alone, again and in one batch; the sampled tokens' log probabilities are compared token by token.取一次 rollout 的回复 token,分别由采样它们的 SGLang 引擎、Megatron 的重算及其两种张量并行规模下的重放、以及 SGLang 的 prefill(单条、重复一次、整批)打分,逐个 token 比较被采样 token 的 log 概率。
Answered已有结论Does FlashREINFORCE's quick-start recipe reproduce the paper on R1-Distill-1.5B?FlashREINFORCE 的快速入门配方能否在 R1-Distill-1.5B 上复现论文?
Yes. Our run stays within about 2 points of the paper's arms at every reported step and ends about 2 points below the trust-region arm; each arm has one seed.
能复现。在论文给出的每个步数上,我们的 run 与论文两组相差都在约 2 分以内,结束时比信任域组低约 2 分;每组只有一个 seed。
One agent, asynchronous REINFORCE with a sequence trust region: the quick-start recipe with the paper's settings on our GPU layout.单个智能体,带序列信任域的异步 REINFORCE:快速入门配方,用论文的设置,GPU 布局按我们的节点。
The two arms the paper reports: with the trust region, and importance sampling only.论文报告的两组:带信任域的,以及只做重要性采样的。
Answered已有结论Does score centering remove the train/inference mismatch, and does it buy held-out accuracy?Score centering 能否消除训练/推理不一致?它能提高保留集准确率吗?
It removes the mismatch, and held-out accuracy stays the same. The mismatch therefore does not limit held-out accuracy.
能消除不一致,保留集准确率保持不变。因此不一致并不限制保留集准确率。
Subtracts the expected score under the sampler's top-32 head, so the importance weight can be one; the trust region stays. Measured against the FlashREINFORCE recipe above.减去采样器 top-32 头下的期望得分,因此重要性权重可以取 1;信任域保留。对照是上面的 FlashREINFORCE 配方。
The same code, seed and weight with centering off.代码、种子和权重都相同,只关掉 centering。
Answered已有结论Does talking help a team of agents on math, and which credit makes it help?在数学题上,智能体之间交流有没有帮助?哪种功劳分配让它有帮助?
Untrained talking hurts, because agents copy each other. Paying each turn for its own gain gave the highest held-out accuracy, in a contrast with no registered bar and two seeds.
未经训练的交流有害,因为智能体会互相照抄。按每一轮自己的增益给奖励时保留集准确率最高;这个对比没有事先登记门槛,只有两个 seed。
Four copies of one policy answer, read each other's notes, and answer again.同一策略的 4 个副本先作答,读彼此的笔记,再作答一次。
The same four answers at the same tokens, without the notes.同样 4 次作答、同样的 token,但不给笔记。
A fifth agent reads four solvers' answers and gives the team's answer.第 5 个智能体读 4 个解答者的答案,给出团队答案。
| Study研究 | Setup做法 | Result against the pre-registered bar对照事先登记门槛的结果 |
|---|---|---|
| 1 | four agents answer, read each other's notes, answer again; scored if any is right4 个智能体作答,读彼此笔记后再答;任一答对即得分 | no gain (−0.97 points, t −1.45)无提升(−0.97 分,t −1.45) |
| 2 | the same game on a hard subset同一游戏换到难题子集 | untrained talking lowers team coverage by 3.9 points: agents copy each other未训练时交流让团队覆盖率降 3.9 分:智能体互相照抄 |
| 3 | coverage or plurality vote, broadcast or Shapley credit覆盖率或多数投票,广播或 Shapley 分配 | both below the bar (+0.02 and +1.58; bar +2.3)都低于门槛(+0.02 和 +1.58;门槛 +2.3) |
| 4 | a cleaner note protocol更干净的笔记协议 | coverage still −4.44 points: the copying is intrinsic覆盖率仍低 4.44 分:照抄是内在的 |
| 5 | credit per turn: round 2 is paid only for its gain over round 1逐轮功劳:第 2 轮只按相对第 1 轮的提升给奖励 | the registered learning gate was missed, so the registered contrasts (vote, own credit) are uninformative; per-turn credit is +6.2 points over silent training on AIME24 (8 of 8 points, 2 seeds), a contrast with no registered bar没有通过事先登记的学习门槛,所以登记的对比(投票、个人功劳)没有结论;逐轮功劳在 AIME24 上比静默训练高 6.2 分(8 个点全部更高,2 个 seed),这个对比没有事先登记门槛 |
| 6 | a trained reader that aggregates four solvers训练一个汇总 4 个解答者的 reader | no better than an untrained reader (+0.3 ± 2.4; bar +2)不比未训练的 reader 好(+0.3 ± 2.4;门槛 +2) |
| 7 | a 27B lead with up to three teammates, trained as in DeepSeek-V4.1-Flash27B 的 lead 带最多 3 个队友,照 DeepSeek-V4.1-Flash 训练 | not met: +0.29 [−1.52, +2.15] with the collaboration bonus, +1.20 [−0.88, +3.30] without未达到:加协作奖励 +0.29 [−1.52, +2.15],不加 +1.20 [−0.88, +3.30] |
Answered已有结论Does DeepSeek-style lead-and-teammates training beat one agent with the same turn and token limits?DeepSeek 式的 lead 加队友训练,能否胜过同预算的单智能体?
No. With the collaboration bonus the lead delegates more often, and the team does not beat the same lead working alone.
不能。有协作奖励时 lead 更常委派,而团队没有胜过同一个 lead 单干。
A Qwen3.8-27B lead delegates to up to three teammates and answers; task reward plus a collaboration bonus minus latency, as in DeepSeek-V4.1-Flash.Qwen3.8-27B 的 lead 把子任务委派给最多 3 个队友,再给出答案;奖励是任务分加协作奖励减延迟,照 DeepSeek-V4.1-Flash。
One agent with the same model, turns and token limits.同一个模型、同样轮数和 token 上限的单个智能体。
Answered已有结论Which starting point can RL train from on Frontier-CS?在 Frontier-CS 上,RL 能从哪个起点开始训练?
EasyPPO's SFT initialization. It finishes most of its answers, while the base model with reasoning mostly runs out of tokens.
EasyPPO 的 SFT 初始化。它大部分回答都能写完,而原版模型开启思考时大多用完了 token。
Qwen3.5-9B with its reasoning on.开启思考的 Qwen3.5-9B。
The same weights answering directly.同一份权重,直接作答。
Qwen3.5-9B fine-tuned on DeepSeek-V3.1 solutions by the EasyPPO authors.EasyPPO 作者用 DeepSeek-V3.1 的解答微调过的 Qwen3.5-9B。
Answered已有结论Does our EasyPPO port train a single agent on Frontier-CS?我们移植的 EasyPPO 能否在 Frontier-CS 上训练单智能体?
Yes. With the paper's Frontier-CS settings, one agent rises from 5.8 at the SFT initialization to 18.1 on the classic validation problems (one seed, two episodes per problem). The attempt in the figure ran out of GPU memory at its first actor step; the experiment record lists the runs that replaced it.
能。用论文的 Frontier-CS 设置,单个智能体在经典验证题上从 SFT 初始化的 5.8 分升到 18.1 分(一个 seed,每题 2 局)。图中这次尝试在第一次更新 actor 时显存不足而中止;实验记录列出了替代它的 run。
One agent writes one program per problem; its judged score is the reward; a critic, trained alone for the first 30 rollouts.一个智能体每题写一个程序,评测分数即奖励;带 critic,前 30 个 rollout 只训练 critic。
Testing检验中Does a team without feedback beat one agent?没有反馈的团队能否胜过单智能体?
Stopped on October 3, when the game with judge feedback below replaced it; its running arms finished as a no-feedback control. After ten actor steps no reward arm had moved apart from one agent. These runs judge the longest code block of an answer, the wrong setting.
10 月 3 日停止,由下面带评测反馈的游戏取代;已在跑的几组跑完,作为无反馈的对照。actor 更新 10 步后,没有一种奖励和单个智能体拉开差距。这些 run 评测回答中最长的代码块,是错误的设置。
Three teammates write programs; the lead reads them without any scores and writes the final one.3 个队友各写程序;lead 看不到任何分数,读完后写最终程序。
A single agent given the team's token budget.给单个智能体和团队同样的 token 预算。
Answered已有结论With judge feedback, which reward split makes a team beat one agent at equal tokens?有评测反馈时,哪种奖励分法能让团队在相同 token 下胜过单智能体?
None of the three. The bonus reward is the best team reward, and it only ties one agent that revises five times with the same feedback; one agent that takes the best of five scores at least as high. Final checkpoints, four episodes per classic problem, one seed.
三种都不能。加分奖励是最好的团队奖励,但只与有同样反馈、串行修改 5 次的单个智能体持平;独立提交 5 次取最高的单个智能体至少一样高。取最终 checkpoint,每道经典题 4 局,一个 seed。
The lead writes four tasks; each subagent may test up to three programs, sees each result and reports; the lead's final submission is the team's. Measured against the EasyPPO single agent above.lead 写 4 条任务;每个 subagent 最多自测 3 个程序,看到每次结果后写报告;lead 的最终提交就是团队的提交。对照是上面的 EasyPPO 单智能体。
One agent submits five times in one chat, seeing each result; only the fifth counts.一个智能体在同一对话里提交 5 次,每次都看到结果;只有第 5 次算数。
One agent submits five times independently; the best counts.一个智能体独立提交 5 次,取最高分。
Answered已有结论Why does each setup with feedback score as it does?带反馈的各做法,得分为什么高、为什么低?
Each setup is held back by its submission. Except under the difference reward, every setup judged a better program before it submitted. The bonus wins because it pays subagents to test, which also teaches the one shared model to write programs that compile. The difference reward loses because a report whose tests all fail makes the lead's final worse, and the reward thus teaches subagents to stop testing. One seed.
各做法都受限于最后的提交。除差分奖励外,每个做法在提交之前都已评测过更好的程序。加分奖励赢,是因为它奖励 subagent 自测,共用的一个模型也因此学会写能编译的程序。差分奖励输,是因为自测全部失败的报告会拖低 lead 的最终程序,这个奖励便教 subagent 不再自测。只有一个 seed。
Testing检验中Why do most programs fail to compile, and does a prompt, the base weights or a lower temperature fix it?为什么大部分程序编译不过?换 prompt、换权重或降低温度能否解决?
Not run. The project moved to Flint on October 21, before this study started. With a shell the problem shrinks: the agent compiles before it submits, and about 2% of its submissions fail to compile.
没有跑。10 月 21 日项目转到 Flint 时,这项研究还没开始。有了 shell,这个问题就小了:智能体提交前先编译,只有约 2% 的提交编译失败。
The SFT initialization with the problem prompt alone.SFT 初始化,只给题目 prompt。
Declare every name, match every type, close every brace.声明每个名字,类型一一对应,括号全部闭合。
The same checklist appended to the problem.同一份清单附在题目之后。
The base model, reasoning on, with the system message.原版模型开启思考,加上同一条 system 消息。
The SFT initialization sampled cooler.SFT 初始化,以较低温度采样。
Answered已有结论Does one agent with a shell (mini-swe-agent) score higher than one answer?给单个智能体一个 shell(mini-swe-agent),得分能否超过一次作答?
No. With the same EasyPPO weights, one answer scores 18.1 and the agent in the shell 9.9 on the classic validation problems. The untrained agent scores 8.7, and 32 rollouts of RL in the shell raise it to 10.1. We did not find what costs the shell those 8 points; the per-step token limit, the reasoning the scaffold drops and the format errors are the candidates. The RL run was stopped at the October 21 restart.
不能。用同一份 EasyPPO 权重,一次作答在经典验证题上得 18.1 分,shell 里的智能体得 9.9 分。未训练的智能体得 8.7 分,在 shell 里 RL 训练 32 个 rollout 后升到 10.1 分。我们没有找到 shell 丢掉这 8 分的原因;候选是每步的 token 上限、框架丢掉的推理和格式错误。这个 RL run 在 10 月 21 日重新开始时停止。
One agent in a sandboxed shell, one bash command per step: it writes and compiles files, runs the samples and its own tests, and submits one file. Only the submission meets the hidden tests, and an episode without a submission scores 0. An episode had at most 40 steps until rollout 20; from rollout 21 only the token budget and the context end it, because 41% of episodes had stopped at the 40th step.一个智能体在沙箱 shell 里工作,每步一条 bash 命令:写文件、编译、运行样例和自己的测试,最后提交一个文件。只有提交的文件会被隐藏测试打分,没有提交的局记 0 分。rollout 20 之前每局最多 40 步;从 rollout 21 起只由 token 预算和上下文长度结束一局,因为有 41% 的局停在了第 40 步。
The same model writes one program in one reply, as in EasyPPO.同一个模型在一次回复里写一个程序,与 EasyPPO 相同。
Answered已有结论Does a swarm of four mini-swe-agents that coordinate only through one shared repository beat one agent with four times the tokens?4 个 mini-swe-agent 组成的 swarm 只通过一个共享仓库协作,能否胜过 token 为 4 倍的单个智能体?
No. Four untrained agents in one repository score 8.6, one agent 8.7, and one agent with four times the tokens 9.4 on the classic validation problems. The swarm generates 5.6 times the tokens of one agent. The best of four single-agent episodes, picked with the hidden tests, scores 17.3, so the loss is in which program gets submitted. The swarm RL run was cancelled at the October 21 restart.
不能。共用一个仓库的 4 个未训练智能体得 8.6 分,单个智能体 8.7 分,token 为 4 倍的单个智能体 9.4 分(经典验证题)。swarm 生成的 token 是单个智能体的 5.6 倍。用隐藏测试从 4 局单智能体里挑最好的一局,得 17.3 分;所以分数丢在“提交哪个程序”上。swarm 的 RL run 在 10 月 21 日重新开始时取消。
Four mini-swe-agents coordinate only through one shared git repository. Each works under its own git name, and the hidden tests judge solution.cpp as it was at the last submit.4 个 mini-swe-agent 只通过一个共享的 git 仓库协作。每个用自己的 git 名字工作;隐藏测试评测最后一次提交时的 solution.cpp。
One mini-swe-agent with four times the tokens and no step limit.一个 mini-swe-agent,token 为 4 倍,不限步数。
Takeaways核心结论
Acknowledgements致谢
The author is pleased to acknowledge that the work reported on in this post was substantially performed using the Princeton Research Computing resources at Princeton University. Princeton Research Computing is a consortium of groups including the Princeton Institute for Computational Science and Engineering (PICSciE) and Research Computing at Princeton University.
本文报告的工作主要使用普林斯顿大学 Princeton Research Computing 的计算资源完成。Princeton Research Computing 是由普林斯顿计算科学与工程研究所(PICSciE)和普林斯顿大学 Research Computing 等团队组成的联合体。
Experiment record实验记录
Citation
@misc{chai2026marl,
title = {How to design reward function in Multi-Agent RL},
author = {Chai, Wenhao},
year = {2026},
howpublished = {Blog post},
url = {https://wenhaochai.com/exps/multi-agent-reward.html}
}