Quantifying the Per-layer Contribution of LLMs
Giving every layer its own next-token loss makes the shallow and middle layers predict far better than the same layers of an ordinary model, and the last layer worse, at every depth and in every residual design we tried. Weighting the intermediate losses so that together they count as much as the last layer's roughly halves this cost, and no change we tested removes it.
给每一层加上各自的下一个词损失,会让浅层和中间层比普通模型的对应层预测得好得多,但是最后一层会变差;这在我们试过的每个深度和每种残差设计里都成立。给中间层损失加权,让它们合起来与最后一层的损失一样重,可以把这个代价大致减半,但是我们测过的任何改动都消除不了它。
Setup设置
We start from the MuonH Qwen3 recipe of the Marin speedrun and add a next-token loss after every layer, read through a head of its own or the shared output head. Most runs train the 130m model of 6 layers and width 512 on 2.6B tokens, or a 48-layer model of the same width and data under ten residual designs. A few reach 1.2b. We judge each layer by its final training loss, the mean of the last 50 steps, and compare it with probes on an ordinary model and with ordinary models of that depth. Training curves put backbone compute 6ND on the x axis, where N counts the parameters up to that layer without the embedding and the heads. All runs use this code.
我们从 Marin speedrun 的 MuonH Qwen3 配方出发,在每一层后加一个预测下一个词的损失,用每一层自己的头或共享的输出头读出。大部分运行训练 130m 模型:6 层,宽 512,训练 2.6B token。另一部分运行用同样的宽度和数据训练 48 层模型,比较十种残差结构。少数运行做到 1.2b。我们用最后 50 步平均的最终训练损失衡量每一层,并和普通模型上的探针以及同深度的普通模型对比。训练曲线的横轴是骨干算力 6ND,N 是算到该层为止的参数量,不含 embedding 和头。所有运行使用这份代码。
Details详细说明
The baseline is the MuonH Qwen3 recipe of the Marin speedrun, from 130m to 1.2b. We add a next-token loss after every layer and plot each layer's final training loss against the layer number. Figures that follow training put backbone compute 6ND on the x axis, where N counts the parameters of the layers up to that layer without the embedding and the heads, and D counts the training tokens so far. The final training loss is the mean over the last 50 steps, and training curves show 50-step running means. We train most setups once. Nine runs of the 130m baseline that differ in seed or EMA but share the data order end within 0.004 nats of each other. At 48 layers, four setups also have a second seed, which moves the last-layer cost by 0.003 nats in Pre-LN, 0.014 in Hyper-Connections, 0.071 in Hyper-Connections with identity mixing and 0.083 in AttnRes (Full). Code
基线采用 Marin speedrun 的 MuonH Qwen3 配方,模型规模从 130m 到 1.2b。我们在每一层后面加上预测下一个词的损失,以层号为横轴,画出各层的最终训练损失。训练过程图的横轴是骨干算力 6ND,其中 N 是算到当前层为止的参数量,不含 embedding 和头,D 是已训练的 token 数。最终训练损失取最后 50 步的平均,训练曲线取 50 步滑动平均。多数设置只训练一次。130m 基线在相同数据顺序、不同种子或 EMA 下运行了 9 次,最终损失相差不超过 0.004 nats。在 48 层,有四种设置还训练了第二个种子,最后一层的代价因此变化:Pre-LN 0.003 nats,Hyper-Connections 0.014,单位混合的 Hyper-Connections 0.071,AttnRes (Full) 0.083。代码
Experiments实验
Answered已有结论Passive and active: how much does each layer contribute?被动与主动:每一层贡献多少?
Read passively, the baseline's early layers predict far worse than ordinary models of the same depth and catch up only at the last layer. Trained actively with separate heads, every layer ends above an ordinary model of its depth, closest in the middle layers, and the last layer ends worse than a shallower ordinary model.
被动读出时,基线模型的前几层远不如同深度的普通模型,到最后一层才追上。用独立头主动训练时,每一层都高于同深度的普通模型,中间几层最接近,最后一层比更浅的普通模型还差。
Every layer predicts through the shared output head, and every layer's loss trains that head.每一层都通过共享的输出头预测,每一层的损失都训练这个输出头。
Every layer predicts through the output head, but only the last layer's loss trains that head.每一层都通过输出头预测,但只有最后一层的损失训练这个输出头。
Every layer predicts through a head of its own.每一层都用各自的独立头预测。
Every layer predicts through a head of its own, but no gradient flows into the backbone. The probe heads share only the output head's gradient clipping, and the final eval loss ends 0.002 nats from the baseline's. The model therefore trains like the baseline.每一层都用自己的头预测,但梯度不传回骨干。这些探针头只和输出头共用梯度裁剪,最终评估损失与基线仅差 0.002 nats,因此模型本身的训练与基线几乎一致。
We train shallower ordinary models of 1 to 5 layers, each with only its output head. The 6-layer model is the baseline, and we compare layer k with the k-layer model.我们单独训练 1 到 5 层的浅层普通模型,每个模型只带自己的输出头。6 层模型就是基线,第 k 层与 k 层普通模型对比。
Details详细说明
Read passively, the baseline's early layers predict far worse than ordinary models of the same depth and catch up only at the last layer. Trained actively with separate heads, every layer ends 0.07 to 0.17 nats above an ordinary model of its depth, closest in the middle layers, and the last layer ends worse than an ordinary 4-layer model.
被动读出时,基线模型的前几层远不如同深度的普通模型,到最后一层才追上。用独立头主动训练时,每一层都比同深度的普通模型高 0.07 到 0.17 nats,中间几层最接近,最后一层则比 4 层普通模型还差。
Answered已有结论Does a deeper network benefit from supervising its shallow layers?更深的网络能否从浅层监督中获益?
No. As at 6 layers, separate heads give the shallow and middle layers a much lower loss and leave the last layers higher than the ordinary model.
不能。和 6 层时一样,独立头让浅层和中间层的损失低得多,最后几层却高于普通模型。
The 130m layer stacked to 48 layers, where every layer predicts through a head of its own.把 130m 模型的层堆叠到 48 层,每一层都用各自的独立头预测。
The same network with probe heads that send no gradient back. It trains like an ordinary 48-layer model, except that the probe heads share the output head's gradient clipping. That group now clips on 92% of logged steps instead of 0.2%, and the factor on the output head's gradient varies from step to step with a median of 0.74.同样的网络,但探针头的梯度不回传。它的训练与 48 层普通模型基本相同,只是探针头与输出头共用梯度裁剪。这使该参数组触发裁剪的记录步比例从 0.2% 升至 92%,输出头梯度的裁剪系数每步变化,中位数为 0.74。
Answered已有结论Does the residual design change what each layer contributes?残差设计会不会改变每一层的贡献?
Not in direction. In every design, separate heads give the shallow and middle layers a much lower loss and the last layers a higher one. The designs differ in where the gap closes and in how much the last layer pays, about half as much under Hyper-Connections as under most others.
不会改变方向。在每种残差设计里,独立头都会让浅层和中间层的损失低得多,同时让最后几层的损失更高。不同设计的区别在于差距在哪里收窄,以及最后一层付出的代价有多大,其中 Hyper-Connections 的代价约为多数其他设计的一半。
Normalize, transform, and add back to the stream.先归一化,经过分支变换后再加回残差流。
Normalize the branch output again before adding it back, which is the baseline's own design.分支输出在加回残差流前再归一化一次,这是基线自身采用的结构。
Scale the normalized input by 1/√ℓ, shrinking it more in deeper layers.归一化后的输入乘以 1/√ℓ,层数越深缩小越多。
Skip normalization before the branch, scale the stream by (2L)^¼, and normalize after the addition.进入分支前不归一化,把残差流乘以 (2L)^¼,与分支输出相加后再归一化。
Normalize before and after the branch, with a residual gain equal to the number of sublayers.分支前后都做归一化,残差增益设为子层总数。
Keep four residual streams, mixed by learned weights, read into the branch, and written back.维护四条残差流,用可学习权重混合后读取到分支中,再写回各条流。
Hyper-Connections with the mixing matrix between streams kept doubly stochastic.在 Hyper-Connections 的基础上,让各条流之间的混合矩阵保持双随机。
Each layer's input is a learned softmax mixture of all earlier layer outputs and the embedding.每一层的输入是前面所有层的输出与 embedding 的可学习 softmax 加权混合。
Take the same softmax mixture over 8 block outputs and the running sum of the current block.同样做 softmax 混合,但只混合 8 个块的输出和当前块内的累加和。
Attention also reads the current token's keys and values from earlier layers.注意力层同时读取当前 token 在前面各层的 key 和 value。
Answered已有结论Why does Hyper-Connections pay only half?为什么 Hyper-Connections 只付一半代价?
Because its extra streams help the separate-heads model far more than the ordinary model. With one stream it pays more than Pre-LN, and four streams are enough. mHC has the same streams and pays more because its ordinary model is better, and keeping the writes positive does not raise the cost.
因为多出的残差流对独立头模型的帮助远大于对普通模型。只有一条流时,它的代价比 Pre-LN 还高,而四条流就够了。mHC 有同样的流,但是代价更高,因为它的普通模型更好;保持写入为正并不会抬高代价。
Answered已有结论Why does supervising shallow layers hurt the final performance?为什么浅层监督会损害最终性能?
About half of the cost comes from the size of the intermediate gradient, which in a 48-layer model is about 40 times the last layer's. Weighting the intermediate losses down to the last layer's total roughly halves it. The other half stays, the last layer does not recover after the intermediate losses stop, and no other change we tried removes it without giving up the early layers' gain.
约一半的代价来自中间层梯度的大小。在 48 层模型里,这一梯度约为最后一层的 40 倍。把中间层损失的权重调低到总和与最后一层相等,可以把代价大致减半。另一半代价仍在,中间层损失停下后最后一层不会恢复,而且我们试过的其他改动如果不放弃前几层的收益,都消除不了它。
Every layer loss has weight 1, and its gradient reaches every layer below it.每层损失的权重都是 1,梯度传向它下面的所有层。
Each layer loss has weight 0.2. Together the five match the weight of the final loss.每层损失的权重是 0.2,五个加起来与最终损失的权重相等。
Each layer loss trains only its head and its own layer. Its gradient stops before the layers below.每层损失只训练自己的读出头和所在的那一层,梯度不传向下面的层。
Each layer loss has weight 0.2 and trains only its own layer.每层损失的权重是 0.2,而且只训练自己那一层。
We test fourteen further changes that attack the cost from other sides, numbered as in the figure. Changes 1 and 2 alter how the optimizer combines the two gradients, 3 to 5 what the readouts learn, 6 to 8 where their gradient flows, 9 and 10 the residual design, and 11 to 14 which readouts train the backbone.
我们又从其他方面测试了十四种处理这个代价的改法,编号与图中一致:改法 1 和 2 改变优化器合并梯度的方式,3 到 5 改变读出学习的目标,6 到 8 限制读出梯度的范围,9 和 10 更换残差结构,11 到 14 改变哪些读出向骨干传梯度。
Details详细说明
The changes are these. Change 1 clips the readout gradients on their own, and change 2 orthogonalizes the final loss's update apart from theirs and adds theirs at 0.2. In changes 3 to 5, the readouts learn the final layer's distribution, weight each token by the final layer's confidence, or read through a small exit adapter. In changes 6 to 8, the readout gradient reaches only the MLPs, only the first 256 positions of each document, or one random layer per step. Changes 9 and 10 switch to Hyper-Connections and AttnRes (Full), the designs with the smallest last-layer cost at 48 layers. In changes 11 to 13, only the readouts of layers 1 to 3, layers 1 and 2, or layer 1 send gradient into the backbone, and the deeper readouts are probes. Change 14 repeats change 12 with a second initialization seed.
具体改法如下。改法 1 单独裁剪读出的梯度。改法 2 先单独对最终损失的更新做正交化,不混入读出的更新,再以 0.2 的权重加上读出的更新。改法 3 到 5 分别让读出拟合最后一层的分布、按最后一层的置信度给每个 token 加权,或经过一个小的出口适配器再读出。改法 6 到 8 让读出梯度只更新 MLP、只取每篇文档的前 256 个位置,或每步只随机监督一层。改法 9 和 10 换用 48 层下最后一层代价最小的两种结构,即 Hyper-Connections 和 AttnRes (Full)。改法 11 到 13 只让第 1 到 3 层、第 1 和 2 层或第 1 层的读出向骨干传梯度,更深的读出都作为探针。改法 14 用第二个初始化种子重复改法 12。
Clipping the readout gradients on their own in change 1 moves neither layer. Both stay within 0.002 nats of separate heads. Exit adapters in change 5 bring layer 1 0.064 nats closer to the ordinary 1-layer model and leave layer 6 at separate heads. Weighting tokens by the final layer's confidence in change 4 cuts the layer-6 cost from 0.110 to 0.073 nats and gives up nearly all of layer 1, which ends 0.94 nats above the ordinary 1-layer model. Weight 0.05 in Figure 12 does better at both layers, at 0.394 and 0.021 nats. Changes 2, 6, 7 and 8 cost more at both layers, and training readouts through the MLPs only in change 6 costs most at layer 1, 1.106 nats, with 0.280 nats at layer 6. Learning the final layer's distribution in change 3 costs most at layer 6, 0.70 nats above the ordinary model and off the lower panel's scale, and layer 1 ends 0.58 nats above the ordinary 1-layer model. The gap opens in the first few hundred steps, while the final distribution carries little, and never closes.
改法 1 单独裁剪读出梯度,两层都无变化,与独立头相差不到 0.002 nats。改法 5 加出口适配器,让第 1 层向 1 层普通模型靠近 0.064 nats,第 6 层停在独立头的位置。改法 4 按最后一层置信度给 token 加权,把第 6 层代价从 0.110 降到 0.073 nats,却几乎放弃第 1 层,比 1 层普通模型高 0.94 nats;图 12 中权重 0.05 在两层都更好,为 0.394 和 0.021 nats。改法 2、6、7、8 在两层都更差;改法 6 只通过 MLP 训练读出,在第 1 层代价最大,为 1.106 nats,第 6 层为 0.280 nats。改法 3 学习最后一层分布,在第 6 层代价最大,比普通模型高 0.70 nats,超出下面面板范围,第 1 层也比 1 层普通模型高 0.58 nats。该差距在最初几百步最后一层分布还几乎不含信息时拉开,之后一直未合拢。
In change 9, Hyper-Connections brings layer 1 to 0.128 nats and layer 6 to 0.079 nats because the design itself helps. Its ordinary 6-layer model ends 0.046 nats below the baseline's, and against it separate heads cost the last layer 0.125 nats, as much as under the baseline design. In change 10, AttnRes (Full) pays the same last-layer cost as the baseline design at 6 layers, and in Figure 8 its seed-1 run at 48 layers pays one too.
改法 9 的 Hyper-Connections 让第 1 层到 0.128 nats、第 6 层到 0.079 nats,原因是结构本身更强:它的 6 层普通模型比基线低 0.046 nats,相对于该模型,独立头在最后一层的代价是 0.125 nats,与基线结构相同。改法 10 的 AttnRes (Full) 在 6 层下最后一层代价与基线相同,图 8 中它 48 层种子 1 的运行也同样有代价。
Sparing more layers near the output lowers the last-layer cost further. In change 11, layers 1 to 3 leave layer 1 0.105 nats above the ordinary 1-layer model and layer 6 at 0.115 nats. In change 12, layers 1 and 2 bring layer 1 to 0.061 nats above it and layer 6 to 0.082 nats. In change 13, layer 1 alone brings layer 6 to 0.048 nats and layer 1 to 0.024 nats, the closest of all changes at both layers. Change 14, a second seed of change 12, gives 0.080 nats at layer 6 against 0.082. The drop exceeds the spread between seeds. The layers whose readouts become probes pay for it. In change 12, layers 3 to 5 end 0.29 to 0.38 nats above the ordinary models of the same depth, against 0.07 to 0.09 nats under separate heads. In change 13, layers 2 to 5 end 0.27 to 0.73 nats above them.
越少监督靠近输出的层,最后一层的代价就越低。改法 11 只让第 1 到 3 层传梯度,第 1 层比 1 层普通模型高 0.105 nats,第 6 层停在 0.115 nats。改法 12 只让第 1 和 2 层传梯度,第 1 层比它高 0.061 nats,第 6 层降到 0.082 nats。改法 13 只让第 1 层传梯度,第 6 层降到 0.048 nats,第 1 层为 0.024 nats,在这两层都是所有改法中最接近的。改法 14 用第二个种子重复改法 12,在第 6 层得到 0.080 nats,对比原来的 0.082,降幅大于种子间的波动。但读出改为探针的那些层会变差:改法 12 中第 3 到 5 层比同深度普通模型高 0.29 到 0.38 nats,独立头下只高 0.07 到 0.09 nats;改法 13 中第 2 到 5 层高 0.27 到 0.73 nats。
Answered已有结论Is the gap in the heads or in the features?差距在读出头,还是在特征?
It lies in the features. Retraining the heads of either 300m model on its frozen backbone finds no better heads, and the probes still read the early layers far worse than the separate heads. Undertrained heads do not explain the gap.
差距在特征里。固定两个 300m 模型的骨干重训读出头,找不到更好的头,探针读前几层仍远不如独立头。读出头训练不足解释不了这一差距。
Each finished 300m model keeps its backbone frozen while its heads train for 2,000 more steps.训练好的 300m 模型固定骨干,只把读出头再训练 2,000 步。
Details详细说明
The gap is in the features. Retraining the heads of either 300m model on its frozen backbone finds no better heads. The separate heads end slightly higher at every layer, the probes barely move, and the probes still read the early layers far worse than the separate heads. Undertrained heads do not explain the gap.
差距在特征里。固定两个 300m 模型的骨干重训读出头,并没有找到更好的头:独立头每一层都略微升高,探针几乎不变,读前几层仍远不如独立头。读出头训练不足解释不了这一差距。
Answered已有结论Does multi-token prediction change what each layer knows?多 token 预测会不会改变每一层知道的东西?
At 130m, only in the last layers. Information about the token after next grows there, and the next-token probes barely move. The early layers do not predict sooner.
在 130m 上只改变最后几层。下下个词的信息只在那里增加,下一个词的探针几乎不变,浅层并未更早预测。
Answered已有结论Does multi-token prediction help at larger sizes?多 token 预测在更大的模型上有帮助吗?
No. At 130m, 300m, 520m and 1.2b, adding a multi-token loss leaves the held-out next-token loss within 0.005 nats of the ordinary objective, with no gain at the largest size.
没有。在 130m、300m、520m 和 1.2b 上,加上多 token 预测损失后,下一个词的留出损失与普通目标相差不超过 0.005 nats,在最大尺度上没有收益。
This test comes from a Lean result. The proofs behind it and the progress of its runs are on the Lean follow-ups page.这项检验源于一个 Lean 结论,背后的证明和各次运行的进度见 Lean 引出的实验。
Answered已有结论Why does AttnRes scramble what its layers hold?为什么 AttnRes 会打乱各层的信息?
The scrambling comes from the mixes, not from the layers. In Kimi K3 each layer reads its own learned mix of the blocks. The mix the next layer reads jumps by up to 1.39 nats from one layer to the next, while the plain sum of the blocks rises by at most 0.21 nats. The model's prediction exists only in the learned mixes and forms only in the last layers.
打乱来自混合,而不是各层本身。在 Kimi K3 中,每一层读取自己学到的各块混合。下一层读取的混合在相邻两层间最多跳动 1.39 nats,而各块的直接求和每层最多只升 0.21 nats。模型的预测只存在于学到的混合中,而且只在最后几层形成。
Details详细说明
Through the output head, every reading stays between 10.2 and 13.9 nats up to layer 78, near the 12.0-nat loss of a uniform guess over the 163,840-token vocabulary. The two mixes reach 6.2 to 6.6 nats at layer 84, 1.5 nats at layer 91 and 0.41 nats at layer 93. The mix the next layer reads jumps by up to 1.39 nats from one layer to the next, while the plain sum of the blocks rises by at most 0.21 nats. The plain sum ends at 6.0 nats, far from the output. The model's prediction exists only in the AttnRes mix. The output mix puts most of its weight on the current block. At the last layer of each of the first six blocks, namely layers 12, 24, 36, 48, 60 and 72, that weight falls to between 0.06 and 0.47 and shifts to the embedding and the finished blocks. One layer later, the new block's partial sum takes 0.85 to 1.0 of it again. K3 predicts 90% of the next tokens here, far more than a model predicts on unseen text. It very likely saw these documents in training. This figure therefore shows where in depth the prediction forms rather than how well K3 models new text.
经输出头读出,每种读法到第 78 层都在 10.2 到 13.9 nats 之间,接近在 163,840 词的词表上均匀猜测的 12.0 nats。两种混合在第 84 层降到 6.2 到 6.6 nats,第 91 层到 1.5 nats,第 93 层到 0.41 nats。下一层读取的混合在相邻两层间最多跳动 1.39 nats,而各块的直接求和每层最多只升 0.21 nats。直接求和最终停在 6.0 nats,离输出很远,模型的预测只存在于 AttnRes 混合中。输出混合把大部分权重放在当前块。在前六块的最后一层,即第 12、24、36、48、60 和 72 层,该权重降到 0.06 到 0.47,转到 embedding 和已完成块上。下一层,新块的部分和又拿回 0.85 到 1.0。K3 能预测这段文本中 90% 的下一个词,远高于模型在未见文本上的水平。它很可能在训练时见过这些文档,因此本图反映预测在什么深度形成,而非 K3 对新文本的建模能力。
Answered已有结论A full check-up of AttnRes: Kimi K3 and our modelsAttnRes 全面体检:Kimi K3 和我们的模型
Separate heads change how our AttnRes models read. Their two mixes agree, attention settles on the older blocks, the late queries align with the output head, and equal weights cost far less. Kimi K3 reads mostly its current block and, like our ordinary models, depends on its learned weights far more than our separate-heads models do.
独立头改变了我们 AttnRes 模型的读取方式。它们的两种混合一致,注意力稳定在较早的块上,后面各层的查询与输出头对齐,均匀权重的代价也小得多。Kimi K3 主要读取当前块,而且和我们的普通模型一样,对学到的权重的依赖远大于我们的独立头模型。
K3 very likely saw this WikiText-103 text in training. We therefore repeat every measurement on Paloma C4. Each figure has five panels: Kimi K3 and our 48-layer AttnRes (Block) and AttnRes (Full), each as an ordinary model and with separate heads.
K3 很可能在训练时见过这份 WikiText-103 文本,因此我们在 Paloma C4 上重复每一项测量。每张图有五个面板:Kimi K3,以及我们的 48 层 AttnRes (Block) 和 AttnRes (Full),各有普通模型与独立头模型两种。
Details详细说明
Over layers 1 to 47 of our ordinary models, the mix the next layer reads is on average 2.85 nats in Block and 4.22 nats in Full worse through the output head than the head's own mix. It jumps by up to 3.55 and 12.47 nats from one layer to the next, while the head's mix falls smoothly. With separate heads, the two readings sit 0.25 and 0.15 nats apart on average, and each layer's own head reads lower still. The plain sum of the sources never matches what the head reads. At layer 48 in the ordinary models, it sits 2.47 nats in Block and 12.19 nats in Full above the head. On Paloma C4, which K3 has likely not seen and where its final loss is 2.05 nats, every reading of K3 stays between 11 and 12.5 nats up to 75% of depth, and the prediction forms in the last layers. Its two mixes stay close, 0.31 nats apart on average, while its plain block sum ends at 6.41 nats.
在我们普通模型的第 1 到 47 层,下一层读取的混合经输出头读出,在 Block 和 Full 中平均比输出头自己的混合差 2.85 和 4.22 nats。它在相邻两层间最多跳动 3.55 和 12.47 nats,输出头自己的混合则平滑下降。用独立头时两种读法平均只差 0.25 和 0.15 nats,每层自己的头读得更低。各来源的直接求和从不等于输出头所读内容。在普通模型第 48 层,它在 Block 和 Full 中比输出头高 2.47 和 12.19 nats。在 K3 很可能未见的 Paloma C4 上,最终损失为 2.05 nats,K3 的每种读法到 75% 深度都在 11 到 12.5 nats 之间,预测在最后几层才形成。它的两种混合平均只差 0.31 nats,各块直接求和最终停在 6.41 nats。
Equal weights everywhere cost our ordinary models 7.51 nats in Block and 13.52 nats in Full, and dropping the embedding costs the Block model 11.67 nats. With separate heads, equal weights cost 1.53 and 3.12 nats, and the embedding 1.28 and 0.68 nats. Reading only the two most recent blocks costs 8.5 to 12.4 nats in all four models. Equal weights and a plain sum match in our models by construction, since every reader normalizes its input. K3 pays 7.58 nats for equal weights and 14.11 nats without the embedding, although its head gives the embedding only 0.01 at the last layer. Making one block uniform costs K3 most for its last two blocks, 1.09 and 2.42 nats.
处处改用均匀权重,普通模型在 Block 和 Full 中代价为 7.51 和 13.52 nats,Block 去掉 embedding 的代价为 11.67 nats。用独立头时,均匀权重代价仅 1.53 和 3.12 nats,去掉 embedding 为 1.28 和 0.68 nats。只读最近两块,四个模型的代价为 8.5 到 12.4 nats。我们的模型中每次读取都归一化输入,故均匀权重与直接求和在构造上损失相同。K3 用均匀权重代价为 7.58 nats,去掉 embedding 为 14.11 nats,尽管输出头在最后一层只给 embedding 0.01 权重。只把一块改为均匀时,K3 在最后两块代价最大,为 1.09 和 2.42 nats。
A follow-up locates the embedding cost. Dropping the embedding from the head alone costs at most 0.001 nats in our four models. The cost sits in the layers' reads, which account for 11.55 of the ordinary Block model's 11.67 nats. In K3, the whole cost sits in the reads of its first block, layers 1 to 12. Dropping the embedding from the head or from layers 13 and later costs nothing, from the MLP reads alone 6.2 nats, and from all reads 14.1 nats. In the first block, the embedding and the current block are the only sources, and the block needs the token input.
一个后续实验定位了 embedding 代价所在。只从输出头去掉 embedding,四个模型代价至多 0.001 nats,代价都在各层读取中,占普通 Block 模型 11.67 nats 中的 11.55 nats。在 K3 中,全部代价都在第一块即第 1 到 12 层的读取中。从输出头或第 13 层以后的读取去掉 embedding 没有代价,只从 MLP 读取去掉代价为 6.2 nats,从所有读取去掉为 14.1 nats。第一块中 embedding 和当前块是仅有的来源,第一块需要 token 输入。
Inside the first block of our ordinary Block model, the reads are nearly binary. Four sublayers, the MLPs of layers 1, 2 and 5 and the attention of layer 6, take the embedding at weights 0.62 to 1.0. The other seven take only the block's partial sum, which carries almost none of the embedding's direction, with cosine within 0.08 of 0. Dropping the embedding from one of the three MLPs costs 0.42 to 0.64 nats, from layer 6's attention 0.02 nats, from the MLPs of layers 1 and 2 together 6.8 nats, and from all four readers 10.8 nats. The token readers can stand in for one another. The reads are binary because the logit gap between the two sources is large. The two RMS-normalized keys sit about 32 apart, and for most reads, whose queries have length 4 to 8, the gap is 32 to 67. Only the reads with short queries of length 0.1 to 0.6 take a weight between 0 and 1. With separate heads, the same block-1 removal costs 0.30 nats.
在我们普通 Block 模型的第一块中,读取几乎是二值的。四个子层读取 embedding,即第 1、2、5 层的 MLP 和第 6 层的注意力,权重为 0.62 到 1.0。其余七个只读这一块的部分和,其中几乎不含 embedding 方向,余弦与 0 相差不到 0.08。从三个 MLP 之一去掉 embedding 代价是 0.42 到 0.64 nats,从第 6 层注意力去掉是 0.02 nats,从第 1、2 层 MLP 一起去掉是 6.8 nats,从四个读取子层全去掉是 10.8 nats。这几个读 token 的子层可互相替代。读取接近二值,是因为两种来源的 logit 差距很大。两个经 RMS 归一化的 key 相距约 32。多数读取的查询长 4 到 8,这些读取的差距是 32 到 67。只有查询长 0.1 到 0.6 的读取,权重才在 0 和 1 之间。用独立头时,同样去掉第一块的 embedding 代价仅 0.30 nats。
Answered已有结论Inside the ten designs: what do separate heads change?十种结构的内部:独立头改变了什么?
Many layers become cheap to delete, the last layer stops mattering, and in most designs the output head reads middle layers almost as well as their own heads. The last MLP stops pushing against the correct token, as it does in several ordinary models.
删掉很多层的代价变小,最后一层不再重要,多数结构里输出头读中间层几乎和各层自己的头一样好。几个普通模型的最后一个 MLP 会压低正确的词,加独立头后不再这样。
Each figure below has ten panels, one per 48-layer design. Red shows the ordinary model, blue the separate-heads model, on the same Paloma C4 text.
下面每张图有十个面板,各对应一种 48 层结构。红色是普通模型,蓝色是独立头模型,都读同一份 Paloma C4 文本。
Details详细说明
In eight of the ten designs, 13 to 28 layers of the separate-heads model each cost less than 0.005 nats to delete, against 0 to 7 in the ordinary models. AttnRes (Block) has only 3 such layers, and DeepNorm has 8. In AttnRes (Block), block regrouping causes this drop. Deleting a layer shifts every later layer into a new block, which costs 0.16 to 0.20 nats in the first block, 0.01 to 0.02 nats in blocks 6 and 7, and nothing in the last block. Without regrouping, 19 layers cost less than 0.005 nats, or 14 to 26 layers at thresholds from 0.003 to 0.01 nats, while every deletion outside the last block costs at least 0.014 nats.
在十种结构中的八种里,独立头模型有 13 到 28 层删掉的代价小于 0.005 nats,而普通模型只有 0 到 7 层。AttnRes (Block) 只有 3 层,DeepNorm 有 8 层。AttnRes (Block) 层数少来自重新分块:删掉一层会把后面各层移入新块,这一步对第一个块的代价是 0.16 到 0.20 nats,在第 6、7 块降到 0.01 到 0.02 nats,在最后一个块为 0。不重新分块时,它有 19 层的代价小于 0.005 nats,阈值取 0.003 到 0.01 nats 时为 14 到 26 层,而最后一个块外每次删除的代价至少是 0.014 nats。
The ordinary models of six designs lean on their last layer, which costs 0.32 to 0.49 nats to delete in Pre-LN, LayerNorm Scaling, DeepNorm, KEEL, mHC and MoDA. With separate heads the last layer costs at most 0.002 nats. Layer 1 costs 2.5 to 8.6 nats to delete in almost every model. The exceptions are ordinary mHC at 0.37 nats, whose last layer matters more, ordinary DeepNorm at 1.16 nats, and ordinary AttnRes (Full), which depends most on layer 44 at 8.99 nats. In ordinary Hyper-Connections, most of layer 1's value is its stream mixing: dropping only its sublayers costs 0.62 nats, and deleting the layer costs 5.69 nats. At layer 1 all four streams equal the embedding, and layer 1's mixing scales them by a token median of about 0.39, 2.76, 2.84 and −0.81, which later layers expect. mHC's nearly doubly stochastic mixing keeps the scales near 1, and its two numbers differ by 0.004 nats.
六种结构的普通模型依赖最后一层:在 Pre-LN、LayerNorm Scaling、DeepNorm、KEEL、mHC 和 MoDA 中,删掉最后一层的代价是 0.32 到 0.49 nats,加独立头后最多只有 0.002 nats。几乎每个模型删掉第 1 层的代价都是 2.5 到 8.6 nats。例外是普通 mHC 的 0.37 nats,其最后一层更重要,还有普通 DeepNorm 的 1.16 nats,以及普通 AttnRes (Full):它最依赖的是第 44 层,删掉这一层的代价达 8.99 nats。普通 Hyper-Connections 第 1 层的作用主要在多流混合:只去子层代价是 0.62 nats,删整层是 5.69 nats。第 1 层四条流都等于 embedding,其混合按 token 中位数把它们缩放约 0.39、2.76、2.84 和 −0.81 倍,后面各层依赖这些缩放。mHC 近似双随机的混合使缩放近于 1,两个数只差 0.004 nats。
Answered已有结论Where do the designs' differences come from?结构之间的差别从哪里来?
Mostly from the extra streams and from the ordinary model. The extra streams of Hyper-Connections lower its separate-heads loss far more than its ordinary model's, and mHC pays more because its ordinary model is better. In Hyper-Connections with identity mixing, capping each token's branch output removes most of the extra separate-heads loss. In AttnRes, the loss spikes come from outsized sources and need token-dependent routing.
主要来自多出的残差流和普通模型。Hyper-Connections 多出的流对独立头损失的降低远大于对普通模型,而 mHC 代价更高是因为它的普通模型更好。在采用单位混合的 Hyper-Connections 中,给每个 token 的分支输出设上限,就能消除大部分多出的独立头损失。AttnRes 的损失尖峰来自过大的源,而且需要依赖 token 的路由才会出现。
These tests follow from our Lean results. The proofs and the progress of the runs are on the Lean follow-ups page.这些检验源自 Lean 结论,背后的证明与各次运行的进度见 Lean 引出的实验。
Answered已有结论Does learned routing between residual paths pay?残差通路之间学到的路由有收益吗?
Not in deep models. Extra residual paths lower the held-out loss at 12, 24 and 48 layers, while learned mixing between them has no clear effect at 12 or 24 layers and costs at 48. Its cost follows how far it changes the scale of the stream sum, and that change grows with depth.
在深层模型里没有。增加残差通路在 12、24 和 48 层都能降低留出损失,但是通路之间学到的混合在 12 层和 24 层没有明显效果,到了 48 层反而带来代价。它的代价随它对流之和尺度的改变而变,而这种改变随深度增大。
Every model in this question is an ordinary model, and every number is the held-out loss of the last layer on Paloma C4. At 48 layers, mHC, which is Hyper-Connections with a constraint on its mixing, beats Hyper-Connections. We expected the constraint to be the reason, so we trained Hyper-Connections with its mixing fixed to the identity. It reached mHC's loss. The papers that introduced these designs report the opposite at 16 layers, so we trained the same pair at 12 and 24 layers.
这一节的每个模型都是普通模型,每个数值都是最后一层在 Paloma C4 上的留出损失。在 48 层,mHC 优于 Hyper-Connections,而 mHC 就是给 Hyper-Connections 的混合加了约束。我们原本以为原因是这个约束,所以把 Hyper-Connections 的混合固定为单位矩阵来训练。结果它达到了 mHC 的留出损失。提出这些设计的论文在 16 层报告了相反的结果,所以我们也在 12 层和 24 层训练了这一对模型。
Takeaways核心结论
Giving every layer its own next-token loss makes the shallow and middle layers predict far better than the same layers of an ordinary model, and the last layer worse, at every depth and in every residual design we tried. Weighting the intermediate losses so that together they count as much as the last layer's roughly halves this cost, and no change we tested removes it.
给每一层加上各自的下一个词损失,会让浅层和中间层比普通模型的对应层预测得好得多,但是最后一层会变差;这在我们试过的每个深度和每种残差设计里都成立。给中间层损失加权,让它们合起来与最后一层的损失一样重,可以把这个代价大致减半,但是我们测过的任何改动都消除不了它。
Acknowledgements致谢
The author is pleased to acknowledge that the work reported on in this post was substantially performed using the Princeton Research Computing resources at Princeton University. Princeton Research Computing is a consortium of groups including the Princeton Institute for Computational Science and Engineering (PICSciE) and Research Computing at Princeton University.
本文报告的工作主要使用普林斯顿大学 Princeton Research Computing 的计算资源完成。Princeton Research Computing 是由普林斯顿计算科学与工程研究所(PICSciE)和普林斯顿大学 Research Computing 等团队组成的联合体。
Experiment record实验记录
Citation
@misc{chai2026perlayer,
title = {Quantifying the Per-layer Contribution of LLMs},
author = {Chai, Wenhao},
year = {2026},
howpublished = {Blog post},
url = {https://wenhaochai.com/exps/per-layer-contribution.html}
}