Quantifying the Per-layer Contribution of LLMs

Setup设置

The baseline is the MuonH Qwen3 recipe of the Marin speedrun, from 130m to 1.2B. We add a next-token loss after every layer and judge each layer by its final training loss, plotted against the layer number. Figures that follow training put backbone compute 6ND on the x axis, where N counts the parameters of the layers up to that one, without the embedding and the heads, and D the training tokens so far. The final training loss is the mean over the last 50 steps, and training curves are 50-step running means. Each setup is trained once; nine runs of the 130m baseline that differ in seed or EMA but share the data order end within 0.004 nats of each other. Code

基线是 Marin speedrun 里的 MuonH Qwen3 配方，规模从 130m 到 1.2B。我们在每一层后面加一个预测下一个词的损失，用每层的最终训练损失来衡量，横轴是层号。随训练变化的图，横轴是骨干算力 6ND：N 是到这一层为止各层的参数量，不算 embedding 和头，D 是到当时为止的训练 token 数。最终训练损失是最后 50 步的平均值，训练曲线是 50 步的滑动平均。每种做法只训练一次；130m 基线跑过 9 次，种子或 EMA 不同、数据顺序相同，它们的最终损失相差不超过 0.004 nats。代码

--- baseline loss (simplified)+++ per-layer losses (simplified; the Code link has the real code) def loss_fn(model, tokens):     targets = tokens[1:]     x = model.embed(tokens[:-1])-    for block in model.blocks:+    loss = 0.0+    for k, block in enumerate(model.blocks[:-1]):         x = block(x)-    return cross_entropy(model.head(model.norm(x)), targets)+        # probes: read the layer, but send no gradient into the model+        h = stop_gradient(x) if probe else x+        # every earlier layer has its own norm and output head, and predicts the next token+        logits = model.layer_heads[k](model.layer_norms[k](h))+        loss += cross_entropy(logits, targets)+    x = model.blocks[-1](x)+    # the usual loss at the last layer, at the same weight+    return loss + cross_entropy(model.head(model.norm(x)), targets)

--- 基线损失（简化）+++ 逐层损失（简化；真实代码见“代码”链接） def loss_fn(model, tokens):     targets = tokens[1:]     x = model.embed(tokens[:-1])-    for block in model.blocks:+    loss = 0.0+    for k, block in enumerate(model.blocks[:-1]):         x = block(x)-    return cross_entropy(model.head(model.norm(x)), targets)+        # 探针：读取这一层，但梯度不传回模型+        h = stop_gradient(x) if probe else x+        # 前面每一层都有自己的 norm 和输出头，也预测下一个词+        logits = model.layer_heads[k](model.layer_norms[k](h))+        loss += cross_entropy(logits, targets)+    x = model.blocks[-1](x)+    # 最后一层仍是原来的损失，权重相同+    return loss + cross_entropy(model.head(model.norm(x)), targets)

Experiments实验

Answered已有结论Passive and active: how much does each layer contribute?被动与主动：每一层贡献多少？

Read passively, the baseline's early layers predict far worse than ordinary models of the same depth and catch up only at the last layer. Trained actively with separate heads, every layer ends 0.07 to 0.17 nats above an ordinary model of its depth, closest in the middle layers, and the last layer ends worse than an ordinary 4-layer model.

被动读取时，基线模型的前几层远不如同深度的普通模型，到最后一层才追上。用独立头主动监督时，每一层都比同深度的普通模型高 0.07 到 0.17 nats，中间几层最接近；最后一层比普通的 4 层模型还差。

Answered已有结论Does a deeper network benefit from supervising its shallow layers?更深的网络能否从浅层监督中获益？

Testing检验中Does the residual design change what each layer contributes?残差设计会不会改变每一层的贡献？

Testing检验中Why does Hyper-Connections pay only half?为什么 Hyper-Connections 只付一半代价？

Testing检验中Why does supervising shallow layers hurt the final performance?为什么浅层监督会损害最终性能？

Answered已有结论Is the gap in the heads or in the features?差距在读出头，还是在特征？

In the features. Retraining the heads of either 300m model on its frozen backbone finds no better heads: the separate heads end slightly higher at every layer, the probes barely move, and the probes still read the early layers far worse than the separate heads. Undertrained heads do not explain the gap.

在特征。在冻结的骨干上重训两个 300m 模型的读出头，没有找到更好的头：独立头每一层都略微变高，探针几乎不变，探针读前几层仍然远不如独立头。读出头没训够解释不了这个差距。

Answered已回答Does multi-token prediction change what each layer knows?多 token 预测会不会改变每一层知道的东西？

Testing检验中Does multi-token prediction help at larger sizes?多 token 预测在更大的模型上有帮助吗？

Testing检验中Why does AttnRes scramble what its layers hold?为什么 AttnRes 会打乱各层的信息？

Takeaways核心结论

Related work相关工作

• Deeply-Supervised Nets, Lee et al. 2015 add a classifier loss at every hidden layer of an image network and find that it helps. We put the same kind of loss on a small language model: each layer gets closer to a shallower model, and the last layer gets worse.
• The logit lens, nostalgebraist 2020 and the tuned lens, Belrose et al. 2023 read the middle layers of a trained model, through its own output head or a small learned map per layer. Our probes do the same while the model trains, and they agree: the prediction forms in the last layers. Figure 16 applies the logit lens itself to every finished model, beside its probes or separate heads.
• LayerSkip, Elhoushi et al. 2024 trains early layers to predict through the shared output head, so that inference can stop early. Our shared-head setup is that training signal; here it costs the last layer about 0.13 nats.
• LLAL, Jin et al. 2026 adds a language-model loss to the layers of an ultra-sparse mixture-of-experts model to stop its experts from dying. We test the loss on a dense model, which has no such failure to fix.
• Multi-token prediction, Gloeckle et al. 2024 trains extra heads that predict several future tokens and reports gains on code tasks that grow with model size, with small models losing. DeepSeek-V3, DeepSeek-AI 2024 predicts the token after next with one sequential module, the form we use. We look at what the extra loss does to each layer, at 130m and across four sizes.

• Deeply-Supervised Nets，Lee et al. 2015 在图像网络的每个隐藏层加分类损失，发现有帮助。我们在小语言模型上加同类损失：每层更接近一个更浅的模型，最后一层却变差。
• logit lens，nostalgebraist 2020 和 tuned lens，Belrose et al. 2023 读取训练好的模型的中间层，用模型自己的输出头，或每层学一个小映射。我们的探针在训练过程中做同样的事，结论一致：预测在最后几层才形成。图 16 把 logit lens 本身用在每个训练完的模型上，和它的探针或独立头放在一起。
• LayerSkip，Elhoushi et al. 2024 让前面的层也通过共享输出头预测，这样推理时可以提前停下。我们的共享头做法就是这种训练信号；在这里它让最后一层多付约 0.13 nats。
• LLAL，Jin et al. 2026 在超稀疏混合专家模型的层上加语言模型损失，防止专家失活。我们在 dense 模型上检验这个损失，而 dense 模型没有这个问题可修。
• 多 token 预测，Gloeckle et al. 2024 训练额外的头来预测后面几个 token，报告说在代码任务上的收益随模型变大而增加，小模型反而变差。DeepSeek-V3，DeepSeek-AI 2024 用一个串联的模块预测下下个词，我们用的就是这种形式。我们看这个额外的损失对每一层做了什么，在 130m 上和四个尺度上。

Acknowledgements致谢

The author is pleased to acknowledge that the work reported on in this post was substantially performed using the Princeton Research Computing resources at Princeton University. Princeton Research Computing is a consortium of groups including the Princeton Institute for Computational Science and Engineering (PICSciE) and Research Computing at Princeton University.

本文报告的工作主要使用普林斯顿大学 Princeton Research Computing 的计算资源完成。Princeton Research Computing 是由普林斯顿计算科学与工程研究所（PICSciE）和普林斯顿大学 Research Computing 等团队组成的联合体。

Experiment record实验记录

Citation

@misc{chai2026perlayer,
  title        = {Quantifying the Per-layer Contribution of LLMs},
  author       = {Chai, Wenhao},
  year         = {2026},
  howpublished = {Blog post},
  url          = {https://wenhaochai.com/exps/per-layer-contribution.html}
}