Experiments›Pretraining on Text Written by a Frontier LLM
Nov 14, 2026

Pretraining on Text Written by a Frontier LLM

By Wenhao Chai作者 Wenhao Chai
[Cover animation: the page's main figure drawn in motion, once its runs finish]【封面动画:主图的 run 跑完后,把主图动态画出来】

Setup设置

The baseline is the 130m model of the MuonH Qwen3 recipe of the Marin speedrun, trained on 2.6B tokens of fineweb-edu-10B with the 128K Marin tokenizer. Every setting of the recipe stays fixed except the text the model reads, which a frontier LLM writes from three random words or from a fineweb-edu document. A run may mix it with fineweb-edu or read it more than once, as long as it reads 2.6B tokens in all; pilot runs read a tenth as many. We compare Paloma loss, the cross-entropy on Paloma's 16 sources of held-out natural text averaged over the sources, and we never read those sources while writing the data. The runs use this code.

基线是 Marin speedrun 的 MuonH Qwen3 配方中的 130m 模型,用 128K 个词的 Marin tokenizer 在 fineweb-edu-10B 上训练 2.6B token。配方的所有设置都不变,只换模型读的文本:文本由一个前沿 LLM 写,出发点是三个随机词或一篇 fineweb-edu 文档。一个 run 可以把这些文本和 fineweb-edu 混合,也可以重复读,只要一共读 2.6B token;先导 run 只读十分之一。我们比较 Paloma 损失,即 Paloma 16 个来源的留出自然文本上的交叉熵,对来源取平均;写数据时,我们从不读这些来源的文本。所有 run 使用这份代码。

Details详细说明

The 130m model has 6 layers of width 512, an MLP width of 1792 and 8 heads: 22.8M parameters in the layers, 65.7M in the embedding and 65.7M in the output head. It trains for 4959 steps of 128 sequences of 4096 tokens with MuonH, whose learning rate of 0.02 holds for the first fifth of training and then falls linearly to zero, at data seed 42. A pilot run keeps the recipe and ends the schedule at 496 steps, so its learning rate falls over its own last four fifths. The LLM writes 15 to 25 texts per call, separated by a marker line. Every prompt asks for plain text exactly as it would appear on the page and names no Paloma source or style. Paloma loss is the mean of the 16 per-source cross-entropies, in nats per token of the Marin tokenizer. Eight runs of the baseline on older code end at a Paloma loss of 4.1865, with a standard deviation of 0.0058 between seeds.

130m 模型有 6 层,宽 512,MLP 宽 1792,8 个头:各层共 22.8M 参数,embedding 65.7M,输出头 65.7M。它用 MuonH 训练 4959 步,每步 128 条长 4096 个 token 的序列;学习率 0.02,在训练的前五分之一保持不变,之后线性降到零;数据种子 42。先导 run 保持配方不变,把调度在第 496 步结束,所以学习率在它自己的后五分之四里下降。这个 LLM 每次调用写 15 到 25 篇文本,篇与篇之间用一行标记隔开。每个提示都要求写成纯文本,和它在页面上的样子完全一样,不提 Paloma 的任何来源或文体。Paloma 损失是 16 个来源各自交叉熵的平均,单位是 Marin tokenizer 每个 token 的 nats。基线在旧代码上跑过 8 次,Paloma 损失为 4.1865,种子之间的标准差为 0.0058。

Experiments实验

Testing检验中How far from natural text is text written from scratch?从零写的文本离自然文本有多远?

The LLM writes each text from three random words and a target length drawn from the lengths of fineweb-edu documents, so no natural document enters the prompt.

LLM 写每篇文本时,只拿到三个随机词和一个目标长度;目标长度按 fineweb-edu 文档的长度分布抽取,所以提示里没有任何自然文档。

Testing检验中Does a natural document in the prompt close the gap?提示里放一篇自然文档,能缩小差距吗?

The generator continues the first 60 words of a fineweb-edu document, or carries a whole document's content into another form of its choosing; training reads only what the generator wrote.

生成器续写一篇 fineweb-edu 文档的前 60 个词,或者把整篇文档的内容换成它自己选的另一种形式来写;训练只读生成器写出的部分。

Testing检验中Is the gap in the format or in the rare tokens?差距在格式上,还是在罕见 token 上?

Text written from scratch already holds more rare tokens than fineweb-edu, while all three arms separate paragraphs with a blank line that fineweb-edu never has. A twin of each arm reads the same text with one newline between paragraphs; another arm writes new text from seed words among the tokenizer's rarest entries.

从零写的文本里,罕见 token 已经比 fineweb-edu 多;三个臂的段落之间却都空一行,而 fineweb-edu 从不空行。每个臂都有一个对照臂,读同样的文本,段落之间只留一个换行;另一个臂用 tokenizer 里最罕见的词做种子词,重新写文本。

Testing检验中Which data gives the lowest Paloma loss?哪种数据的 Paloma 损失最低?

Mixing and repetition count as data choices, so every setup with its passes and its share of fineweb-edu appears here.

混合和重复都算数据的选择,所以每种数据连同它的遍数和 fineweb-edu 所占的比例,都放在这里比较。

Testing检验中Why does mixing help?混合为什么有效?

Half fineweb-edu and half text written from scratch scores below fineweb-edu alone, and almost all of the gain sits in code and RedPajama, two sources full of blank lines and indentation. Five arms take the mix apart: the same mix with the synthetic text's blank lines collapsed to one newline, smaller shares of synthetic text, and the other two kinds of synthetic text in place of text written from scratch.

fineweb-edu 与从零写的文本各占一半,Paloma 损失比只用 fineweb-edu 低;提升几乎都在代码和 RedPajama 上,这两个来源里空行和缩进很多。五个臂把混合拆开来看:同样的混合,但合成文本的空行都合成一个换行;合成文本占更小的比例;用另外两种合成文本代替从零写的文本。

Takeaways核心结论

Acknowledgements致谢

The author is pleased to acknowledge that the work reported on in this post was substantially performed using the Princeton Research Computing resources at Princeton University. Princeton Research Computing is a consortium of groups including the Princeton Institute for Computational Science and Engineering (PICSciE) and Research Computing at Princeton University.

本文报告的工作主要使用普林斯顿大学 Princeton Research Computing 的计算资源完成。Princeton Research Computing 是由普林斯顿计算科学与工程研究所(PICSciE)和普林斯顿大学 Research Computing 等团队组成的联合体。

Experiment record实验记录

Citation

@misc{chai2026synthpretrain,
  title        = {Pretraining on Text Written by a Frontier LLM},
  author       = {Chai, Wenhao},
  year         = {2026},
  howpublished = {Blog post},
  url          = {https://wenhaochai.com/exps/synth-pretrain.html}
}
Experiments Back to top回到顶部