How Vocabulary Size Shapes Overfitting

Setup设置

The baseline is the MuonH Qwen3 recipe of the Marin speedrun, which covers 130m to 1.2B parameters, with the Marin tokenizer of 128K tokens. We cut the tokenizer to the first 64K, 32K, 16K or 8K tokens its BPE training learned. The smallest vocabulary is the 256 bytes, the tokenizer left with no merges at all. We train each model on the same text: all of fineweb-edu-10B, or a random part of it repeated 2.4 or 8 times. A smaller vocabulary cuts the same text into more tokens, so its runs take proportionally more steps. Each tokenizer draws its own random repeated part, with the same number of bytes. At 300m and 520m the warmup stays 1000 steps, so it is a smaller share of the longer runs. The input and output tables are untied, so a smaller vocabulary also has fewer parameters: the size names count the Marin tokenizer's model, and at 300m the 8K model has well under half of them. We keep the rest of the model fixed and do not control for this. We compare Paloma bits per byte, the loss in bits divided by the UTF-8 bytes of the text, so tokenizers of different sizes share one scale. The runs use this code.

基线是 Marin speedrun 的 MuonH Qwen3 配方，覆盖 130m 到 1.2B 参数，使用 128K 个词的 Marin tokenizer。我们把 tokenizer 截到它的 BPE 训练最先学到的 64K、32K、16K 或 8K 个词。最小的词表是 256 个字节，也就是一个合并都不剩的 tokenizer。我们让每个模型训练同样的文本：全部 fineweb-edu-10B，或其中随机的一部分重复 2.4 遍或 8 遍。词表越小，同样的文本被切成越多的词，所以它的 run 按比例多训练一些步。每种 tokenizer 各自随机抽取要重复的那一部分，字节数相同。在 300m 和 520m 上，预热固定为 1000 步，所以在步数更多的 run 里占比更小。输入表和输出表不共享，所以词表越小，参数也越少：规模的名字按 Marin tokenizer 的模型计算，在 300m 上 8K 模型的参数不到它的一半。我们保持模型其余部分不变，没有控制这一点。我们比较 Paloma 的每字节比特数，即以 bit 计的损失除以文本的 UTF-8 字节数，所以不同大小的词表在同一个尺度上比较。所有 run 使用这份代码。

--- baseline (simplified)+++ truncated tokenizer (simplified; the Code link has the real code) tok = load_tokenizer("marin-tokenizer")+# keep the first K tokens BPE learned and the merges among them+tok.vocab = {t: i for t, i in tok.vocab.items() if i < K}+tok.merges = [(a, b) for a, b in tok.merges if a + b in tok.vocab] tokens = tok.encode(text)-model = Qwen3(vocab_size=tok.vocab_size)-steps = recipe.steps+model = Qwen3(vocab_size=K)+# the same text is more tokens: scale the steps by the token ratio+steps = round(recipe.steps * len(tokens) / len(marin_tokens)) train(model, tokens, steps)+# compare tokenizers per byte of text, not per token+bits_per_byte = paloma_loss_nats * tokens_per_doc / (log(2) * utf8_bytes_per_doc)

--- 基线（简化）+++ 截短的 tokenizer（简化；真实代码见“代码”链接） tok = load_tokenizer("marin-tokenizer")+# 保留 BPE 最先学到的 K 个词，以及它们之间的 merge+tok.vocab = {t: i for t, i in tok.vocab.items() if i < K}+tok.merges = [(a, b) for a, b in tok.merges if a + b in tok.vocab] tokens = tok.encode(text)-model = Qwen3(vocab_size=tok.vocab_size)-steps = recipe.steps+model = Qwen3(vocab_size=K)+# 同样的文本变成更多的词：步数按词数比例放大+steps = round(recipe.steps * len(tokens) / len(marin_tokens)) train(model, tokens, steps)+# 按文本的字节比较不同 tokenizer，而不是按词+bits_per_byte = paloma_loss_nats * tokens_per_doc / (log(2) * utf8_bytes_per_doc)

Experiments实验

Answered已有结论Does a smaller vocabulary make repeated data hurt less?更小的词表能减轻重复数据的伤害吗？

Yes, when the data repeats many times: at 8 passes each halving of the vocabulary lowers the cost of repetition; the 64K, 32K and 16K truncations end about level with the Marin tokenizer and the 8K a little behind, while on fresh data they all trail it. The byte vocabulary loses almost nothing to 8 passes. At 2.4 passes repetition costs every tokenizer little, and the vocabulary size makes no clear difference.

是，在数据重复很多次时成立：重复 8 遍时，词表每减半一次，重复带来的损失就更小；64K、32K 和 16K 版本最终和 Marin tokenizer 大致持平，8K 版本稍差，而在不重复的数据上它们都落后。字节词表重复 8 遍几乎没有损失。重复 2.4 遍时，每种 tokenizer 受的损失都很小，词表大小没有明显影响。

Answered已有结论Does it hold at other model sizes?在其他规模的模型上也成立吗？

Yes. At 130m, 300m and 520m the 8K truncation loses less to 8 passes than the Marin tokenizer, and the byte vocabulary loses the least of all. The saving of the 8K truncation is largest at 130m and about the same at 300m and 520m, and at 520m the two tokenizers end level at 8 passes.

是。在 130m、300m 和 520m 上，8K 版本重复 8 遍的损失都比 Marin tokenizer 小，字节词表的损失最小。8K 版本省下的损失在 130m 上最大，在 300m 和 520m 上差不多；在 520m 上，两种 tokenizer 重复 8 遍后最终持平。

Answered已有结论Is the trend real?这个趋势是真的吗？

Yes for the gap between the Marin tokenizer and the 8K truncation: with a second seed, each one's cost moves far less than that gap. The first halving, from 128K to 64K at 300m, changes the cost by about as much as the two seeds differ, so that step alone is not resolved. Counting the bytes of each repeated part confirms that both tokenizers repeat the same amount of text.

对 Marin tokenizer 与 8K 版本之间的差距来说是：换第二个种子后，两者的代价变化都远小于这个差距。300m 上第一次减半（128K 到 64K）的变化和两个种子之间的差别差不多，单看这一步还不能确定。实际数出的每个重复部分的字节数也确认两种 tokenizer 重复的文本量相同。

Answered已有结论Do the input rows of rare tokens overfit?少见词的输入行会过拟合吗？

Yes, and with each of two seeds they carry about half of the gap to the 8K truncation. Without their own input rows, the rare tokens make the model worse on all data and about as good at 8 passes, where it still beats the 8K truncation. When the rare input rows learn for one pass and then stop, the cost falls by about two fifths of that gap. That model fits all of the data between the Marin tokenizer and the composed model. With each of two seeds it is slightly better than the Marin tokenizer at 8 passes, by less than the spread between the composed model's seeds.

会，而且在两个种子下，它们都大约占了与 8K 版本之间差距的一半。去掉少见词自己的输入行后，模型在全部数据上变差，在重复 8 遍时和原来差不多，仍比 8K 版本好。如果少见词的输入行先学习一遍，然后停止，代价减少这段差距的大约五分之二。这个模型在全部数据上的结果介于 Marin tokenizer 和组合模型之间。两个种子下，它重复 8 遍时都比 Marin tokenizer 稍好，差别小于组合模型两个种子之间的差异。

Answered已有结论Do the output rows of rare tokens overfit?少见词的输出行会过拟合吗？

Partly. The cost of the Marin tokenizer is far highest where the target is a rare token, and the 8K truncation's peak there is much smaller. Two causal tests failed: a model whose rare output rows are built from the 8K pieces' rows, or held at their random initial values, falls far behind the baseline on all of the data, so its cost of repetition says nothing about the rare rows alone. In a third test the rare output rows learn for one pass and then stop. This removes about half of the extra cost that the Marin tokenizer pays over the 8K truncation. The model also fits all of the data worse than the 8K truncation, so the large vocabulary's edge on new text depends on these rows too. When the rows keep learning at a tenth of their step instead, the model keeps more of its fit on all of the data and loses about as much to repetition as the Marin tokenizer, so the slower rows give no trade between the two.

部分会。目标词是少见词时，Marin tokenizer 的代价远高于其他位置，8K 版本在那里的峰小得多。两个因果检验都没有成功：少见词的输出行由 8K 片段行组成，或者固定在随机初值，模型在全部数据上都远远落后于基线，所以它重复数据的代价不能单独说明少见词的输出行。第三个检验让少见词的输出行先学习一遍，然后停止。这去掉了 Marin tokenizer 比 8K 版本多出的代价的大约一半。这个模型在全部数据上的拟合也比 8K 版本差，所以大词表在新文本上的优势也依赖这些行。如果这些行改为以十分之一的步长继续学习，模型在全部数据上的拟合保住得更多，受重复的伤害和 Marin tokenizer 差不多，所以放慢这些行换不来两者之间的折中。

We tried the causal test of Q4 on the output side: the model still scores all 128K tokens, but each token's output row is built from the rows of its 8K pieces, so the rare tokens have no output rows of their own. With the mean of the pieces' rows, a token can never score above its best piece, so the model cannot rate ' walking' above ' walk'. With the sum, ' walking' scores ' walk' plus 'ing', which has no such cap. Both models train far worse than the baseline on all of the data: after 900 steps the sum is still about 0.6 nats per token behind, about as far as the mean. A composed input table costs little by comparison. An output head composed this way cannot fit the data, so we stopped the test there. Holding the rare tokens' output rows at their random initial values fails the same way: after 1000 steps that model is about 0.9 nats per token behind the baseline. In the test below, the rare rows therefore learn during the first pass and stop only when the repetition starts.

我们在输出端也试了 Q4 那样的因果检验：模型仍然给全部 128K 个词打分，但每个词的输出行由它的 8K 片段行组成，所以少见词没有自己的输出行。取片段行的平均时，一个词的分数不会超过它得分最高的片段，所以模型不能认为 ' walking' 比 ' walk' 更可能。求和时，' walking' 的分数等于 ' walk' 加 'ing'，没有这个上限。两个模型在全部数据上都比基线差很多：训练 900 步后，求和版每个词仍然落后约 0.6 nats，和平均版差不多。相比之下，组合输入表的损失很小。这样组合出来的输出头拟合不了数据，所以我们停止了这个检验。把少见词的输出行固定在随机初值也一样失败：训练 1000 步后，这个模型每个词落后基线约 0.9 nats。所以在下面的检验里，少见词的输出行在第一遍正常学习，到重复开始时才停止。

Answered已有结论Does a window of fewer bytes cost less?覆盖字节更少的上下文窗口，代价更小吗？

Not for the Marin tokenizer. A window that holds about the bytes of the 8K truncation's costs as much as the full window. Not for the byte vocabulary either: a window four times longer leaves its cost far below the Marin tokenizer's at 130m and 300m, and raises it by no more than two seeds differ.

对 Marin tokenizer 来说不会。一个装下的字节和 8K 版本窗口差不多的窗口，代价和完整窗口一样。对字节词表也不会：在 130m 和 300m 上，长四倍的窗口下它的代价都仍远低于 Marin tokenizer，升高的幅度不超过两个种子之间的差别。

Answered已有结论Do more optimizer steps cost less?更多的优化步数，代价更小吗？

No. With a third more steps over the same text, the Marin tokenizer's cost of 8 passes barely moves.

不会。在同样的文本上多走三分之一的步数，Marin tokenizer 重复 8 遍的代价几乎不变。

Answered已有结论Does a smaller vocabulary still help when the model is overtrained?模型过度训练时，更小的词表还能减轻重复的伤害吗？

We could not tell. With one seed, the cost of 8 passes at 200 tokens per parameter falls as the vocabulary shrinks and goes below zero for the smallest vocabularies. A second seed moves the 8K truncation's cost by more than that whole range, so these runs cannot rank the vocabularies. The mean of the two seeds is lower for the 8K truncation than for the Marin tokenizer, by less than the gap between the 8K truncation's seeds.

分不清。只看一个种子，每参数 200 个 token 时重复 8 遍的代价随词表变小而下降，最小的几个词表低于零。第二个种子把 8K 版本的代价挪动的量，比所有词表之间的整个差距还大，所以这些 run 排不出词表的先后。两个种子平均后，8K 版本的代价比 Marin tokenizer 低；低出的量小于 8K 版本两个种子之间的差距。

Answered已有结论Is it only the parameter count?只是因为参数更少吗？

We could not tell. With rank-48 tables the Marin tokenizer loses far less to 8 passes, close to the 8K truncation. It also fits all of the data worse than the 8K truncation, so the smaller cost may come from fewer parameters or from a weaker fit, and this test cannot separate the two.

分不清。表换成秩 48 后，Marin tokenizer 重复 8 遍的代价小了很多，接近 8K 版本。它在全部数据上也比 8K 版本差，所以代价变小可能来自参数更少，也可能来自拟合更弱，这个实验分不开两者。

Answered已有结论Can a large vocabulary repeat data well?大词表也能抗住重复吗？

Partly. At 300m, replacing a rare token's input row at random by the mean of its 8K pieces' rows, in training only, lets the Marin tokenizer end ahead at 8 passes of both the unchanged Marin tokenizer and the 8K truncation, with each of three seeds and by more than seeds differ. It still loses about twice as much to repetition as the 8K truncation, and it fits all of the data a little worse than the unchanged Marin tokenizer. Replacing only in the repeated passes removes that loss on all of the data and keeps the lead at 8 passes, with two seeds. With a new order of the repeated part in every pass the lead grows further, with each of three seeds.

部分可以。在 300m 上，只在训练时把少见词的输入行随机换成其 8K 片段行的平均，Marin tokenizer 重复 8 遍时同时超过不做改动的 Marin tokenizer 和 8K 版本，三个种子都是如此，领先大于种子之间的差别。它受重复的伤害仍是 8K 版本的两倍左右，在全部数据上也比不做改动时略差。只在重复的几遍里替换，就没有了全部数据上的这点损失，重复 8 遍的领先仍在，两个种子都是如此。再加上每遍重新打乱，领先还会变大，三个种子都是如此。

Takeaways核心结论

Related work相关工作

• Kharitonov, Baroni and Hupkes 2021 find that a larger BPE vocabulary makes Transformers memorize their training data more, even at a fixed number of parameters, and suggest the shorter sequences as the cause. We measure the cost of repeated data on held-out text, which their memorization tests leave open.
• Muennighoff et al. 2023 find that up to about four epochs of repeated data are worth nearly as much as fresh data, and that the value of repetition falls after that. Our 2.4-pass runs cost every tokenizer little, and the vocabulary matters at 8 passes.
• Chai and Xiong 2026 price a repeated token against one epoch and against fresh data, and find that lower-entropy sources degrade faster under repetition. They keep the tokenizer fixed; we vary it.
• Tao et al. 2024 find that larger models deserve larger vocabularies when the data does not repeat. On all of the data the Marin tokenizer ends best among the BPE vocabularies at every size we ran, so our runs do not test how the best vocabulary moves with size; repetition changes the picture.
• Sampled Softmax with Over-Tokenized Transformer is where this question came up: large n-gram input tables overfit repeated data, and the vocabulary is the embedding table of the plain model.

• Kharitonov、Baroni 和 Hupkes 2021 发现，BPE 词表越大，Transformer 越会记住训练数据，即使参数量固定也一样，并推测原因是序列变短。我们测的是重复数据在 held-out 文本上的代价，这是他们的记忆测试没有回答的。
• Muennighoff et al. 2023 发现，重复数据在大约四个 epoch 以内几乎和新数据一样有用，之后价值下降。我们重复 2.4 遍的 run 对每种 tokenizer 的代价都很小，词表大小在 8 遍时才起作用。
• Chai 和 Xiong 2026 把重复的 token 和一个 epoch、和新数据分别比较定价，发现低熵的数据源在重复下退化得更快。他们固定 tokenizer；我们改变它。
• Tao et al. 2024 发现，在数据不重复时，越大的模型适合越大的词表。在全部数据上，我们跑过的每个规模中 Marin tokenizer 都是 BPE 词表里最好的，所以我们的 run 没有检验最优词表如何随规模变化；数据重复会改变这个结论。
• Sampled Softmax with Over-Tokenized Transformer 是这个问题的来源：大的 n-gram 输入表会在重复数据上过拟合，而词表就是普通模型的 embedding 表。

Acknowledgements致谢

The author is pleased to acknowledge that the work reported on in this post was substantially performed using the Princeton Research Computing resources at Princeton University. Princeton Research Computing is a consortium of groups including the Princeton Institute for Computational Science and Engineering (PICSciE) and Research Computing at Princeton University.

本文报告的工作主要使用普林斯顿大学 Princeton Research Computing 的计算资源完成。Princeton Research Computing 是由普林斯顿计算科学与工程研究所（PICSciE）和普林斯顿大学 Research Computing 等团队组成的联合体。

Experiment record实验记录

Citation

@misc{chai2026vocab,
  title        = {How Vocabulary Size Shapes Overfitting},
  author       = {Chai, Wenhao},
  year         = {2026},
  howpublished = {Blog post},
  url          = {https://wenhaochai.com/exps/vocab-overfitting.html}
}