Blog›How to Scale Text-to-image Models: a Handbook

Nov 13, 2026

How to Scale Text-to-image Models: a Handbook

By Wenhao Chai作者 Wenhao Chai

This post records how we built scaling laws and a model ladder for generative models on GPIC (Giant Permissive Image Corpus, https://arxiv.org/abs/2605.30341v1), 100M text-image pairs. The goals are 1) to give generative modeling a complete research baseline after ImageNet, and 2) to bring the science of LLM pretraining into visual generation. The code and raw data of every experiment in this post are linked from the card below.

本blog记录了在GPIC（Giant Permissive Image Corpus）（https://arxiv.org/abs/2605.30341v1）100M text-image pairs上进行generative models scaling law和ladder搭建的过程。目的是1）为generative models建立一个imagenet之后的完整research baseline，2）把LLM pretraining里的science引入vision generation领域。本文涉及到的所有实验数据点的代码和实验原始数据在卡片链接中。

Experiment page · since Nov 1, 2026实验页面 · 2026 年 11 月 1 日起 Generating Distributions

ImageNet should retire. It has only 1.28M images, each with a class label. This has two drawbacks. 1) The data are extremely scarce: almost every paper overfits over tens to hundreds of epochs, while LLM pretraining research usually trains for a single epoch. 2) The conditioning signal is very weak, only a class label, which indirectly encourages abusing CFG to chase scores. In practice we use generative models for text-to-image or video generation. People in arxiv.org/abs/2606.24888 already found that there is no strong signal that the techniques we use on ImageNet transfer to a real text-to-image setup.

imagenet必须要退役了，imagenet has only 1.28M with class label。这有两点劣势，1）数据极其稀少，几乎所有paper都在数十上百epoch下过拟合，而LLM的pretrain research通常只会训练1个epoch，2）条件信号非常之弱，仅仅只是一个类别标签，间接导致了CFG的滥用刷分的行为。实际情况下，我们使用generative models都是for text-to-image or video generation。people in arxiv.org/abs/2606.24888 already found that there is no strong signal that the tech we use in imagenet could be transferred to real text-to-image setup。

Choosing the metric选择指标

To build a scaling law and a ladder, we must first decide what the x and y axes are. For x we roughly take compute; one could take data instead when data is the bound, but this post starts from the classic compute-efficiency view. For y, unlike LLMs, generative models have nothing as good as bits per byte, a quantity that is the training objective itself and is comparable across recipes. We usually use distances between feature distributions such as FID or FD-DINOv2¹, plus other auxiliary metrics, because they need only the generated samples and not the algorithm; they can compare GANs and few-step diffusion models directly, for example. However, we find that FD correlates extremely strongly with the held-out loss, and even with the training loss (Figure 1). Compared with FD, the held-out loss also varies less across random seeds (the pink dots in Figure 2), and when comparing recipes it is more robust and separates real differences from noise (the teal dots in Figure 2). We therefore use the held-out loss as the main metric⁶ and report the other metrics for reference.

为了有一个scaling law和ladder，我们必须找到x和y轴分别是什么，我们大致把x轴选择为compute，当然也可以选择是data如果考虑data bound，但是本文先从传统的compute-efficiency视角来考虑。而对于y轴，与LLM不同，generative models没有像bpb一样好的东西，本身就是优化目标，同时可以跨越不同的recipe比较。我们一般使用FID或者FD-DINOv2¹这样特征空间的分布差距metric和其他一些辅助metric，因为这只需要生成结果就可以进行计算，而并不要求具体的算法。例如可以直接用来对比GAN的结果，few-step diffusion models的结果等等。然而我们发现FD和held-out loss，甚至training loss间有极其强的相关性，见图1。并且相比于FD，held-out loss由于随机种子的波动更小，见图2的粉色点，在对比不同recipe时也更稳健，能和随机噪声区分开来，见图2的青色点。我们选择使用held-out loss作为主要的metric⁶，同时也报告其他metric作为参考。

I know you can't wait to see the scaling law, but trust me: choosing the right metric is the foundation of everything.

我知道大家迫不及待想看到scaling law的内容，但是相信我，选对指标是一切的基石。

Evaluation setup. The held-out loss is the training objective itself, on fixed data and fixed noise: 3,072 images of the GPIC validation set (center-cropped to 256 × 256) with their own captions, at 16 noise levels (chosen as described in the section on noise levels), with one fixed noise per image and level, the same for every model. For the generation metrics we generate 50,000 images from the 50,000 captions of the GPIC test set: 50 Euler steps, no CFG (only the conditional branch), and every prompt starts from the same noise (the official GPIC protocol). We compare them with the 50,000 test images: FD, PRDC and MMD on DINOv2 ViT-L/14 features, and FID on Inception features. Unless stated otherwise, every metric uses the raw weights, not the EMA.

评测设置。held-out loss 就是训练目标本身，只是数据和噪声都固定：GPIC 验证集的 3072 张图（中心裁剪到 256 × 256）和它们自己的 caption，16 个噪声水平（取法见「held-out loss 的噪声水平」一节），每张图在每个水平上的噪声固定，所有模型都用同一组。生成指标用 GPIC 测试集的 5 万条 caption 生成 5 万张图：Euler 采样 50 步，不用 CFG（只跑条件分支），所有 prompt 从同一个噪声出发（GPIC 官方协议）。生成的图和测试集的 5 万张原图比较：在 DINOv2 ViT-L/14 特征上算 FD、PRDC 和 MMD，在 Inception 特征上算 FID。没有特别说明时，所有指标都用原始权重，不用 EMA。

Evaluation metrics and the held-out loss评测指标与 held-out loss

Figure 1. 380 checkpoints; FD and PRDC: 50K samples.图 1　380 个检查点；FD 与 PRDC：5 万个样本

Held-out loss and FD-DINOv2held-out loss 与 FD-DINOv2

Figure 2. Seed repeats: one configuration (d9, 3.6e18 FLOPs, Muon learning rate 1.36e-2) trained with 4 seeds. Sweep groups: 60 sweeps, each at one compute budget with one setting changed. Faint dots lie within the seed spread. All raw weights. Range = maximum minus minimum over the 380 checkpoints of Figure 1.图 2　种子重复：同一配置（d9、3.6e18 FLOPs、Muon 学习率 1.36e-2）用 4 个种子训练；扫描组：60 组扫描，每组同一算力、只改一个设置；浅色点在种子波动之内；均为原始权重；变化范围 = 图 1 中 380 个检查点的最大值减最小值

Background: scaling laws背景：Scaling law

We first introduce the basic idea of a scaling law. A scaling law describes how the loss changes with the resources put in. Kaplan et al. (2020) found for language models that the loss is a power law in parameters, data and compute, close to a straight line on log-log axes. That line has two uses. First, a set of small models predicts the loss of a large model before the large model is trained. Second, for a given compute budget it says how large the model should be and how much data it should see. Hoffmann et al. (2022) answered the second question with iso-FLOP curves (Figure 3): fix the compute, train a set of models of different sizes, and the one with the lowest loss is the optimal size for that budget. Finding the optimum at several budgets gives a fit of how the optimal size grows with compute. Their conclusion: parameters and training tokens should grow in equal proportion. Liang et al. (2024) ran the same experiment on diffusion transformers and found that the pretraining loss is also a power law in compute, with the same trend as FID (Figure 4). Chickering et al. (2026) (Abra) studied the question at larger compute: MMDiT text-to-image models from 60M to 2B parameters, 1e19 to 1e22 FLOPs, also fitting a held-out loss. They argue that different generation metrics give different optimal allocations: FID and KID favour more data, CLIPScore and CMMD more parameters.

我们先介绍一下scaling law的基本概念。Scaling law 描述的是 loss 怎样随投入的资源变化。Kaplan et al. (2020) 在语言模型上发现，loss 与参数量、数据量和算力之间都是幂律关系，在 log-log 坐标下近似一条直线。这条直线有两个用处。第一，用一组小模型就能预测大模型的 loss，不用先把大模型训出来。第二，给定一份算力，可以算出模型该多大、数据该多少。Hoffmann et al. (2022) 用 iso-FLOP 曲线回答了第二个问题（图 3）。做法是固定算力，训练一组大小不同的模型，loss 最低的那个就是这份算力下的最优大小。在几份算力上各找到最优点，就能拟合出最优大小怎样随算力增长。他们的结论是，参数量和训练 token 数应该等比例增长。Liang et al. (2024) 在 diffusion transformer 上做了同样的实验，发现 pretraining loss 与算力之间也是幂律，而且 loss 的趋势和 FID 一致（图 4）。Chickering et al. (2026)（Abra）在更大的算力上研究了这个问题。他们用 MMDiT 结构的文生图模型，参数从 6000 万到 20 亿，算力从 1e19 到 1e22 FLOPs，拟合的也是留出 loss。他们认为不同的生成指标给出不同的最优配比：FID 和 KID 偏向多给数据，CLIPScore 和 CMMD 偏向多给参数。

Ge et al. (2026) (Chimera) applied the same method to a hybrid architecture in latent space (linear attention plus sparse MoE). Their data do not repeat, so they fit the training loss directly, and N counts the activated parameters each token actually uses. On 256 px images, all three estimates give a near-even split: N ∝ C^0.48 to C^0.52. Video leans slightly toward parameters: N ∝ C^0.53 to C^0.56.

Ge et al. (2026)（Chimera）把同样的做法用到了 latent 空间的混合结构上（线性注意力加稀疏 MoE）。他们的数据不重复，所以直接拟合训练 loss；N 取每个 token 实际用到的激活参数。在 256 px 图像上，三种估计都给出接近对半的配比：N ∝ C^0.48 到 C^0.52。视频稍微偏向参数：N ∝ C^0.53 到 C^0.56。

Chinchilla's iso-FLOP curves — Figure 3. Source: Hoffmann et al. (2022), Figure 3. Left: loss against parameters at each compute budget; middle and right: optimal parameters and training tokens against compute; the green line marks Gopher's compute.图 3　图片来源：Hoffmann et al. (2022) 图 3。左：每种算力下 loss 随参数量的变化；中、右：最优参数量和训练 token 数随算力的变化，绿线标出 Gopher 的算力。

iso-FLOP curves of diffusion transformers — Figure 4. Source: Liang et al. (2024), Figure 1. (a) Loss against parameters at each compute budget; the purple dots are the minima of the parabolas. (b)(c) Optimal parameters and training tokens against compute; the red dot is the prediction at 1.5e21 FLOPs.图 4　图片来源：Liang et al. (2024) 图 1。(a) 每种算力下 loss 随参数量的变化，紫点是每条抛物线的最低点；(b)(c) 最优参数量和训练 token 数随算力的变化，红点是 1.5e21 FLOPs 下的预测。

TPP (tokens per parameter)TPP（token per parameter）

The core result of a scaling law is the relation between the number of model parameters and the number of tokens. Liang et al. (2024) consider only non-repeated data; by their results the optimal TPP is 350. The optimum of Chickering et al. (2026) is at 200, and they find diffusion models tolerate data repetition better than language models. This post considers both unique and total tokens and allows moderate repetition. For the concrete setup, Table 1 lists how different works count the parameters N, the data D and the compute C. We count image and text parameters separately in more detail, and count both unique and total tokens. You can also see clearly in the figure that the held-out loss gives steadier fits than FD.

scaling law的核心结论是model parameter和token number的关系。Liang et al. (2024) 只考虑了数据不重复的情况，根据论文中的结论，最优TPP应该为350。而 Chickering et al. (2026)的最优点在200，并且发现扩散模型比语言模型更能承受数据重复。本文将同时考虑unique token和total token，考虑适当的数据重复。在具体setup上，表1列出了不同统计方式对于参数N，数据量D，算力C。本文将更加细致的分别考虑图像和文本的参数，以及对于token同时考虑unique token和总token。在图中你也可以明显看到held-out loss相比FD的拟合更加稳定。

How N, D and compute are countedN、D 和算力的口径

	This post	Liang et al. (2024)	Abra (2026)	Chimera (2026)	LLM
Model	pixel-space JiT	latent DiT	latent MMDiT	latent hybrid backbone (linear attention + MoE)	Transformer
N	image parameters, excluding conditioning parameters, adaLN and embeddings	excluding embeddings²; text and image share parameters	all parameters, including embeddings and the text stream, not separated	activated parameters: the parameters each token actually uses	Kaplan excludes embeddings, Chinchilla includes them
D	image tokens	image + text tokens	image tokens	visual latent tokens	text tokens
Compute C	6ND	actual FLOPs	actual FLOPs	6ND	6ND
Optimal TPP	50	about 350³	165⁴	about 90⁵	about 20

	本文	Liang et al. (2024)	Abra (2026)	Chimera (2026)	LLM
模型	像素空间 JiT	latent DiT	latent MMDiT	latent 混合骨干（线性注意力 + MoE）	Transformer
N	图像参数，不含 conditioning参数，adaLN 和 embedding	不含 embedding²；文本和图像共用参数	全部参数，含 embedding 和文本流参数，不分开	激活参数：每个 token 实际用到的参数	Kaplan 不含 embedding，Chinchilla 含
D	图像 token	图像 + 文本 token	图像 token	视觉 latent token	文本 token
算力 C	6ND	实际 FLOPs	实际 FLOPs	6ND	6ND
最优 TPP	50	约 350³	165⁴	约 90⁵	约 20

Table 1.表 1

iso-FLOP fits by recipe and metric各配方和指标下的 iso-FLOP 拟合

Figure 5. Panel titles give the optimizer and the learning-rate cooldown. Large panels: Muon with a 50% cooldown, the main recipe of this post (GPIC Full data, linear cooldown over the last 50% of steps, learning rate following the rule with compute, raw weights); left held-out loss, right FD; finished runs only. Small panels: Muon with a 20% cooldown (the best of 3 learning rates per size, raw weights), and three earlier recipes, Muon and AdamW on GPIC Full and AdamW on the GPIC Lite subset, all at a constant learning rate without cooldown, with the held-out loss of the raw weights and the FD of the EMA weights. Each curve is a parabola in log N at one budget; the pink numbers are the TPP* at the minimum (D = image tokens seen, N = image parameters).图 5　标题里写的是优化器和学习率的冷却方式。大图：Muon，50% 冷却，即本文的主配方（GPIC Full 数据，最后 50% 的步数线性冷却，学习率按规则随算力变化，原始权重），左为 held-out loss，右为 FD，只画已经跑完的 run。小图：Muon，20% 冷却（每个大小取 3 个学习率中最好的，原始权重）；以及三个旧配方：Muon 和 AdamW 在 GPIC Full 数据上、AdamW 在 GPIC Lite 子集上，都用恒定学习率、不冷却，held-out loss 用原始权重，FD 用 EMA 权重。每条曲线是一个算力下 log N 上的抛物线；粉红数字是最低点处的 TPP*（D 为看过的图像 token，N 为图像参数）。

Our Scaling LawOur Scaling Law

This section answers one question: for a given compute budget, how large should the model and the data be? Following the approach described by Weng (2026), we fit one formula L(N, D) to the 34 runs in the large panel of Figure 5 and solve it directly for the optimum. N is the image parameters, D the image tokens seen, and compute is C = 6ND.

这一节回答一个问题：给定算力，模型和数据各应该多大。我们按 Weng (2026) 介绍的做法，用一个公式 L(N, D) 拟合图 5 大图的 34 个 run，再从这个公式直接解出最优点。N 是图像参数，D 是看过的图像 token，算力按 C = 6ND 计算。

L(N,D)=E+(AN^α+BD^β)^k

Compute C算力 C

Optimal image parameters N*最优图像参数 N*

Optimal image tokens D*最优图像 token D*

TPP

held-out lossheld-out loss

With k = 1 this is the Chinchilla formula. Our data need k = 0.23, the same as the Skaling of Videau et al. (2026). Without k, the formula over-predicts the loss at large compute (fitted on the runs up to 1.4e19 FLOPs, it is 1.1% high at 1.1e20), and the interval of the optimum is wide. k changes only the level of the loss, not the location of the optimum, so the formulas for N* and D* are the same as Chinchilla's.

k = 1 时就是 Chinchilla 的公式。我们的数据需要 k = 0.23，和 Videau et al. (2026) 的 Skaling 相同。不加 k，公式对大算力的 loss 预测偏高（只用 1.4e19 FLOPs 以下的 run 拟合，对 1.1e20 高 1.1%），最优点的区间也很宽。k 只改变 loss 的高低，不改变最优点的位置，所以 N* 和 D* 的公式和 Chinchilla 的一样。

Constants of the formula公式的常数

Constant	Value [90% interval]
E	0.0489 [0.0451, 0.0513]
A	2.33e-4
B	5.01e-3
α	0.53 [0.50, 0.57]
β	0.57 [0.54, 0.60]
k	0.23 [0.19, 0.27]

常数	值 [90% 区间]
E	0.0489 [0.0451, 0.0513]
A	2.33e-4
B	5.01e-3
α	0.53 [0.50, 0.57]
β	0.57 [0.54, 0.60]
k	0.23 [0.19, 0.27]

Table 2.表 2

The formula gives N* ∝ C^0.52 [0.51, 0.53] and D* ∝ C^0.48 (Figure 6). The two exponents are close, so the optimal TPP falls only slowly with compute: from 55 (3.6e18 FLOPs) to 48 (1.1e20 FLOPs), so Table 1 uses 50. Extrapolated to 1e21 FLOPs, the optimum is about 2B image parameters and 85B image tokens (TPP about 44). The black dots in Figure 6 are the minima of the parabolas in Figure 5 and agree with the formula. The formula also predicts the loss: error 0.036%; the four runs at 2.3e20 FLOPs (d15, d16, d18, d20) were not in the fit, and the prediction and the measurement differ by less than 0.09%.

公式给出 N* ∝ C^0.52 [0.51, 0.53]，D* ∝ C^0.48（图 6）。两个指数接近，所以最优 TPP 随算力只慢慢下降：从 55（3.6e18 FLOPs）到 48（1.1e20 FLOPs），所以表 1 取 50。外推到 1e21 FLOPs，最优是约 20 亿图像参数、850 亿图像 token（TPP 约 44）。图 6 的黑点是图 5 抛物线的最低点，和公式一致。公式也能预测 loss：误差 0.036%；2.3e20 FLOPs 的四个 run（d15、d16、d18、d20）没有参与拟合，预测和实测相差不到 0.09%。

This allocation is almost the same as that of Ge et al. (2026). Their architecture, data, optimizer (AdamW) and space (latent) all differ from ours; their parametric fit on images gives N* ∝ C^0.516 and D* ∝ C^0.484. From their fitted formula, the optimal TPP is about 90⁵, less than twice ours; the two works count N differently (Table 1).

这个配比和 Ge et al. (2026) 的结果几乎相同。他们的结构、数据、优化器（AdamW）和空间（latent）都和我们不同，图像上的参数拟合给出 N* ∝ C^0.516、D* ∝ C^0.484。按他们的拟合式推算，最优 TPP 约 90⁵，是我们的不到 2 倍；两边 N 的口径不同（表 1）。

Optimal model size and data最优模型大小和数据量

Figure 6. Black dots: the minima of the parabolas in Figure 5. Solid lines: the optimum from the formula, with the 90% interval as a band (200 resamples of the runs); dashed lines: extrapolation.图 6　黑点：图 5 抛物线的最低点。实线：公式给出的最优点，色带是 90% 区间（重抽 run 200 次）；虚线：外推。

Final LadderFinal Ladder

The final ladder is the run with the lowest held-out loss at each budget in Figure 5. All of them use the recipe in Table 3, with TPP between 36 and 57, close to 50. At 2.3e20 FLOPs, d16 and d18 differ by only 0.03%, and d15 and d20 on either side are both higher, so the minimum at this budget lies between d16 and d18.

final ladder 就是图 5 里每档算力 held-out loss 最低的 run。它们都用表 3 的配方，TPP 在 36 到 57 之间，接近 50。2.3e20 FLOPs 时，d16 和 d18 只差 0.03%，两侧的 d15 和 d20 都更高，所以这一档的最低点在 d16 和 d18 之间。

The final ladder's recipefinal ladder 的配方

Setting	Value
Optimizer	Muon (matrix parameters) + AdamW (the rest)
Muon learning rate	1.36e-2 × (C / 3.6e18)^−0.5
AdamW learning rate	6.4e-4 × (all parameters / 109.4M)^−0.547
Weight decay	Muon: 5 / (Muon learning rate × steps); AdamW: 0
Batch	1024 images
Learning-rate schedule	warmup over the first 1% of steps, linear decay to 3e-5 over the last 50%
EMA	none
Data	GPIC Full, 99.98M images
Aspect ratio	the ladder's shapes: width = 128 × (image layers − 4)
Text layers	2 to 4, as in the ladder

设置	值
优化器	Muon（矩阵参数）+ AdamW（其余参数）
Muon 学习率	1.36e-2 ×（C / 3.6e18）^−0.5
AdamW 学习率	6.4e-4 ×（全部参数 / 1.094 亿）^−0.547
Weight decay	Muon：5 /（Muon 学习率 × 步数）；AdamW：0
Batch	1024 张图
学习率曲线	前 1% 的步数 warmup，最后 50% 的步数线性降到 3e-5
EMA	不用
数据	GPIC Full，9998 万张图
长宽比	ladder 的形状：宽 = 128 ×（图像层数 − 4）
文本层	2 到 4 个，同 ladder

Table 3.表 3

The final ladderfinal ladder

Size	d10	d11	d12	d13	d14	d16	d18
Compute (FLOPs)	3.6e18	7.1e18	1.4e19	2.9e19	5.7e19	1.1e20	2.3e20
Width	768	896	1024	1152	1280	1536	1792
Image layers	10	11	12	13	14	16	18
Text layers	2	2	3	3	3	4	4
Image parameters	106M	159M	226M	311M	413M	679M	1040M
All parameters	171M	252M	376M	507M	665M	1122M	1680M
Steps	21,248	28,800	40,192	59,136	89,600	108,800	143,872
Training images	21.8M	29.5M	41.2M	60.6M	91.8M	111M	147M
Muon learning rate	1.36e-2	9.62e-3	6.80e-3	4.81e-3	3.40e-3	2.40e-3	1.70e-3
AdamW learning rate	5.01e-4	4.06e-4	3.26e-4	2.77e-4	2.38e-4	1.79e-4	1.44e-4
Muon weight decay	0.0173	0.0181	0.0183	0.0176	0.0164	0.0191	0.0204
GPU-hours (H100)	3.8	6.8	11.7	22.5	40.4	76.5	150
held-out loss	0.06788	0.06707	0.06628	0.06558	0.06487	0.06416	0.06351
FD-DINOv2	949.8	828.5	673.3	534.8	443.5	373.5	325.3

大小	d10	d11	d12	d13	d14	d16	d18
算力（FLOPs）	3.6e18	7.1e18	1.4e19	2.9e19	5.7e19	1.1e20	2.3e20
宽度	768	896	1024	1152	1280	1536	1792
图像层	10	11	12	13	14	16	18
文本层	2	2	3	3	3	4	4
图像参数	106M	159M	226M	311M	413M	679M	1040M
全部参数	171M	252M	376M	507M	665M	1122M	1680M
步数	21,248	28,800	40,192	59,136	89,600	108,800	143,872
训练图片数	21.8M	29.5M	41.2M	60.6M	91.8M	111M	147M
Muon 学习率	1.36e-2	9.62e-3	6.80e-3	4.81e-3	3.40e-3	2.40e-3	1.70e-3
AdamW 学习率	5.01e-4	4.06e-4	3.26e-4	2.77e-4	2.38e-4	1.79e-4	1.44e-4
Muon weight decay	0.0173	0.0181	0.0183	0.0176	0.0164	0.0191	0.0204
GPU 小时（H100）	3.8	6.8	11.7	22.5	40.4	76.5	150
held-out loss	0.06788	0.06707	0.06628	0.06558	0.06487	0.06416	0.06351
FD-DINOv2	949.8	828.5	673.3	534.8	443.5	373.5	325.3

Table 4.表 4

HyperparametersHyperparameters

Each subsection below is one group of hyperparameters that we swept, with one figure each. The recipe in Table 3 comes from these sweeps.

下面每节是一组扫过的超参数，每节一张图。表 3 的配方来自这些扫描。

Removing the EMA去掉 EMA

Generative models are almost always evaluated with EMA weights; EDM2, for example, finds FID very sensitive to the length of the EMA (Karras et al., 2024). Language models almost always decay the learning rate at the end of training (Hägele et al., 2024). Both do the same thing: remove the noise in the weights at the end of training. The EMA averages the weights over time; learning-rate decay shrinks the steps so the weights settle on their own. We compared the two at equal compute (Figure 7), and the two metrics give different answers. On the held-out loss the cooldown is better at every budget, by 0.4% to 1.0%; on FD the EMA gets better with compute and is 7.5% lower at 1.1e20 FLOPs. This may be why the two fields have different habits: image generation selects models by FD, language modeling by loss. We use the held-out loss as the main metric, so we remove the EMA: over the last 50% of steps the learning rate falls linearly to 3e-5, and we evaluate the raw weights. The EMA has two more problems: its decay was set for 400K steps, so in short runs the EMA still holds part of the initial weights; and at equal compute different sizes run different numbers of steps, so their EMAs lag by different amounts.

生成模型几乎都评测 EMA 权重，例如 EDM2 发现 FID 对 EMA 的长度很敏感（Karras et al., 2024）；语言模型几乎都在训练末期衰减学习率（Hägele et al., 2024）。两者做的是同一件事：消掉训练末期权重里的噪声。EMA 在时间上平均权重；学习率衰减让步长变小，权重自己停下来。我们在同样的算力下比较了两种做法（图 7），两个指标给出不同的答案：held-out loss 上，冷却在每个算力下都更好，低 0.4% 到 1.0%；FD 上，算力越大 EMA 越好，1.1e20 FLOPs 时低 7.5%。这可能就是两个领域习惯不同的原因：图像生成用 FD 选模型，语言模型用 loss 选模型。我们用 held-out loss 作主指标，所以去掉 EMA：最后 50% 的步数里把学习率线性降到 3e-5，直接评测原始权重。EMA 还有两个问题：它的衰减是为 40 万步设的，短 run 的 EMA 里还留着初始权重；同一算力下不同大小的步数不同，EMA 的滞后也不同。

Removing the EMA去掉 EMA

Figure 7. The best value over sizes at each budget. Both recipes use Muon and GPIC Full; in the new recipe the learning rate falls with compute by the rule.图 7　每个算力下各个大小中最好的值。两个配方都用 Muon 和 GPIC Full；新配方的学习率按规则随算力下降。

Optimizer优化器

Sweep: Muon and AdamW train d9 to d12 on GPIC Full, 3.6e18 to 2.9e19 FLOPs. Result: at the same size and compute, AdamW's held-out loss is 0.7% to 2.1% higher and its FD 16% to 32% higher.

扫描：Muon 和 AdamW 训练 d9 到 d12，GPIC Full，3.6e18 到 2.9e19 FLOPs。结论：同样的大小和算力下，AdamW 的 held-out loss 高 0.7% 到 2.1%，FD 高 16% 到 32%。

Ge et al. (2026) report the opposite: Muon's loss is always worse than AdamW's and better only early in training at large learning rates, across architectures and sizes (their §5.7). Their explanation is that Adam's updates concentrate in a few directions, which suits the denoising objective better. We reach the opposite conclusion in pixel space, and the reason is not yet clear.

Ge et al. (2026) 的结果相反：他们报告 Muon 的 loss 一直比 AdamW 差，只在大学习率的训练初期更好，换结构、换大小都是这样（原文 §5.7）。他们的解释是 Adam 的更新集中在少数方向上，更适合去噪目标。我们在像素空间得到相反的结论，原因还不清楚。

Muon and AdamWMuon 和 AdamW

Figure 8. The lowest held-out loss among d9 to d12 at each budget. Both ladders train on GPIC Full at a constant learning rate, without cooldown.图 8　每个算力下 d9 到 d12 中最低的 held-out loss。两个 ladder 都在 GPIC Full 上用恒定学习率训练，不冷却。

Learning rate学习率

Sweep: d9 to d14 with the Muon learning rate from ×1/8 to ×4 of the rule, 3.6e18 to 1.4e19 FLOPs; d12, d14 and d16 at 5.7e19 FLOPs from 1.7e-3 to 1.36e-2. Result: the optimum is flat: at 7.1e18 and 1.4e19 FLOPs, ×0.5 and ×1 are within 0.06% of each other, and on d16 an 8× range of learning rates differs by only 0.13%; up to 1.4e19 FLOPs, ×2 is worse than the rule at every size. At 5.7e19 FLOPs, ×0.5 is 0.05% to 0.11% above the rule at d12, d14 and d16.

扫描：d9 到 d14，Muon 学习率取规则值的 ×1/8 到 ×4，3.6e18 到 1.4e19 FLOPs；d12、d14 和 d16 在 5.7e19 FLOPs 取 1.7e-3 到 1.36e-2。结论：最优点很平：7.1e18 和 1.4e19 FLOPs 时，×0.5 和 ×1 相差不到 0.06%，d16 上 8 倍的学习率范围只差 0.13%；1.4e19 FLOPs 以内，×2 在每个大小上都比规则值差。5.7e19 FLOPs 时，×0.5 在 d12、d14 和 d16 上都比规则值高 0.05% 到 0.11%。

Ge et al. (2026) quantified the cost of not adjusting the learning rate with size. With one learning rate shared by all sizes, their optimal learning rate differs 6× across widths, and the fitted exponent of N* moves from about 0.5 to 0.58 to 0.59, overestimating the optimal model. After scaling the learning rate by each module's width and depth (HeteroP), one base learning rate stays close to optimal over a 56× range of parameters.

Ge et al. (2026) 量化了学习率不随大小调整的代价。所有大小共用一个学习率时，他们的最优学习率在不同宽度上相差 6 倍，拟合出的 N* 指数从约 0.5 变成 0.58 到 0.59，最优模型被高估。按每个模块的宽度和深度缩放学习率（HeteroP）之后，同一个基础学习率在 56 倍的参数范围内都接近最优。

Learning rate学习率

Figure 9. Rule learning rate: 1.36e-2 × (C / 3.6e18)^−0.5, inspired by Marin. One panel per compute budget and one line per size; the y axis is the change relative to that line's rule learning rate (×1). 3.6e18 FLOPs uses no weight decay; 7.1e18 and 1.4e19 FLOPs use weight decay 5 / (learning rate × steps); these three panels cool down over the last 20% of steps. 5.7e19 FLOPs and the second d11 line at 1.4e19 FLOPs (×0.5, ×1, ×1.4) cool down over the last 50% of steps. Hover over a point for its learning rate and change.图 9　规则学习率：1.36e-2 ×（C / 3.6e18）^−0.5。inspired by Marin。每个算力一栏，每条线是一个大小，纵轴是相对该线规则学习率（×1）的变化。3.6e18 FLOPs 不加 weight decay；7.1e18 和 1.4e19 FLOPs 的 weight decay 为 5 /（学习率 × 步数）；这三栏都在最后 20% 的步数冷却。5.7e19 FLOPs 和 1.4e19 FLOPs 里 d11 的第二条线（×0.5、×1、×1.4）在最后 50% 的步数冷却。鼠标移到点上可以看学习率和变化。

Weight decayWeight decay

Sweep: d11 with Muon weight decay 0, 0.016, 0.05 and 0.1; 0.016 and 0.05 each at three learning rates. At d12 and 2.9e19 FLOPs, we replace the 5 in Table 3's decay = 5 / (learning rate × steps) with 2.5 and 10. Result: at learning rate 3.4e-3, decay 0.05 lowers the held-out loss by 0.40%; the larger the learning rate, the smaller the best decay. At d12, 5 is best: 2.5 raises the held-out loss by 0.10% and 10 by 0.05%. FD instead falls as the decay grows: 1.6% lower at 10 than at 5.

扫描：d11，Muon weight decay 取 0、0.016、0.05 和 0.1；0.016 和 0.05 各试三个学习率。d12 在 2.9e19 FLOPs 上把表 3 中 decay = 5 /（学习率 × 步数）的 5 换成 2.5 和 10。结论：学习率 3.4e-3 时，decay 0.05 让 held-out loss 低 0.40%；学习率越大，最好的 decay 越小。d12 上 5 最好：2.5 高 0.10%，10 高 0.05%；FD 则是 decay 越大越低，10 比 5 低 1.6%。

Weight decayWeight decay

Figure 10. Top: held-out loss; bottom: FD-DINOv2 (50K samples); both as the change from a reference run, each panel on its own vertical range. Left: d11, against learning rate 3.4e-3 without decay. Right: d12 at learning rate 4.81e-3, against Table 3's recipe (decay × learning rate × steps = 5).图 10　上：held-out loss；下：FD-DINOv2（5 万个样本）；都是相对参照 run 的变化，每个面板的纵轴范围不同。左：d11，参照是学习率 3.4e-3、不加 decay 的 run。右：d12，学习率 4.81e-3，参照是表 3 的配方（decay × 学习率 × 步数 = 5）。

Batch sizeBatch size

Sweep: at equal compute, batch 512 to 4096 for d9 and d14, 512 to 2048 for d12 at 2.9e19 FLOPs, and 1024 and 2048 for d16 at 5.7e19 FLOPs. Result: larger batches are worse; doubling the batch raises the held-out loss by up to 0.60%. For d12, going from 512 to 1024 changes almost nothing (0.005%), and from 1024 to 2048 raises it by 0.10%.

扫描：同一算力下，d9 和 d14 的 batch 取 512 到 4096，d12 在 2.9e19 FLOPs 取 512 到 2048，d16 在 5.7e19 FLOPs 取 1024 和 2048。结论：batch 越大越差，batch 加倍时 held-out loss 最多高 0.60%；d12 从 512 加到 1024 几乎不变（差 0.005%），从 1024 加到 2048 高 0.10%。

Batch sizeBatch size

Figure 11. Runs of one size use the same compute; the number of steps scales inversely with the batch.图 11　同一大小的 run 算力相同，步数随 batch 反比变化。

Cooldown冷却

Sweep: d11 with a linear cooldown over the last 20%, 50% or 90% of steps, plus a 1−√ cooldown over 90%, each at three learning rates. Result: 50% is the best, with a held-out loss 0.22% to 0.24% below 20%.

扫描：d11，在最后 20%、50% 或 90% 的步数里线性冷却，另加 90% 的 1−√ 冷却，各用三个学习率。结论：50% 最好，held-out loss 比 20% 低 0.22% 到 0.24%。

Cooldown冷却

Figure 12. d11, 7.1e18 FLOPs.图 12　d11，7.1e18 FLOPs。

Data repetition数据重复

Sweep: d12 (2.9e19 FLOPs) and d9 (1.4e19 FLOPs) with fewer unique images, each image seen 0.82 to 684 times; d10 on only 101K images. Result: up to about 50 passes the held-out loss does not change, by less than the seed noise; beyond that it gets worse, and sooner for larger models: d9 is 0.56% higher at 196 passes and 6.6% higher at 684; d12 is 0.18% higher at 59 passes and 2.3% higher at 206; d10 on only 101K images is 10% higher. Meanwhile FD falls (d9 by up to 9.5%, d12 by up to 18.5%), so FD does not show the cost of repeated data.

扫描：d12（2.9e19 FLOPs）和 d9（1.4e19 FLOPs）减少唯一图像，每张图看 0.82 到 684 遍；d10 只用 10.1 万张图。结论：约 50 遍以内 held-out loss 不变，差别小于种子噪声；再多就变差，模型越大越早：d9 在 196 遍时高 0.56%，684 遍时高 6.6%；d12 在 59 遍时高 0.18%，206 遍时高 2.3%；d10 只用 10.1 万张图时高 10%。这时 FD 反而下降（d9 最多低 9.5%，d12 最多低 18.5%），所以 FD 看不出重复数据的代价。

Data repetition数据重复

Figure 13. Runs of one size train the same number of steps and change only the number of unique images. Four d9 runs that differ only in the seed span 0.07%. d10 uses an earlier recipe (learning rate 3.4e-3, no weight decay, 20% cooldown) and trains on GPIC Full, GPIC Lite (10.01M images) or the first 80 shards (1.0M images); on only 101K images (436 passes) its held-out loss is 10% higher, outside the chart. Left: held-out loss; right: FD-DINOv2 (50K samples); both as the change from the run with the fewest passes.图 13　同一大小的 run 步数相同，只改唯一图像的数量。只差种子的四个 d9 run 之间差 0.07%。d10 用旧配方（学习率 3.4e-3，不加 weight decay，20% 冷却），分别只用 GPIC Full、GPIC Lite（1001 万张）和前 80 个分片（100 万张）训练；只用 10.1 万张图（436 遍）时 held-out loss 高 10%，不在图内。左：held-out loss；右：FD-DINOv2（5 万个样本）；都是相对遍数最少的 run 的变化。

Aspect ratio长宽比

Sweep: at about 110M, 250M and 670M parameters, 4 to 8 shapes each, from wide and shallow to narrow and deep, at equal compute. Aspect ratio = width (hidden dim) ÷ image layers: from 128 to 14 at 110M, from 320 to 5.6 at 250M and from 171 to 5.0 at 670M; the ladder's shapes are at 71, 81 and 91. Result: the best shape is deeper and narrower than the ladder's, and the minima of all three sweeps lie at aspect ratios of 28 to 34: 512 × 15 at 110M (ratio 34), 640 × 23 at 250M (28) and 896 × 32 at 670M (28), with held-out losses 0.10%, 0.24% and 0.43% below the ladder's. At 670M, 768 × 44 (17) is within 0.03% of the minimum. Deeper shapes get worse again: at an aspect ratio of 5 (512 × 102) the loss is level with the ladder's.

扫描：参数量固定在约 1.1 亿、2.5 亿和 6.7 亿，从宽而浅到窄而深各取 4 到 8 种形状，算力相同。长宽比 = 宽度（hidden dim）÷ 图像层数：1.1 亿参数时从 128 扫到 14，2.5 亿从 320 扫到 5.6，6.7 亿从 171 扫到 5.0；ladder 的形状是 71、81 和 91。结论：最好的形状比 ladder 更深更窄，三条扫描的最低点长宽比都在 28 到 34 之间：1.1 亿参数是 512 × 15（长宽比 34），2.5 亿是 640 × 23（28），6.7 亿是 896 × 32（28），held-out loss 比 ladder 低 0.10%、0.24% 和 0.43%。6.7 亿参数下 768 × 44（17）和最低点只差 0.03%。再深又变差：长宽比降到 5（512 × 102）时，和 ladder 持平。

Aspect ratio长宽比

Figure 14. Shapes of one size use the same compute: 1.5e18 FLOPs at about 110M parameters, 7.9e18 at about 250M and 5.4e19 at about 670M. The number under each line's minimum is its aspect ratio (width ÷ image layers); hover over a point for each shape's aspect ratio.图 14　同一参数量的形状用同一算力：约 1.1 亿参数 1.5e18 FLOPs，约 2.5 亿 7.9e18，约 6.7 亿 5.4e19。每条线最低点下方的数字是长宽比（宽度 ÷ 图像层数）；鼠标移到点上可以看每个形状的长宽比。

Image layers and text layers图像层和文本层

Sweep: at d11's width, image layers and text layers from 12+0 to 9+12; for d9, d11 and d14, only the number of text layers (1, 2, 4, 8). Result: the ladder's 11+2 is the best; changing only the text layers moves the held-out loss by at most 0.10%. At equal compute, d12 is best with 3 text layers (the ladder's value); 1, 6 and 12 are 0.10%, 0.06% and 0.16% higher. At d9 and 1.4e19 FLOPs, 1, 6 and 12 text layers are +0.09%, −0.05% and +0.03% against the ladder's 2, about the size of the difference between seeds (0.07%).

扫描：d11 的宽度下，图像层和文本层从 12+0 到 9+12；d9、d11 和 d14 只改文本层数（1、2、4、8）。结论：ladder 的 11+2 最好；只改文本层数时，held-out loss 最多变 0.10%。同算力下，d12 用 3 个文本层（ladder 的值）最好，1、6、12 个分别高 0.10%、0.06% 和 0.16%。d9 在 1.4e19 FLOPs 时，1、6、12 个文本层相对 ladder 的 2 个分别是 +0.09%、−0.05% 和 +0.03%，和种子之间的差别（0.07%）相当。

Number of text layers文本层数

Figure 15. Left: the image layers stay fixed and only the text layers change, so more text layers cost more compute (8 text layers add 8% to 12%). Middle: d9 (1.4e19 FLOPs, relative to 2 text layers) and d12 (2.9e19 FLOPs, relative to 3 text layers), steps adjusted to the number of text layers so the compute is equal. Right: d11's width, 7.1e18 FLOPs, relative to 11+2.图 15　左：图像层数不变，只改文本层数，所以文本层越多算力越多（8 个文本层多 8% 到 12%）。中：d9（1.4e19 FLOPs，相对 2 个文本层）和 d12（2.9e19 FLOPs，相对 3 个文本层），步数随文本层数调整，算力相同。右：d11 的宽度，7.1e18 FLOPs，相对 11+2。

InfraInfra

We first cut one step of the official code from 263 ms to 207 ms (−21%), then added optimizations to the ladder's code one at a time. The MFU of the whole ladder (weighted by each size's GPU-hours) rose from 22.9% to 37.3%; the large models gained the most, d18 from 28.1% to 42.2%.

我们先把官方代码的一步从 263 ms 降到 207 ms（−21%），再在 ladder 的代码上逐项加优化。整个 ladder 的 MFU（按各个大小的 GPU 小时加权）从 22.9% 升到 37.3%；大模型升得最多，d18 从 28.1% 升到 42.2%。

How MFU evolvedMFU 的进化

Figure 16. 8 H100s. MFU = training FLOPs ÷ (GPU time × 989.4 TFLOP/s, the H100's bf16 peak). Added in order: FlashAttention-3, FSDP2, resizing on the GPU and uint8 crops; fused Triton kernels (residual + RMSNorm + adaLN, q/k RMSNorm + RoPE); batch 1024 without gradient accumulation, Muon, one caption-length bucket shared by all GPUs; a fused Newton–Schulz iteration and patchify by reshape.图 16　8 张 H100。MFU = 训练 FLOPs ÷（GPU 时间 × 989.4 TFLOP/s，H100 的 bf16 峰值）。依次加入：FlashAttention-3、FSDP2、在 GPU 上缩放和 uint8 裁剪；融合的 Triton 内核（残差 + RMSNorm + adaLN，q/k RMSNorm + RoPE）；batch 1024 不做梯度累积、Muon、所有 GPU 共用一个 caption 长度桶；融合的 Newton–Schulz 迭代、patchify 改用 reshape。

MFU per size每个大小的 MFU

Figure 17. Batch 1024, Muon, shared caption buckets, 8 H100s; without the last step (fused Newton–Schulz, patchify).图 17　batch 1024、Muon、共享 caption 桶，8 张 H100；不含最后一步（融合 Newton–Schulz、patchify）。

Noise levels of the held-out lossheld-out loss 的噪声水平

The held-out loss is the mean loss over 16 noise levels, the quantiles of the training t distribution (Figure 18; t from 0.09 to 0.67; t = 0 is pure noise, t = 1 the clean image). Figure 19 shows the final ladder's loss at 16 uniform levels: the middle levels are the easiest and the two ends the hardest. As the model grows, the cleaner the level, the more its loss falls: from d10 to d16 it falls 1.8% at t = 0.03 and 29% at t = 0.97.

held-out loss 是 16 个噪声水平上的平均 loss，这 16 个水平取训练时 t 分布的分位点（图 18；t 从 0.09 到 0.67；t = 0 是纯噪声，t = 1 是原图）。图 19 是 final ladder 在 16 个均匀水平上的 loss：中间的水平最容易，两端最难。模型变大时，越干净的水平 loss 降得越多：从 d10 到 d16，t = 0.03 只降 1.8%，t = 0.97 降 29%。

Sampling distribution of the noise level噪声水平的采样分布

Figure 18. In training, t is drawn from logit-normal(−0.8, 0.8); only 3% of samples have t above 0.67. The default levels are its 16 quantiles; the uniform levels run from 0.03 to 0.97. Grey line: SD3's logit-normal(0, 1).图 18　训练时 t 从 logit-normal(−0.8, 0.8) 采样，只有 3% 的样本 t 大于 0.67。默认水平是这个分布的 16 个分位点；均匀水平从 0.03 到 0.97。灰线：SD3 用的 logit-normal(0, 1)。

Loss at each noise level每个噪声水平上的 loss

Figure 19. The final ladder's 6 runs from 3.6e18 to 1.1e20 FLOPs; 16 uniform levels (t from 0.03 to 0.97), 3072 unseen images. Left: the loss (log axis); right: the change from d10.图 19　final ladder 中 3.6e18 到 1.1e20 FLOPs 的 6 个 run；16 个均匀的水平（t 从 0.03 到 0.97），3072 张没见过的图。左：loss（对数坐标）；右：相对 d10 的变化。

Another choice changes little. Under every weighting we tried, the exponent of N* lies between 0.47 and 0.54 (0.52 in this post); with SD3's distribution, only 0.9% of run pairs at equal compute change order. What changes is the TPP: 42 to 70 at 1.1e20 FLOPs (48 in this post). At single levels, the cleaner the level, the larger the optimal model: from t = 0.15 to 0.65 the exponent rises from 0.48 to 0.56 (Figure 20).

换一种取法，结论基本不变。我们试过的各种加权下，N* 的指数在 0.47 到 0.54 之间（本文 0.52）；换成 SD3 的分布时，同一算力下只有 0.9% 的 run 对排序翻转。会变的是 TPP：1.1e20 FLOPs 时在 42 到 70 之间（本文 48）。单个水平上，水平越干净，最优模型越大：t 从 0.15 到 0.65，指数从 0.48 升到 0.56（图 20）。

Scaling laws at each noise level每个噪声水平上的 scaling law

Figure 20. Each point fits our formula with the held-out loss of a single noise level (34 main ladder runs). Left: the exponent of N*; right: the optimal TPP at 1.1e20 FLOPs (466 at t = 0.03, outside the chart).图 20　每个点只用一个噪声水平的 held-out loss 拟合本文的公式（34 个主 ladder run）。左：N* 的指数；右：1.1e20 FLOPs 的最优 TPP（t = 0.03 时是 466，不在图内）。

Open QuestionsOpen Questions

Large batches. Our batch is fixed at 1024 images; at equal compute, doubling the batch raises the held-out loss by up to 0.60% (Figure 11). But to shorten training by adding GPUs the batch must grow, and the learning rate and weight decay must be retuned with it.

大 batch。我们的 batch 固定为 1024 张图；同样的算力下，batch 加倍会让 held-out loss 最多高 0.60%（图 11）。但要靠加 GPU 缩短训练时间，batch 就必须变大，学习率和 weight decay 也要跟着重新调。

Noise levels. Which noise levels the held-out loss uses changes the optimal split between model and data: the exponent of N* lies between 0.47 and 0.54, and the optimal TPP at 1.1e20 FLOPs between 42 and 70; the cleaner the level, the larger the optimal model (previous section). We do not know yet which choice best tracks generation quality; the model barely trains on levels above t = 0.67. One way to answer it: compare the run ranking each choice gives with the ranking from generation metrics (FD, human ratings).

噪声水平的取法。held-out loss 取哪些噪声水平，会改变最优的模型与数据配比：N* 的指数在 0.47 到 0.54 之间，1.1e20 FLOPs 的最优 TPP 在 42 到 70 之间；水平越干净，最优模型越大（见上一节）。哪种取法最能代表生成质量，我们还不知道；t 大于 0.67 的水平，模型在训练时几乎没见过。要回答它，可以比较每种取法给出的 run 排序和生成指标（FD、人工评测）给出的排序。

Zeng et al. (2026) (i1) test the architecture and data choices of text-to-image diffusion models one at a time in controlled experiments, then train a 3B-parameter model on public data. It complements this post: we fix the architecture and the data and study how to split compute between model size and data, along with training hyperparameters such as the optimizer and the learning rate; i1 fixes the training recipe (Adam, constant learning rate 1e-4, batch 512) and studies architecture and data.

Zeng et al. (2026)（i1）用受控实验逐项检验文生图 diffusion 模型的架构和数据设计，再用公开数据训练出 3B 参数的模型。它和本文互补：本文固定架构和数据，研究算力怎么分给模型大小和数据量，以及优化器、学习率这些训练超参数；i1 固定训练配方（Adam，恒定学习率 1e-4，batch 512），研究架构和数据。

Captions and data mixing. i1 finds clear gains from a stronger captioner, from training only on long captions and rewriting prompts at inference, and from equal sampling weights across datasets. Our images and captions are fixed (GPIC Full), and we did not study these.

Caption 和数据配比。i1 发现，更强的 captioner、只用长 caption 训练并在推理时改写 prompt、让每个数据集的采样权重相等，都有明显收益。我们的图像和 caption 是固定的（GPIC Full），没有研究这些。

AcknowledgementsAcknowledgements

We thank Mingyang Deng, Boya Zeng, Haocheng Xi, Ang Cao, Sihan Xu, Wentao Guo and Enxin Song for discussions.

感谢 Mingyang Deng、Boya Zeng, Haocheng Xi、Ang Cao, Sihan Xu, Wentao Guo和 Enxin Song的讨论。

Citation

@misc{chai2026scalingt2i,
  title        = {How to Scale Text-to-image Models: a Handbook},
  author       = {Chai, Wenhao},
  year         = {2026},
  howpublished = {Blog post},
  url          = {https://wenhaochai.com/blogs/scaling-generative-models-101.html}
}

All posts全部文章 Back to top回到顶部

How to Scale Text-to-image Models: a Handbook

Choosing the metric选择指标

Evaluation metrics and the held-out loss评测指标与 held-out loss

Held-out loss and FD-DINOv2held-out loss 与 FD-DINOv2

Background: scaling laws背景：Scaling law

Chinchilla's iso-FLOP curvesChinchilla 的 iso-FLOP 曲线

iso-FLOP curves of diffusion transformersDiffusion Transformer 的 iso-FLOP 曲线

TPP (tokens per parameter)TPP（token per parameter）

How N, D and compute are countedN、D 和算力的口径

iso-FLOP fits by recipe and metric各配方和指标下的 iso-FLOP 拟合

Our Scaling LawOur Scaling Law

Constants of the formula公式的常数

Optimal model size and data最优模型大小和数据量

Final LadderFinal Ladder

The final ladder's recipefinal ladder 的配方

The final ladderfinal ladder

HyperparametersHyperparameters

Removing the EMA去掉 EMA

Removing the EMA去掉 EMA

Optimizer优化器

Muon and AdamWMuon 和 AdamW

Learning rate学习率

Learning rate学习率

Weight decayWeight decay

Weight decayWeight decay

Batch sizeBatch size

Batch sizeBatch size

Cooldown冷却

Cooldown冷却

Data repetition数据重复

Data repetition数据重复

Aspect ratio长宽比

Aspect ratio长宽比

Image layers and text layers图像层和文本层

Number of text layers文本层数

InfraInfra

How MFU evolvedMFU 的进化

MFU per size每个大小的 MFU

Noise levels of the held-out lossheld-out loss 的噪声水平

Sampling distribution of the noise level噪声水平的采样分布

Loss at each noise level每个噪声水平上的 loss

Scaling laws at each noise level每个噪声水平上的 scaling law

Open QuestionsOpen Questions

AcknowledgementsAcknowledgements

Citation