Generating Distributions

By Wenhao Chai作者 Wenhao Chai

Setup设置

We train the official baseline of GPIC, the PixelGen recipe in the GPIC repository: a transformer of about 1.1B parameters that denoises 256×256 images directly in pixel space, conditioned on a frozen Qwen3-1.7B text encoder. The data is GPIC Lite, training shards 0 to 800 of the Giant Permissive Image Corpus, which hold about 10M image–caption pairs in 1.28 TB of WebDataset tar files. We evaluate with the official gpic_eval kit, which generates 50K images from the test captions and reports FD-DINOv2, the Fréchet distance between DINOv2 features of the generated and the test images (lower is better). The page compares the wall-clock time of each stage of the loop, from data loading through training and sampling to evaluation, while the training math stays the same.

我们训练 GPIC 的官方基线，即 GPIC 仓库里的 PixelGen 配方：一个约 1.1B 参数的 transformer，直接在像素空间给 256×256 的图像去噪，条件来自冻结的 Qwen3-1.7B 文本编码器。数据是 GPIC Lite，即 Giant Permissive Image Corpus 的第 0 到 800 个训练分片，约 1000 万对图文，存成 1.28 TB 的 WebDataset tar 文件。评测用官方的 gpic_eval 工具：按测试集的 5 万条描述生成 5 万张图，报告 FD-DINOv2，即生成图和测试图在 DINOv2 特征上的 Fréchet 距离，越低越好。本页比较整个流程每个环节的实际耗时，从读数据、训练、采样一直到评测，训练的数学始终不变。

Details详细说明

The model is the JiT-T2I transformer of PixelGen, trained with flow matching to predict the clean image (x-prediction). It has a hidden width of 1536, 24 attention heads, 16 image blocks and 4 blocks that refine the text features, and cuts the image into 16-pixel patches, so a 256×256 image becomes 256 tokens. Captions are padded to 300 tokens before the text encoder. Each image has one caption: a list of tags, or a short, medium or long description, in proportions of 1%, 45%, 45% and 9%. The official run uses AdamW with learning rate 1e-4 and betas (0.9, 0.95), a global batch of 256 on 8 GPUs, bf16 and an EMA decay of 0.9999. It trains for 400K steps, about one epoch of GPIC Full, and reaches an FD-DINOv2 of about 76. The evaluation samples with 50 Euler steps and classifier-free guidance, extracts DINOv2 ViT-L/14 features at 224 pixels, compares them with the statistics of the test set that the kit ships, and also reports precision, recall, density, coverage and MMD.

模型是 PixelGen 的 JiT-T2I transformer，用 flow matching 训练，直接预测干净图像（x-prediction）。它的隐藏维度为 1536，有 24 个注意力头、16 个图像块和 4 个细化文本特征的块，图像切成 16 像素的 patch，所以一张 256×256 的图对应 256 个 token。描述在送入文本编码器前补齐到 300 个 token。每张图有一条描述，形式是一串标签，或短、中、长三种描述之一，比例分别为 1%、45%、45% 和 9%。官方 run 使用 AdamW，学习率 1e-4，betas 为 (0.9, 0.95)，8 张 GPU 上全局 batch 为 256，bf16，EMA 衰减 0.9999。它训练 40 万步，约为 GPIC Full 的一个 epoch，FD-DINOv2 约为 76。评测用 50 步 Euler 采样并加 classifier-free guidance，在 224 像素下提取 DINOv2 ViT-L/14 特征，与工具自带的测试集统计量比较，同时报告 precision、recall、density、coverage 和 MMD。

Experiments实验

Testing检验中Does the official pipeline run end to end on GPIC Lite on our cluster and behave like the official run?官方流程能在我们的集群上用 GPIC Lite 完整跑通，并且表现和官方 run 一致吗？

Testing检验中Where does the time of a training step and of an evaluation go?一步训练和一次评测的时间花在哪里？

A training step splits into data loading and JPEG decoding, the Qwen3-1.7B text encoder, the denoiser's forward and backward passes, and the optimizer step with the EMA update. An evaluation splits into sampling and DINOv2 feature extraction.

一步训练分成几段：读数据和解码 JPEG、Qwen3-1.7B 文本编码器、去噪网络的前向和反向，以及优化器更新和 EMA 更新。一次评测分成采样和提取 DINOv2 特征两段。

Testing检验中How much faster can the whole loop get with the training math unchanged?训练的数学不变，整个流程能快多少？

A speedup counts only when the model, the data, the batch size and the optimizer stay those of the official configuration. We work from the infrastructure, namely the data path, parallelism, compilation and CUDA graphs, down to the kernels for attention, the fused normalization and modulation, and the optimizer.

只有模型、数据、batch 大小和优化器都与官方配置相同，提速才算数。我们从基础设施做起，即数据通路、并行方式、编译和 CUDA graph，一直做到 kernel：注意力、融合的归一化与调制，以及优化器。

The first bar of each panel also answers question 2.

每个子图的第一根柱子也回答了问题 2。

Testing检验中Does the champion recipe, moved to pixel space, beat PixelGen at the same training compute?冠军方案搬到像素空间后，在相同训练算力下能胜过 PixelGen 吗？

The champion recipe is the best recipe of the text-to-image track of Autonomous Pretraining at Scale, here moved to pixel space, with the official Qwen3-1.7B text encoder, no classifier-free guidance and packed captions.

冠军方案是 Autonomous Pretraining at Scale 文生图赛道的最佳方案。这里把它搬到像素空间，用官方的 Qwen3-1.7B 文本编码器，不用 classifier-free guidance，并把描述打包输入。

Testing检验中Does Muon train PixelGen to better samples than AdamW?用 Muon 训练 PixelGen，样本质量比 AdamW 更好吗？

The matrices inside the transformer blocks take Muon, with its update scaled to the size of AdamW's, and everything else keeps the official AdamW; a variant splits the fused attention, MLP and modulation matrices into their parts before orthogonalizing.

Transformer 块里的矩阵改用 Muon，更新幅度缩放到与 AdamW 相当，其余参数仍用官方的 AdamW。一个变体先把融合的注意力、MLP 和调制矩阵拆成各自的部分，再做正交化。

Takeaways核心结论

Acknowledgements致谢

The author is pleased to acknowledge that the work reported on in this post was substantially performed using the Princeton Research Computing resources at Princeton University. Princeton Research Computing is a consortium of groups including the Princeton Institute for Computational Science and Engineering (PICSciE) and Research Computing at Princeton University.

本文报告的工作主要使用普林斯顿大学 Princeton Research Computing 的计算资源完成。Princeton Research Computing 是由普林斯顿计算科学与工程研究所（PICSciE）和普林斯顿大学 Research Computing 等团队组成的联合体。

Experiment record实验记录

Citation

@misc{chai2026gpicspeed,
  title        = {Generating Distributions},
  author       = {Chai, Wenhao},
  year         = {2026},
  howpublished = {Blog post},
  url          = {https://wenhaochai.com/exps/gpic-speed.html}
}

Experiments Back to top回到顶部