The Open LLM Architecture Lineage开源 LLM 架构谱系

The lineage谱系图

How to read it如何阅读

The graph grows one model at a time, in release order. Each new model is compared with every model already placed and attaches under the one whose design it changes least, mechanism-level differences counting before all others; the line under a model name lists exactly those changes, mechanism-level ones first. A model that changes nothing, only its size, is folded into the model it copies: the node keeps the earliest name, and the card and the hover card list every name it stands for. DeepSeek R1 and all four Kimi K2 releases live inside the DeepSeek V3 node that way, Laguna S 2.1 inside Laguna XS 2.1. Squares are models nothing descends from yet.

图按发布顺序一个模型一个模型地生长。每个新模型与已放入图中的所有模型比较，挂在设计改动最少的那个之下，机制级差异先于其他一切计数；模型名下方那一行列出的正是这些改动，机制级的排在前面。只改规模、设计不变的模型被并入它所复制的那个模型：节点保留最早的名字，卡片和悬浮卡列出它代表的全部名字。DeepSeek R1 与四个 Kimi K2 版本就这样并入 DeepSeek V3 节点，Laguna S 2.1 并入 Laguna XS 2.1。方块表示尚无后代的模型。

A model usually has more than one parent. Besides the closest design, every mechanism it adds draws one more edge from the earliest model in the graph that already carried it in the same form, unless that model is already an ancestor through another edge: the mixture of experts traces to DeepSeek V3, gated attention to Qwen3-Next, Kimi Delta Attention to Kimi Linear, hyper-connections to DeepSeek V4-Pro. When even the closest design needs more than 32 changes, the model hangs off GPT-2 XL, the single origin; no model currently needs that.

一个模型通常有不止一个父节点。除了最近的设计之外，它新增的每一种机制都会从图中最早以相同形式具备该机制的模型引出一条边，除非该模型已经经由另一条边成为祖先：混合专家追溯到 DeepSeek V3，门控注意力追溯到 Qwen3-Next，Kimi Delta Attention 追溯到 Kimi Linear，超连接追溯到 DeepSeek V4-Pro。当连最近的设计也需要超过 32 项改动时，模型直接挂在唯一的起点 GPT-2 XL 之下；目前没有模型需要这样。

How the graph was built这张图是怎么来的

The pipeline is four steps, and the graph is a deterministic function of the configs. Code and data are at github.com/wenhaochai/llm-architecture-lineage.

整条流水线共四步，图完全由 config 决定。代码与数据见 github.com/wenhaochai/llm-architecture-lineage。

1. Collect the configs. The gallery page lists 103 model cards, each with a link to the model's config.json on Hugging Face. The script downloads all of them. Eleven repositories are gated; for six of those an ungated mirror carries the identical file, Mistral Large 3 ships a native params.json that maps one-to-one onto the same fields, BTL-3 is a LoRA adapter whose adapter_config.json names Qwen3.6-27B as its base, and Soofi-S states in its model card and technical report that it adopts the Nemotron 3 Nano architecture without modification, which the paper's hyperparameter table confirms. 收集 config。图库页面列出 103 张模型卡，每张都链接到 Hugging Face 上的 config.json，脚本全部下载。其中 11 个仓库是 gated：6 个有内容完全相同的非 gated 镜像；Mistral Large 3 提供原生 params.json，字段一一对应；BTL-3 是 LoRA adapter，其 adapter_config.json 写明基座为 Qwen3.6-27B；Soofi-S 的模型卡和技术报告都说明它原样采用 Nemotron 3 Nano 架构，论文的超参表可以核对。
2. Clean and rename the keys. The 103 configs use 571 distinct keys. A hand-written taxonomy drops the 155 that carry no architecture: 68 decoding defaults, tokenizer ids, and Hugging Face bookkeeping; 40 dtype, kernel, parallelism, and implementation switches; 26 training-only settings such as initialisation, dropout, and auxiliary losses; 11 multi-token-prediction heads; 8 multimodal leftovers; the 2 that name the modeling code. The remaining 416 architecture keys are renamed through an alias table into 170 canonical fields, 98 of which merge two or more spellings of the same quantity: seven names for the number of active experts, six for the norm epsilon, four for QK-Norm. Nested sub-configs are flattened into the same fields, the per-layer index lists inside them become layer schedules by kind, and each field sits in one of four sections that follow the forward pass of a decoder block, embeddings and output, token mixing, channel mixing, block structure and residual stream, with token mixing split further into layer schedule, attention heads, latent attention, sparse attention, linear and recurrent mixers, and positions. A per-layer list that runs on into the multi-token-prediction heads is cut at the last decoder layer. A schedule's ratio is read off its repeating period; a minority kind that sits in one block at the front, such as the two sliding-window layers that open DeepSeek V4-Flash, carries no ratio, and in DeepSeek V4 a compression ratio of 0 marks such a layer, so those entries move into the schedule. Every field also carries one of three kinds. Design fields, 61 of them, are mechanism choices, layer schedules, and switches: attention kind, router scoring, sliding-window pattern, gated attention. Scale fields, 67, are sizes and counts that grow with the model: layers, width, heads, experts, latent ranks, window length, and weight tying, which small models use and large ones drop. Tuning fields, 47, are continuous hyper-parameters and conventions: epsilons, RoPE thetas, scaling schemes and dimension interleaving, clamps, muP multipliers. A key written with its default value says what an omitted key says, so the extractor settles both to the same thing: rope_type: default and moe_layer_freq: 1 and first_k_dense_replace: 0 are dropped, and an absent attention_bias, tie_word_embeddings, hidden_act, norm_topk_prob or router scoring is filled with the value the class uses when the key is missing. A few design traits never appear as keys at all because the modeling class fixes them, QK-Norm in every Gemma 3 and OLMo 2 layer, gated attention in Qwen3-Next; the extractor adds those, together with five derived fields: the attention kind read off head counts and latent ranks, the sequence-mixer kind, the rounded local-to-global and mixer-to-attention layer ratios, and attention residuals, which only appear as a block size. Every key of every config is covered exactly once. The card under the graph is that config, section by section, with the original spelling or the implying model class under each field; fields the model changed relative to its parent are set in red. 清理并统一命名。103 个 config 共用到 571 个不同 key。先用一张手工分类表去掉 155 个与架构无关的：68 个解码默认值、tokenizer id 与 Hugging Face 记账字段；40 个 dtype、kernel、并行与实现开关；26 个仅训练期有效的设置，如初始化、dropout、辅助损失；11 个多 token 预测头字段；8 个多模态残留；2 个标识建模代码的字段。剩下 416 个架构 key 通过别名表重命名为 170 个统一字段，其中 98 个字段合并了同一量的两种以上写法：激活专家数有 7 种写法，归一化 epsilon 有 6 种，QK-Norm 有 4 种。嵌套的子配置被展平到同一套字段，其中的逐层索引列表变成按层类型计数的调度；每个字段归入四个板块之一，板块顺序就是一个 decoder block 的前向顺序：嵌入与输出、序列混合、通道混合、块结构与残差流；序列混合下再分层调度、注意力头、潜在注意力、稀疏注意力、线性与循环混合器、位置编码。延伸到多 token 预测头的逐层列表在最后一个 decoder 层截断。调度的比例从其重复周期读出；若少数层类型只集中在开头一段，如 DeepSeek V4-Flash 开头的两层滑动窗口，则不记比例，而 DeepSeek V4 中压缩比为 0 正是这种层，这些条目归入层调度。每个字段同时带有三种类别之一。设计字段 61 个，是机制选择、层调度与开关：注意力种类、路由打分方式、滑动窗口模式、门控注意力。规模字段 67 个，是随模型变大的尺寸与数量：层数、宽度、头数、专家数、潜在秩、窗口长度，以及小模型共享、大模型不共享的输入输出嵌入。调参字段 47 个，是连续超参数与约定：epsilon、RoPE theta、缩放方案与维度交错方式、钳制值、muP 系数。以默认值写出的 key 与省略该 key 含义相同，抽取器把两者统一：rope_type: default、moe_layer_freq: 1、first_k_dense_replace: 0 一律删去；缺失的 attention_bias、tie_word_embeddings、hidden_act、norm_topk_prob 与路由打分方式，则按建模类在缺省时使用的值补齐。另有少数设计特征从不以 key 出现，因为建模类已经固定了它们，如 Gemma 3 与 OLMo 2 每层的 QK-Norm、Qwen3-Next 的门控注意力；抽取器把这些补进统一 config，另加五个派生字段：由头数与潜在秩读出的注意力种类、序列混合器种类、取整后的局部与全局层比、混合层与注意力层比，以及只以块大小出现的注意力残差。每个 config 的每个 key 都恰好被覆盖一次。图下方的卡片就是这份 config，逐板块列出，每个字段下方标出原始写法或推断它的建模类；相对父节点改动过的字段用红色标出。
3. Decide what counts as a change. The line under each model name lists what its config changes relative to its closest design. A design field counts whenever its value differs; per-layer schedules are compared by the kinds of layer they contain and by their period ratio, so a longer copy of the same schedule is no change. Scale fields never count by value: 61 layers becoming 43, or 128 heads becoming 64, is a smaller copy of the same design. Fifteen of them mark a mechanism when they appear, latent ranks for MLA, an indexer budget for sparse attention, Mamba heads for a state-space mixer, and only that appearance is reported. Tuning fields never count. One decision is also counted once: the per-layer schedule states which mixers run in what proportion, so the local-to-global ratio, the window switches and the mixer kind restate it and stay silent; the attention kind states that attention is latent; and when a mechanism itself changes, its own settings are skipped, so moving from dense to a mixture of experts is one change rather than five. A model whose config differs from its parent only in excluded fields reads as "same design, different scale" and is folded into that parent. 规定什么算改动。每个模型名下方那一行列出它的 config 相对最近设计改了什么。设计字段的值不同就算改动；逐层调度按其包含的层类型与周期比例比较，同一调度的加长版不算改动。规模字段的数值从不算改动：61 层变 43 层、128 头变 64 头，只是同一设计的缩小版。其中 15 个规模字段一旦出现即标志某种机制，如 MLA 的潜在秩、稀疏注意力的索引预算、状态空间混合器的 Mamba 头数，只报告这种出现。调参字段从不算改动。同一个决定也只计一次：逐层调度已经说明哪些混合器按什么比例排布，因此局部与全局比、窗口开关、混合器种类都是它的复述，不再计入；注意力种类本身已说明是否为潜在注意力；某个机制发生变化时，它自己的各项设置一并跳过，于是从稠密改为混合专家算一次改动而不是五次。与父节点只在被排除字段上不同的模型，显示为“设计相同，只有规模不同”，并被并入该父节点。
4. Insert models in release order. GPT-2 XL opens the graph. Each later model is compared with every model already placed, except scale copies, using the change list of step 3. Candidates are ranked first by how many mechanism-level fields differ, nineteen fields that decide what the block computes: attention kind, layer schedule, sequence mixer, dense or MoE, sparse attention, latent attention, gated attention, QK-Norm, positional encoding, NoPE layers, hyper-connections, attention residuals, gated residuals, looped depth, per-layer embeddings, cross-layer KV sharing, encoder-decoder structure, and the two n-gram memories; then by the total number of changes; ties go to the same modeling class, then the same organisation, then the earlier release, since the earlier of two equally close designs is where that design came from. A candidate that keeps every mechanism therefore beats one that shares more small settings but swaps, say, Mamba-2 for attention. Zero changes make the new model a scale copy: it is folded into its parent's node, listed there by name, and never eligible as a parent. If even the closest placed model needs more than 32 changes, the model hangs off GPT-2 XL instead; no model currently does, the largest gap being 10. Then, for every mechanism-level field the model adds or switches to, one more edge comes from the earliest placed model that already carried that field with the same value. Edges that a longer path already implies are dropped: when A reaches C through B, the direct edge from A to C goes, so the figure keeps only the shortest set of edges that spans the same ancestry. The model's generation is one more than the largest generation among its parents, and that number is its column. 37 of the 103 models are scale copies, the six Qwen3 sizes folding into three nodes, so the figure shows 66 nodes and 80 edges across 10 generations. Names never carry a parameter count; when two drawn nodes would read the same, such as the dense and MoE Qwen3, the full gallery title steps in. 按发布顺序逐个插入。GPT-2 XL 开启整张图。之后每个模型与已放入的所有模型比较，规模副本除外，使用第 3 步的改动列表。候选先按机制级字段的改动数排序，这十九个字段决定一个 block 算什么：注意力种类、逐层调度、序列混合器、稠密或 MoE、稀疏注意力、潜在注意力、门控注意力、QK-Norm、位置编码、NoPE 层、超连接、注意力残差、门控残差、循环深度、逐层嵌入、跨层 KV 共享、编码器-解码器结构以及两种 n-gram 记忆；再按总改动数排序；并列时优先同一建模类，其次同一机构，再其次发布更早者，因为两个同样接近的设计中更早的那个才是来源。因此保留全部机制的候选，会胜过共享更多细节设置却把 Mamba-2 换成注意力的候选。零改动使新模型成为规模副本：并入父节点，在那里按名字列出，且永不具备做父节点的资格。若连最近的已放入模型也需要超过 32 项改动，则该模型直接挂在 GPT-2 XL 之下；目前没有模型触发，最大的差距是 10 项。随后，模型新增或切换到的每个机制级字段，都从图中最早以相同取值具备它的模型引出一条边。更长路径已经蕴含的边会被去掉：A 经 B 可以到达 C 时，A 到 C 的直接边删除，图中只保留能覆盖同一祖先关系的最少边集。模型的代数是其所有父节点代数最大值加一，这个数就是它所在的列。103 个模型中有 37 个是规模副本，六个 Qwen3 尺寸并成三个节点，因此图中显示 66 个节点、80 条边、共 10 代。名称一律不带参数量；只有两个节点会显示成同一个名字时，如稠密与 MoE 的 Qwen3，才改用图库的完整标题。

Two caveats. The gallery is a curated sample, so a mechanism's "origin" is its first appearance among these 103 models; Mamba-2 hybrids, for instance, predate Nemotron 3 Nano in the literature. And a closest-design edge measures the distance between two configs; which papers a team actually read is a separate question that this graph leaves open.

两点说明。图库是人工挑选的样本，因此某种机制的“来源”指它在这 103 个模型中的首次出现；以文献计，Mamba-2 混合架构早于 Nemotron 3 Nano。此外，最近设计的连线度量的是两个 config 之间的距离；某个团队实际参考了哪些工作，这张图并不回答。

Citation

@misc{chai2026lineage,
  title        = {The Open LLM Architecture Lineage},
  author       = {Chai, Wenhao},
  year         = {2026},
  howpublished = {Blog post},
  url          = {https://wenhaochai.com/blogs/llm-architecture-lineage.html}
}

Model list and config links come from Sebastian Raschka's LLM Architecture Gallery; all configs come from the linked Hugging Face repositories.

模型列表与 config 链接来自 Sebastian Raschka 的 LLM Architecture Gallery，全部 config 来自对应的 Hugging Face 仓库。