Transformer 块

阅读约需 57分钟 分钟,11节中含8个示例。

版次 2026.11.16 示例于 2026年11月19日 通过 86个示例中有86个通过 变化内容

AI 翻译。由人工以英文核验。 我们如何制作图书

译自英文版图书,版次 2026.11.16。 阅读原文

读完本章,你将能够用注意力、归一化、门控前馈层和位置信息这几个部分,通过残差连接组装出一个 transformer 块,再把多个块堆叠成完整的语言模型。你会在商店已训练的模型上逐一检查每个部分,把整个模型与 PyTorch 自带的实现相比对,度量每个块和每个细节的贡献,并统计参数的分布。

第 6 章构建了注意力:每个位置回看之前的位置,并对找到的内容取加权平均。仅有注意力还不是语言模型,而商店(一家线上家具和家居用品店)需要的不止注意力。它在位置之间传递信息,却几乎不对信息做变换,而且它几乎没有词序的概念。直接堆叠多层注意力还会出现退化:缺少块中的其他部分,纯注意力的输出会趋向于每个位置上都是同一个向量,层数越多,趋同越快(Dong et al., 2021)。transformer 块在注意力周围加入了一些组成部分,用来变换每个位置的表示、提供词序,并防止深层堆叠发生退化(Vaswani et al., 2017)。

本章按照 LLaMA 模型推广开来的形式构建这个块:用 RMSNorm 对每个部分的输入做归一化,使用门控前馈层,并在每一层中使用旋转位置嵌入(Touvron et al., 2023,第 2.2 节)。第 2 章读过其分词器的 SmolLM2 也是这样构建的。它的 3.6 亿参数模型有 960 个维度、32 层和 15 个头,与商店的模型有两处细节不同,下文会讲到:它的各个头共享键和值,它的旋转位置转动得更慢(配置)。

商店的模型在商店的商品文本上训练,由 4 个这样的块构成,位于一张嵌入表和一个输出层之间,第 6 章读过它的各个头。本书把每个块称为模型的一层。在每个位置上,每个块按顺序做两件事:

  1. 对向量做归一化,运行注意力,再把结果加回到向量上;
  2. 再做一次归一化,运行前馈层,同样把结果加回去。

将各部分的输出加到残差流上

最近一次修改见于版次 2026.11.16

在第 5 章的窗口模型中,每一层都替换其输入:隐藏层的输出就是输出层所能看到的全部。Transformer 块则是在其输入上做加法。每个部分根据当前向量计算出一些东西,并把输出加到该向量上:x ← x + attention(x),然后 x ← x + feed-forward(x)。把一个部分的输出加到它的输入上,而不是替换输入,称为残差连接。它来自图像模型:在那里,只有当每一层都在其输入上做加法而不是替换输入时,很深的层堆叠才能训练得好(He et al., 2016)。

由此产生两个结果。有了这些加法,信号可以"直接从一个块传到任何其他块",前向如此,作为梯度时反向也如此,因此深层堆叠的训练比没有这些加法时可靠得多(He et al., 2016b)。同时,每个位置的向量都成为一个共享的工作空间,各部分从中读取,也向其中写入:词元的嵌入从底部进入,每个部分加上自己计算的结果,输出层在顶部读取这个和。Elhage 等人把这个累加的和,即每个位置的向量加上各部分的输出,称为残差流(Elhage et al., 2021)。

让商店的模型处理一个验证集产品,并在每一层测量注意力所加之物的长度、前馈层所加之物的长度,以及两次加法之后向量的长度,再对各位置取平均。这里向量的长度是其各数平方和的平方根:

residual-stream.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 通过
"""Runs the store's model on one validation product, one block at a time, and prints how much attention and thefeed-forward layer add to the residual stream at each layer. The prose's point rests on the two lengths printedfirst and last: the vectors entering layer 1 and leaving the last layer."""import numpy as npfrom lib import transformerfrom lib.products import product_textsweights, _, manifest = transformer.load()ids, sep = product_texts("validation", 1)[0]          # the first validation product; sep, its <sep> position, isn't needed herex = weights["embedding"][np.array(ids)]               # the stream entering block 1: each token's embeddingprint("length of the vectors entering layer 1:", f"{np.linalg.norm(x, axis=1).mean():.2f}")for layer in range(manifest["settings"]["layers"]):    layer_weights = transformer.layer_weights(layer)    added_by_attention, _ = transformer.attention(transformer.rms_norm(x, layer_weights["norm1"]), layer_weights, manifest["settings"]["heads"])    x = x + added_by_attention    normalized = transformer.rms_norm(x, layer_weights["norm2"])    added_by_feed_forward = (transformer.silu(normalized @ layer_weights["gate"]) * (normalized @ layer_weights["up"])) @ layer_weights["down"]    x = x + added_by_feed_forward    print(f"layer {layer + 1}: attention adds {np.linalg.norm(added_by_attention, axis=1).mean():.2f}, "          f"feed-forward adds {np.linalg.norm(added_by_feed_forward, axis=1).mean():.2f}, "          f"vectors now {np.linalg.norm(x, axis=1).mean():.2f}")print("length of the vectors leaving the last layer:", f"{np.linalg.norm(x, axis=1).mean():.2f}")print("largest number in the stream:", f"{np.abs(x).max():.1f}")
输出python residual-stream.py
length of the vectors entering layer 1: 1.17
layer 1: attention adds 2.90, feed-forward adds 3.00, vectors now 4.65
layer 2: attention adds 2.05, feed-forward adds 2.90, vectors now 6.14
layer 3: attention adds 2.26, feed-forward adds 4.12, vectors now 8.46
layer 4: attention adds 4.08, feed-forward adds 6.33, vectors now 13.16
length of the vectors leaving the last layer: 13.16
largest number in the stream: 50.6

残差流随层数增长,因为每次加法都叠在上一次之上。它的平均长度从进入第一层时的 1.17 增长到离开最后一层时的 13.16。每一层都向它添加内容,注意力所加之物与前馈层所加之物大小相当,不过前馈层在每一层加得都更多,在靠后的层中多得更明显:两个部分在每一层都在做实质性的工作。靠近顶部的层读取的是很长的向量,它所需的权重尺度会与读取底部较短向量的层大不相同,除非有某种机制对每一层读取的内容重新缩放;下一节就做这件事。

**工作准则:**把每个块都写成对残差流的加法,而不是替换,并让残差流的宽度在整个模型中保持不变,使各次加法能够对齐。调试深层模型时,测量每个部分加了什么:几乎什么都不加的部分,几乎什么都没做。

在每个部件之前做归一化

最近一次修改见于版次 2026.11.11

正如上一节所测量的,残差流会随着层数增加而变大,因此每个部件都应当读取它的一份重新缩放后的副本。归一化(normalization)在部件读取向量之前,把向量缩放到标准大小,这样无论残差流是短是长,部件的权重都能以同样的方式工作。最初的 Transformer 在每次相加之后做归一化,使用的是层归一化(layer normalization),即减去每个向量的均值,再除以它的标准差(Ba 等,2016)。Xiong 等人表明,改为在每个部件之前做归一化,训练会更稳定,因为被重新缩放的只是各部件读取的那份副本,而不是残差流本身(Xiong 等,2020)。而 RMSNorm 去掉了均值:它把每个向量除以其均方根(即各数平方后取平均、再开平方),并将每个维度乘以一个学习得到的增益(Zhang 和 Sennrich,2019):

RMSNorm(x) = x / √(mean(x²) + ε) · g

其中 ε 是一个很小的数,这里取 10⁻⁵,用来保证除法安全;g 是增益,初始值为 1。Zhang 和 Sennrich 发现 RMSNorm 的准确度与层归一化大致相当,而且更快。商店的模型在注意力之前、前馈层之前,以及输出层之前又一次使用它。

对同一个验证商品的残差流,在模型的两端手工做归一化,即进入第一个块时和离开最后一个块时,增益保持为 1,并把每个位置的向量与 lib.transformer.rms_norm 的结果比较:

rms-norm-by-hand.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 通过
"""Normalizes one validation product's residual stream by hand, entering the first block and leaving the last, andasserts that the result matches the shared module's rms_norm. The prose's point rests on the root mean square beforeand after normalizing, and on the largest mean of a position's numbers, which RMSNorm does not remove."""import numpy as npfrom lib import transformerfrom lib.products import product_textsids, _ = product_texts("validation", 1)[0]            # the first validation product; its <sep> position isn't neededstreams, _ = transformer.run(ids)                      # the residual stream entering block 1 and leaving each blockgain = np.ones(streams[0].shape[1])                    # the learned gain set to 1, to show the normalization alonefor label, x in [("entering block 1", streams[0]), ("leaving the last block", streams[-1])]:    root_mean_square = np.sqrt((x ** 2).mean(1, keepdims=True) + 1e-5)   # one per position; 1e-5 is the ε    by_hand = x / root_mean_square * gain    print(f"root mean square {label}, before:", f"{root_mean_square.mean():.2f}")    print(f"root mean square {label}, after:", f"{np.sqrt((by_hand ** 2).mean(1)).mean():.2f}")    assert np.allclose(by_hand, transformer.rms_norm(x, gain), atol=1e-4)   # the library's function, to 4 decimal places    print(f"same as lib.transformer.rms_norm, {label}:", "yes")    # layer normalization would subtract the mean first; RMSNorm doesn't, so this mean need not be zero    print(f"largest mean of a position's numbers after, {label}:", f"{np.abs(by_hand.mean(1)).max():.2f}")
输出python rms-norm-by-hand.py
root mean square entering block 1, before: 0.10
root mean square entering block 1, after: 1.00
same as lib.transformer.rms_norm, entering block 1: yes
largest mean of a position's numbers after, entering block 1: 0.23
root mean square leaving the last block, before: 1.16
root mean square leaving the last block, after: 1.00
same as lib.transformer.rms_norm, leaving the last block: yes
largest mean of a position's numbers after, leaving the last block: 0.26

归一化使两端处于相同的尺度。均方根平均在进入第一个块时为 0.10,在离开最后一个块时为 1.16,归一化之后两端都是 1.00;手工计算的结果与库的结果在两端都一致。RMSNorm 只做缩放而不做中心化:归一化之后,一个位置上各数的平均值仍可能偏离零,这里最多偏离 0.26。这正是它相对于层归一化省去的那一步。

因此,尽管残差流在各层之间变长了许多倍,每个注意力层和前馈层读取的向量仍具有相同的尺度,再乘以各自学习得到的增益。

工作准则: 在块的每个部件之前做归一化,而不是之后,并在输出层之前再对残差流做一次归一化。除非要与使用层归一化的模型保持一致,否则使用 RMSNorm。加载其他模型的权重时,首先要检查它使用哪种归一化、用在何处、ε 取何值:细节不匹配示例展示了微小的不匹配会带来什么代价。

这部分尚未翻译,因此以英文显示。

Transform each position with a gated feed-forward layer

最近一次修改见于版次 2026.11.16

Attention moves information between positions; the feed-forward layer transforms it, at each position separately. It's the same idea as chapter 5's hidden layer: widen the vector, apply a function that isn't a straight line, and narrow the result back to the stream's width. The store's model widens 128 numbers to 352.

The form the LLaMA models use is gated. A gated feed-forward layer computes two widened vectors from the input, passes one through a function called SiLU, and multiplies the two number by number, so the first decides how much of the second gets through (Shazeer, 2020):

FFN(x) = (SiLU(x W_gate) ⊙ (x W_up)) W_down

where ⊙ multiplies number by number, and SiLU(z) = z · σ(z), with σ the logistic function from chapter 4. SiLU passes large positive numbers almost unchanged and turns large negative ones to nearly zero, smoothly. Shazeer compared several gated forms with the plain one, at the same number of parameters, and found the gated ones predicted better; this form is called SwiGLU.

What does the gate shut? Compute SiLU at three points, then run the first block's real feed-forward layer on the same validation product and count, at each position, how many of its hidden units the gate nearly shuts:

gated-feed-forward.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 通过
"""Computes SiLU at three points, then runs block 1's feed-forward layer on one validation product and counts the hiddenunits its gate nearly shuts at each position. The prose's point rests on the three SiLU values and on that count."""import numpy as npfrom lib import transformerfrom lib.products import product_textsprint("silu(-3), silu(0), silu(3):", ", ".join(f"{v:.2f}" for v in transformer.silu(np.array([-3.0, 0.0, 3.0]))))ids, _ = product_texts("validation", 1)[0]            # the first validation product; its <sep> position isn't neededheads = transformer.load()[2]["settings"]["heads"]    # load returns (weights, vocabulary, manifest): index 2 is the manifestfirst = transformer.layer_weights(0)                   # block 1's weightsx = transformer.load()[0]["embedding"][np.array(ids)]  # index 0 is the weights: the embeddings, the stream entering block 1attended, _ = transformer.attention(transformer.rms_norm(x, first["norm1"]), first, heads)normalized = transformer.rms_norm(x + attended, first["norm2"])   # what block 1's feed-forward layer readsgate = transformer.silu(normalized @ first["gate"])    # how much of each hidden unit to let throughup = normalized @ first["up"]                          # what each hidden unit carriesoutput = (gate * up) @ first["down"]                   # gated, then projected back to the stream's widthshut = (np.abs(gate) < 0.05).sum(1)                    # per position; 0.05 is the threshold for "nearly shut"print("hidden units:", gate.shape[1])print("hidden units nearly shut at a position (gate under 0.05 in size), on average:", f"{shut.mean():.0f}")print("fewest and most nearly shut at a position:", f"{shut.min()} and {shut.max()}")print("positions:", len(ids))print("output shape:", output.shape)
输出python gated-feed-forward.py
silu(-3), silu(0), silu(3): -0.14, 0.00, 2.86
hidden units: 352
hidden units nearly shut at a position (gate under 0.05 in size), on average: 49
fewest and most nearly shut at a position: 33 and 67
positions: 133
output shape: (133, 128)

SiLU of −3, 0 and 3 gives -0.14, 0.00, 2.86: nearly zero for the negative number, nearly unchanged for the positive one.

Which units the gate shuts depends on the input. At a typical position it nearly shuts 49 of the first block's 352 hidden units, and the count ranges between 33 and 67 over the positions. So each position uses its own mix of them. The output has one vector per position at the stream's width again, (133, 128), ready to add.

The feed-forward layers hold most of a block's parameters: in the store's model, three matrices of 128 by 352, against attention's four of 128 by 128. Geva and colleagues found that in trained models these layers work like stores of patterns: each hidden unit responds to particular patterns in the text, and pushes the prediction towards particular next tokens (Geva et al., 2021).

Working rule: Use a gated feed-forward layer with SiLU. Set its width so its three matrices hold as many parameters as a plain layer four times the stream's width, the original transformer's ratio, 2,048 for 512 (Vaswani et al., 2017, section 3.3): that's about 8/3 times the width. SmolLM2-360M uses exactly 8/3, 2,560 for 960. The store's model rounds 8/3 of its width up to a multiple of 32, a rule of thumb that keeps matrix sizes convenient for hardware. When you load another model, check whether its feed-forward layer is gated, since a gated layer has three matrices where a plain one has two.

这部分尚未翻译,因此以英文显示。

Give attention positions with rotations

最近一次修改见于版次 2026.11.16

A weighted average ignores the order of what it averages, as chapter 6 showed, so attention gets at most a weak sense of position from the causal mask alone. The original transformer added a vector for each position to each token's embedding (Vaswani et al., 2017, section 3.5). The LLaMA models instead use rotary position embeddings, RoPE, which work inside attention, at each layer (Su et al., 2021; Touvron et al., 2023, section 2.2). Before scoring, RoPE rotates each query and key by angles that grow with its position. The vector's d numbers are taken in pairs, number i with number i + d/2 in the store's model, and each pair (a, b) is treated as a point on a plane and turned around the origin. At position p, pair i turns by the angle p·θᵢ, where θᵢ = 10000^(−i/(d/2)):

(a, b) → (a cos pθᵢ − b sin pθᵢ, a sin pθᵢ + b cos pθᵢ)

So the first pairs turn fast, a full radian per position, and the last pairs slowly. The line angles = ... in lib.transformer.rope computes p·θᵢ for every position and pair, and the line after it applies the rotation.

Rotation has two properties that suit attention. It doesn't change a vector's length, so it adds position without distorting content. And if a query at position m is turned by mθᵢ and a key at position n by nθᵢ, their dot product depends only on the difference, (m − n)θᵢ: the score depends on how far apart they are, not on where they sit in the text. A head can learn to look 3 tokens back, and it works the same at the start of a name and deep in a description.

Rotate a random query and key of 32 numbers, the size of the store's heads, for positions 0 to 19, and compare scaled scores, as chapter 6 defines them, at three pairs, which tests that:

rope-by-hand.py Python 3.12.15 · numpy 2.5.5 通过
"""Rotates one random query and key for each of 20 positions, checks that the rotation keeps their lengths, and scoresthree pairs of positions. The prose's point rests on the two scores for pairs 3 apart, which match, and the score forthe pair 6 apart, which doesn't."""import numpy as npfrom lib.transformer import roperng = np.random.default_rng(2026)                             # a fixed seed: the same random query and key on every runsize = 32                                                      # 32 numbers: the store's width of 128 split across 4 headsquery, key = rng.normal(0, 1, size), rng.normal(0, 1, size)positions = 20rotated_q = rope(np.tile(query, (positions, 1)))              # the same query, rotated for each position 0 to 19rotated_k = rope(np.tile(key, (positions, 1)))kept = np.allclose(np.linalg.norm(rotated_q, axis=1), np.linalg.norm(query)) and \    np.allclose(np.linalg.norm(rotated_k, axis=1), np.linalg.norm(key))   # rotation turns a vector, never changes its lengthassert keptprint("length kept by the rotation, for the query and the key:", "yes")for i, j in [(2, 5), (12, 15), (2, 8)]:                       # (key position, query position): the first two are 3 apart, the third 6    print(f"score with the key at {i} and the query at {j}:", f"{rotated_q[j] @ rotated_k[i] / np.sqrt(size):.3f}")   # scaled, as chapter 6 defines it
输出python rope-by-hand.py
length kept by the rotation, for the query and the key: yes
score with the key at 2 and the query at 5: 0.217
score with the key at 12 and the query at 15: 0.217
score with the key at 2 and the query at 8: -0.765

Pairs the same distance apart score the same, wherever they sit. The key at position 2 and the query at 5 score 0.217, and the key at 12 and the query at 15 score 0.217, the same, because both pairs are 3 apart. The key at 2 and the query at 8, 6 apart, score -0.765: the same query and key, scored differently because they're further apart. The rotation keeps both vectors' lengths, which the example asserts.

RoPE has no learned parameters: the angles are fixed, and the model learns queries and keys that use them. The store's model applies it to the queries and keys of every head in every layer, which gives chapter 6's previous-token heads a direct way to find the previous token, alongside the weak signal the mask provides.

Working rule: Use rotary position embeddings, applied to queries and keys, never values. When you load another model's weights, check how its RoPE pairs the numbers and what base it uses. Some implementations pair neighbouring numbers; others pair the first half with the second, as lib.transformer.rope does and as SmolLM2's configuration states with "rope_interleaved": false. SmolLM2 uses a base of 100,000 where the store's model uses 10,000.

将输出层与嵌入绑定

最近一次修改见于版次 2026.11.16

各个部分都构建完成后,整个模型已经足够短小,可以一次读完。lib.transformer.block就是一个块,与本章此前的描述完全一致:

def block(x: np.ndarray, layer_weights: dict[str, np.ndarray], heads: int, positions=rope):
    """One transformer block: attention, then the gated feed-forward layer, each added to the residual stream."""
    attended, weights = attention(rms_norm(x, layer_weights["norm1"]), layer_weights, heads, positions)
    x = x + attended
    normalized = rms_norm(x, layer_weights["norm2"])
    x = x + (silu(normalized @ layer_weights["gate"]) * (normalized @ layer_weights["up"])) @ layer_weights["down"]
    return x, weights

lib.transformer.run依次让每个词元的嵌入通过各个块,而lib.transformer.forward中的这几行代码把最终的数据流转换为词表中每个词元各一个的 logit,即第 5 章的 softmax 将其转换为概率的那些分数:

    streams, kept = run(ids, name, positions, skip)
    weights, _, _ = load(name)
    logits = rms_norm(streams[-1], weights["final_norm"]) @ weights["embedding"].T

最后一行还包含一个想法,即权重绑定:输出层不再自行学习一个矩阵,而是复用嵌入表的转置(即把它横过来使用)。某个词元的 logit 就是最终向量与该词元嵌入的点积。Press 和 Wolf 发现,绑定不仅节省了输出层的参数,还降低了困惑度(Press and Wolf, 2017)。SmolLM2 同样做了绑定(其配置中为"tie_word_embeddings": true),因为对于这种规模的模型,单独设置输出层会占整个模型的很大一部分。参数量显示了绑定为商店的模型节省了多少。

工作准则: 在小模型中将输出层与嵌入绑定,并且在加载模型时,先检查它是否做了绑定,再去寻找单独的输出矩阵。让你自己的前向传播保持足够简短,可以整体读完,就像lib.transformer那样,这样后续各节所检验的每一个细节都清晰可见。

这部分尚未翻译,因此以英文显示。

Check the model against PyTorch

最近一次修改见于版次 2026.11.11

PyTorch trained the store's model once, in recordings/transformer.py, as it trained chapter 5's (see Train with PyTorch), and the script saved PyTorch's own logits for every position of one validation product. The training script uses PyTorch's own RMSNorm, causal attention and SiLU, so comparing them with lib.transformer checks the NumPy code against a library's, not against a second copy of itself. RoPE is the exception: PyTorch has no built-in version, so both programs write it out, and the check confirms only that they agree. The rope-by-hand example checks its properties instead.

To test the whole forward pass, recompute the saved product's logits in NumPy and compare every position with PyTorch's:

transformer-same-as-pytorch.py Python 3.12.15 · numpy 2.5.5 通过
"""Recomputes the store's model's logits in NumPy for the validation product the training script saved, and checks everyone against PyTorch's to four decimal places. The prose's point rests on the count of logits compared, and on theassertion that they all match."""import numpy as npfrom lib import transformer_, _, manifest = transformer.load()check = np.load("data/recorded/transformer/check.npz")      # one validation product's ids, and PyTorch's logitsours = transformer.forward(list(check["ids"]))               # forward takes a list of token ids, not an arrayprint("PyTorch version:", manifest["torch"])print("positions compared:", len(check["ids"]))print("logits compared:", f"{ours.size:,}")assert np.abs(ours - check["logits"]).max() < 1e-4           # 1e-4 is four decimal placesprint("same as PyTorch to 4 decimal places:", "yes")
输出python transformer-same-as-pytorch.py
PyTorch version: 2.14.1+cpu
positions compared: 133
logits compared: 550,886
same as PyTorch to 4 decimal places: yes

The NumPy model reproduces all 550,886 of PyTorch's logits, for 133 positions, to four decimal places, with PyTorch 2.14.1+cpu. Because the check covers a whole text, it covers the parts that depend on position, the mask and the rotations, as well as the norms, gates and tied output.

Working rule: Check your own forward pass against the training library's outputs on a full text, not a single position. Where the library has its own implementation of a part, train with it, so the check compares two independent implementations; where it doesn't, as with RoPE here, test the part's properties by hand.

去掉一个块,看它的作用

最近一次修改见于版次 2026.11.16

模型已经检查过了;现在来衡量它的各个块各贡献了什么。由于每个块都向残差流中累加内容,去掉某个块后模型仍能运行:残差流原样穿过该处,后一个块照常读取它。去掉某一部分,就可以做一个简单的实验,称为消融(ablation):移除一个部分,衡量模型因此损失了多少。它显示出每个块对已训练模型的预测贡献多大,而这一点单看权重是看不出来的。

去掉哪个块损害最大?先预测一个,然后对前 100 个验证集商品的名称和描述打分,共 7,624 次预测,分别使用完整模型,以及依次去掉一个块的模型:

skip-a-block.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 通过
"""Scores the names and descriptions of the first 100 validation products with the full model, and with each block leftout in turn, reporting perplexity and the top token's average probability. The prose's point rests on the perplexitywith each block left out, set against the full model's."""import numpy as npfrom lib import transformerfrom lib.products import product_textslayers = transformer.load()[2]["settings"]["layers"]     # load returns (weights, vocabulary, manifest): index 2 is the manifesttexts = [ids for ids, _ in product_texts("validation", 100)]   # the <sep> positions aren't needed: each text is scored wholedef measure(skip=None):    """Perplexity per token, and the average probability of the top token, with one block left out."""    surprise, top, count = 0.0, 0.0, 0    for ids in texts:        p = transformer.softmax(transformer.forward(ids, skip=skip)[:-1])   # [:-1] drops the last position: nothing follows it        surprise -= np.log(p[np.arange(len(ids) - 1), ids[1:]]).sum()       # position i predicts token i + 1        top += p.max(1).sum()        count += len(ids) - 1    return np.exp(surprise / count), top / count                             # perplexity: exp of the average surprise per tokenprint("tokens scored, names and descriptions:", f"{sum(len(ids) - 1 for ids in texts):,}")for skip in [None] + list(range(layers)):                # None keeps all four blocks: the full model    label = "all blocks" if skip is None else f"without block {skip + 1}"   # skip counts blocks from 0, so block skip + 1 is left out    perplexity, top = measure(skip)    print(f"{label}:", f"{perplexity:,.1f}")    print(f"{label}, average probability of the top token:", f"{top:.2f}")
输出python skip-a-block.py
tokens scored, names and descriptions: 7,624
all blocks: 24.3
all blocks, average probability of the top token: 0.42
without block 1: 17,896.9
without block 1, average probability of the top token: 0.21
without block 2: 77.8
without block 2, average probability of the top token: 0.29
without block 3: 64.3
without block 3, average probability of the top token: 0.34
without block 4: 77.3
without block 4, average probability of the top token: 0.37

去掉第 2、3 或 4 个块,每个词元的困惑度都会增大到原来的两倍以上,从四个块齐全时的 24.3 分别升到 77.8、64.3 和 77.3:后面的每个块都贡献了模型能力的一部分,没有哪个是可有可无的。这里的困惑度同时涵盖名称和描述,因此不能与第 3 章的基线或第 5 章的窗口模型相比,后两者只在验证集的名称上打分。第 8 章会做这一比较。

去掉第 1 个块的失败方式则不同。困惑度升到 17,896.9,比在全部 4,142 个词元中均匀猜测还要差,后者的得分是 4,142。

原因在于后面的块所预期的输入。它们是在一个已经被第 1 个块写入过的残差流上训练的,没有它,它们读到的向量就与训练时见过的都不相同。它们的预测变得不那么有把握,最高概率词元的平均概率从 0.42 降到 0.21,而且把这份概率放在了错误的词元上。第 1 个块把原始嵌入变成其后一切所依赖的形式,部分做法是从已有文本中收集上下文,就像第 6 章中第一层的三个注意力头所做的那样。已训练模型中的各个块并不是相互独立的模块:每个块都依赖前面各块写入的内容。

工作准则: 要了解模型的某个部分起什么作用,就把它移除并加以衡量,而不是从权重去推断。预期第一个块最重要,因为其后的一切都建立在它之上;除此之外要靠实测,因为这里第二个和最后一个块的重要性大致相当,且都高于第三个。对于移除某部分后损失很小的情形,要谨慎解读:其余部分是与它一同训练的,所以这一测量说明的是已训练模型依赖什么,而不是这个部分单独能做什么。

衡量不匹配的细节带来的代价

最近一次修改见于版次 2026.11.16

块中的每一个细节都必须与其所加载的权重相匹配,而不匹配并不会自行显现:模型照样会输出概率。加载另一个模型的权重时,最常出错的细节在 RoPE 中,因为各种实现在配对数字的方式和基数上各不相同,SmolLM2 的基数就与本店模型的不同。

每次只改动 RoPE 的一个细节,就能看出各自的代价:用同样的 100 条验证集商品,分别对训练所得的权重评分;改为配对相邻的数字而不是前后两半;基数取 SmolLM2 所用的 100,000 而不是本店的 10,000;以及完全不使用 RoPE:

mismatched-details.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 通过
"""Scores the same 100 validation products with the trained weights, changing one detail of RoPE at a time: how it pairsnumbers, its base, and whether it is applied at all. The prose's point rests on each variant's perplexity, printedbeside the trained model's."""import numpy as npfrom lib import transformerfrom lib.products import product_textstexts = [ids for ids, _ in product_texts("validation", 100)]   # the <sep> positions aren't needed: each text is scored wholedef neighbours(x, base=10000.0):    """RoPE that pairs neighbouring numbers, (0, 1), (2, 3) and so on, instead of the first half with the second."""    half = x.shape[-1] // 2    rotated = transformer.rope(np.concatenate([x[..., 0::2], x[..., 1::2]], axis=-1), base)   # even numbers, then odd: rope pairs halves, so these are neighbours    out = np.empty_like(x)    out[..., 0::2], out[..., 1::2] = rotated[..., :half], rotated[..., half:]                 # each rotated number back where its dimension was    return outfor label, positions in [("as trained", transformer.rope),                         ("RoPE pairing neighbouring numbers", neighbours),                         ("RoPE with base 100,000", lambda x: transformer.rope(x, base=100000.0)),   # 100,000 is SmolLM2's base                         ("no RoPE", lambda x: x)]:          # no RoPE: the queries and keys pass through unrotated    print(f"perplexity, {label}:", f"{transformer.perplexity(texts, positions=positions):,.1f}")
输出python mismatched-details.py
perplexity, as trained: 24.3
perplexity, RoPE pairing neighbouring numbers: 112.2
perplexity, RoPE with base 100,000: 26.4
perplexity, no RoPE: 182.0

其中两种不匹配,第一份生成的文本样本就会暴露。配对相邻数字而不是前后两半,会使困惑度从 24.3 升至 112.2;去掉 RoPE 则升至 182.0。

基数才是危险的一种。基数取 100,000 时,困惑度为 26.4,仅比 24.3 略差,既没有报错,输出看上去也很流畅,这样的错误是有可能被发布出去的。

与 SmolLM2 的另一处差异是分组查询注意力:它的 15 个查询头共享 5 组键和值,因此其键矩阵和值矩阵的大小是查询矩阵的三分之一(Ainslie et al., 2023)。把这样的权重加载到要求每个头都有各自键和值的模型中,会因形状不符而报错,这正是人们希望出现的那种不匹配。

工作准则: 加载另一个模型的权重时,要把它的配置与你的代码逐行核对:归一化及其 ε、RoPE 的配对方式和基数、前馈层的门控、输出层是否共享权重,以及有多少个头共享键和值。然后用已知的文本评分,并与原实现的输出对照;损失偏小而又没有报错,才是最该担心的情形。

这部分尚未翻译,因此以英文显示。

Count where the parameters go

最近一次修改见于版次 2026.11.11

With every part built, count what the model learns, and where. A model's parameters are its capacity to learn, and where they sit says what it spends that capacity on. The trained model's count, by part:

transformer-parameters.py Python 3.12.15 · numpy 2.5.5 通过
"""Counts the store's model's parameters by part from its saved weights, and prints its settings. The prose's pointrests on the total, the feed-forward count and the embedding's share of it."""from lib import transformerweights, vocabulary, manifest = transformer.load()settings = manifest["settings"]# one table serves as input and output, so the embedding is counted once; a weight's group is the last part of its namegroups = {"embedding (shared with the output)": ["embedding"], "attention": ["q", "k", "v", "o"],          "feed-forward": ["gate", "up", "down"], "normalization": ["norm1", "norm2", "final_norm"]}for group, parts in groups.items():    count = sum(v.size for k, v in weights.items() if k.split(".")[-1] in parts)   # a weight's name ends in its part: layer3.q is attention's q    print(f"{group}:", f"{count:,}")print("total:", f"{sum(v.size for v in weights.values()):,}")print("embedding's share of the total:", f"{weights['embedding'].size / sum(v.size for v in weights.values()):.0%}")print("layers:", settings["layers"])print("width:", settings["dimensions"])                  # the width of the residual stream, which the manifest calls dimensionsprint("heads:", settings["heads"])print("feed-forward width:", settings["feed_forward"])print("longest text, in tokens:", settings["longest"])  # longer texts were cut to this many tokensprint("vocabulary:", f"{len(vocabulary):,}")             # the store tokenizer's tokens, plus the four markers
输出python transformer-parameters.py
embedding (shared with the output): 530,176
attention: 262,144
feed-forward: 540,672
normalization: 1,152
total: 1,334,144
embedding's share of the total: 40%
layers: 4
width: 128
heads: 4
feed-forward width: 352
longest text, in tokens: 160
vocabulary: 4,142

The model has 1,334,144 parameters, slightly fewer than the 1,394,732 of chapter 5's window model. The two are about the same size but spend their parameters differently. The window model put about three quarters of its parameters in its output layer, and could see four tokens back. The transformer puts most of its parameters in its blocks, 540,672 of them in feed-forward layers alone. It shares its embeddings with its output, and can see every earlier token in a text of up to 160 tokens.

The embedding table, shared with the output layer, holds 530,176 parameters, 40% of the total, because it has a row of 128 numbers for each of the 4,142 tokens. Those are the store tokenizer's vocabulary plus four markers: start, separator, end, and padding, which fills out short texts in a training batch. Chapter 8 explains how the model was trained, and measures it against the baseline.

Working rule: Count parameters by part whenever you design or load a model. At small sizes, the embedding table is a large share of the total, so tie it to the output layer, and choose the vocabulary size with that cost in mind, as chapter 2's choice of merges did.

生产环境中会出什么问题

最近一次修改见于版次 2026.11.11

Transformer 的各个部件必须严丝合缝地配合,而大多数问题出在“差一点就能配合”的部件上:

  • 细节不匹配可能悄无声息地出错。 归一化放在哪里、RoPE 如何给数字配对及其基数、前馈层是否带门控、输出是否与嵌入共享权重,这些都必须与训练得到的权重一致。有些不匹配会立刻产生无意义的输出;另一些,例如 RoPE 基数(此处已测量),只会损失少量准确率,却不报任何错误。请对照参考输出进行检查。
  • 超出训练长度的文本未经检验。 RoPE 可以为任意位置给出角度,但如果模型是在最长 160 个词元的文本上训练的,它从未学过如何使用更远位置的角度。在你实测过更长的文本之前,应把训练长度视为模型的上限。
  • 在低精度下,较大的值可能溢出。 位数较少的格式在每个常见数量级上保持相同的相对精度,但 float16 无法表示大于 65,504 的值,而在 8 位格式中,少数较大的离群值就会毁掉其他所有值的精度(Dettmers et al., 2022)。对于此处测量的商品,商店模型的残差流峰值为 50.6,远低于 float16 的上限;在以较低精度部署模型之前,请检查模型残差流中的最大值。
  • 把参考输出与权重放在一起保存。 结果取决于所用的机器,正如第 4 章所解释的那样,因此要把权重、参考输出和完整的训练脚本一并保存,并在代码或库发生任何变动后重新运行检查。
  • 每个部件在每个位置上都要运行。 前馈层在每个位置上的开销相同,而注意力的开销随长度的平方增长,如第 6 章所示。在优化其中任何一个之前,请先针对你的文本测量哪一个占主导。

本章的决定

  • **本书存储系统中的 transformer 采用 LLaMA 风格的模块,**规模很小:每个部分之前做 RMSNorm,注意力中使用旋转位置编码,前馈层是带门控的 SiLU 层,输出层与嵌入共享权重。它有 4 层,宽度为 128,4 个头,前馈宽度为 352。SmolLM2-360M 的组成部分种类相同,但规模更大,使用分组查询注意力,RoPE 基数为 100,000。
  • **lib.transformer 中的 NumPy 前向传播是本书的参考实现,**它在一个验证产品的每个位置上,与 PyTorch 自带的 RMSNorm、注意力和 SiLU 对照检查,精确到小数点后四位。之后每个用到该模型的示例都运行这一实现。
  • **每个模块都向残差流中累加,每个部分读取的都是残差流经过归一化的副本,**因此残差流可以随层数增长,而每个部分看到的向量尺度保持固定。
  • **判断各部分的作用,方法是测量去掉它们或改变它们之后的模型。**去掉任何一个模块,都会使本书存储系统的模型明显变差,去掉第一个模块则使模型无法使用,所以本书把训练好的模型的各个模块视为一个整体。RoPE 基数错误只会带来很小的损失,而且不会引发任何错误,因此加载的权重总要与参考输出对照检查。
  • **该模型的规模与第 5 章的窗口模型相当,只是参数用在各个模块上,而不是输出层上。**与输出层共享的嵌入表仍占其中的 40%,因此词表大小依然是本书持续关注的一项成本。

本章,在本版次中

版次 2026.11.16示例于 2026年11月19日 通过本章 8个示例 个示例中有 8 个通过

译自英文版图书,版次 2026.11.16。

引用本节

格式

每种格式均引用英文原书,本译本以其为准。

每种格式都会注明该版次。其链接会打开当前版次,节标题处会说明该节文本最后一次修改的时间。访问日期和检索日期即为您打开此页的当天。

本页内容

第 7 章,共 11 章: Transformer 块

  1. 将各部分的输出加到残差流上
  2. 在每个部件之前做归一化
  3. Transform each position with a gated feed-forward layer
  4. Give attention positions with rotations
  5. 将输出层与嵌入绑定
  6. Check the model against PyTorch
  7. 去掉一个块,看它的作用
  8. 衡量不匹配的细节带来的代价
  9. Count where the parameters go
  10. 生产环境中会出什么问题
  11. 本章的决定

返回顶部全部章节