The transformer block
閱讀約需 60 分鐘 分鐘,11 節 中共有 8 個範例。
此部分尚未翻譯,因此以英文呈現。
By the end of this chapter you can assemble a transformer block from its parts, joined by residual connections: attention, normalization, a gated feed-forward layer and position information. You can then stack blocks into a complete language model. You'll check each part on the store's trained model, check the whole model against PyTorch's own implementations, measure what each block and each detail contributes, and count where the parameters go.
Chapter 6 built attention: each position looks back at earlier ones and takes a weighted average of what it finds. Attention alone isn't a language model, and the store, an online furniture and home-goods shop, needs more than attention. It moves information between positions but transforms little of it, and it has almost no sense of word order. Layers of it stacked directly also degrade: without the other parts of a block, the output of pure attention tends towards the same vector at every position, faster with every layer (Dong et al., 2021). The transformer block adds parts around attention that transform each position, give it word order and keep a deep stack from degrading (Vaswani et al., 2017).
This chapter builds the block in the form the LLaMA models popularized: normalization of each part's input with RMSNorm, a gated feed-forward layer and rotary position embeddings in every layer (Touvron et al., 2023, section 2.2). SmolLM2, whose tokenizer chapter 2 read, is built the same way. Its 360-million-parameter model has 960 dimensions, 32 layers and 15 heads, and differs from the store's model in two details covered below: its heads share keys and values, and its rotary positions turn more slowly (configuration).
The store's model, trained on the store's product text, is 4 of these blocks between an embedding table and an output layer, and chapter 6 read its heads. This book calls each block a layer of the model. At every position, each block does two things in order:
- normalize the vector, run attention, and add the result back to the vector;
- normalize again, run a feed-forward layer, and add that result back too.
將各部分的輸出加到殘差流
在第 5 章的視窗模型中,每一層都會取代它的輸入:隱藏層的輸出就是輸出層所看到的全部內容。Transformer 區塊則是在輸入上做加法。每個部分根據目前的向量計算出某些內容,再把輸出加回該向量:x ← x + attention(x),接著 x ← x + feed-forward(x)。把某部分的輸出加到它的輸入上,而不是取代輸入,稱為殘差連接(residual connection)。這個做法源自影像模型:非常深的層疊,只有在每一層都是對輸入做加法而非取代時,才能訓練得好(He et al., 2016)。
由此產生兩項結果。有了這些加法,訊號可以「直接從一個區塊傳到任何其他區塊」,往前傳遞,也以梯度的形式往後傳遞,因此深層堆疊的訓練比沒有加法時穩定得多(He et al., 2016b)。此外,每個位置的向量成為一個共用的工作空間,各部分都從中讀取,也都寫入其中:詞元的嵌入從底部進入,每個部分加上自己算出的內容,輸出層則在頂部讀取總和。Elhage 等人把這個累加的總和,也就是每個位置的向量加上各部分輸出之後的結果,稱為殘差流(residual stream)(Elhage et al., 2021)。
在一件驗證集商品上執行商店的模型,並在每一層測量注意力所加入內容的長度、前饋層所加入內容的長度,以及兩次加法之後向量的長度,再對各位置取平均。這裡向量的長度是各數值平方和的平方根:
1"""Runs the store's model on one validation product, one block at a time, and prints how much attention and the2feed-forward layer add to the residual stream at each layer. The prose's point rests on the two lengths printed3first and last: the vectors entering layer 1 and leaving the last layer."""4import numpy as np56from lib import transformer7from lib.products import product_texts89weights, _, manifest = transformer.load()10ids, sep = product_texts("validation", 1)[0] # the first validation product; sep, its <sep> position, isn't needed here11x = weights["embedding"][np.array(ids)] # the stream entering block 1: each token's embedding12print("length of the vectors entering layer 1:", f"{np.linalg.norm(x, axis=1).mean():.2f}")13for layer in range(manifest["settings"]["layers"]):14 layer_weights = transformer.layer_weights(layer)15 added_by_attention, _ = transformer.attention(transformer.rms_norm(x, layer_weights["norm1"]), layer_weights, manifest["settings"]["heads"])16 x = x + added_by_attention17 normalized = transformer.rms_norm(x, layer_weights["norm2"])18 added_by_feed_forward = (transformer.silu(normalized @ layer_weights["gate"]) * (normalized @ layer_weights["up"])) @ layer_weights["down"]19 x = x + added_by_feed_forward20 print(f"layer {layer + 1}: attention adds {np.linalg.norm(added_by_attention, axis=1).mean():.2f}, "21 f"feed-forward adds {np.linalg.norm(added_by_feed_forward, axis=1).mean():.2f}, "22 f"vectors now {np.linalg.norm(x, axis=1).mean():.2f}")23print("length of the vectors leaving the last layer:", f"{np.linalg.norm(x, axis=1).mean():.2f}")24print("largest number in the stream:", f"{np.abs(x).max():.1f}")
python residual-stream.pylength of the vectors entering layer 1: 1.17
layer 1: attention adds 2.90, feed-forward adds 3.00, vectors now 4.65
layer 2: attention adds 2.05, feed-forward adds 2.90, vectors now 6.14
layer 3: attention adds 2.26, feed-forward adds 4.12, vectors now 8.46
layer 4: attention adds 4.08, feed-forward adds 6.33, vectors now 13.16
length of the vectors leaving the last layer: 13.16
largest number in the stream: 50.6
殘差流會隨著層數增加而變長,因為每次加法都疊在前一次之上。它的平均長度從進入第一層時的 1.17,增長到離開最後一層時的 13.16。每一層都在往其中加入內容,注意力加入的量與前饋層加入的量大小相當,不過前饋層在每一層加入的都比較多,越往後的層差距越大:兩個部分在每一層都做了實質的工作。若沒有東西對各層所讀取的內容重新縮放,那麼讀取頂端長向量的層,所需的權重尺度就會與讀取底端短向量的層大不相同;下一節便處理這件事。
實用原則: 將每個區塊都寫成對殘差流的加法,而不是取代,並讓殘差流的寬度在整個模型中保持一致,使各次加法能夠對齊。除錯深層模型時,測量每個部分加入了多少內容:加入量幾乎為零的部分,就幾乎沒有做任何事。
在每個部分之前先正規化
如上一節所量測的,殘差流會隨著層數增加而變大,因此每個部分都應該讀取它重新縮放後的副本。正規化(normalization)會在某個部分讀取向量之前,先把向量縮放到標準大小,使該部分的權重無論殘差流是短是長都能以相同方式運作。原始的 transformer 在每次相加之後才進行正規化,採用的是層正規化(layer normalization),它會減去每個向量的平均值,再除以其標準差(Ba et al., 2016)。Xiong 等人指出,改成在每個部分之前正規化,訓練會更穩定,因為被縮放的只有各部分所讀取的副本,殘差流本身從不被縮放(Xiong et al., 2020)。而 RMSNorm 則省去了平均值這一步:它把每個向量除以其均方根(root mean square,即各數值平方後取平均再開平方根),再將每個維度乘上一個學習得到的增益(gain)(Zhang and Sennrich, 2019):
RMSNorm(x) = x / √(mean(x²) + ε) · g
其中 ε 是一個極小的數,此處為 10⁻⁵,用來確保除法安全;g 是增益,初始值為 1。Zhang 與 Sennrich 發現 RMSNorm 的準確度與層正規化相當,而且速度更快。商店的模型在注意力層之前、前饋層之前,以及輸出層之前再一次使用它。
我們以手算的方式,對同一個驗證商品的殘差流在模型的兩端進行正規化,也就是進入第一個區塊時與離開最後一個區塊時,增益維持為 1,並將每個位置的向量與 lib.transformer.rms_norm 的結果比較:
1"""Normalizes one validation product's residual stream by hand, entering the first block and leaving the last, and2asserts that the result matches the shared module's rms_norm. The prose's point rests on the root mean square before3and after normalizing, and on the largest mean of a position's numbers, which RMSNorm does not remove."""4import numpy as np56from lib import transformer7from lib.products import product_texts89ids, _ = product_texts("validation", 1)[0] # the first validation product; its <sep> position isn't needed10streams, _ = transformer.run(ids) # the residual stream entering block 1 and leaving each block11gain = np.ones(streams[0].shape[1]) # the learned gain set to 1, to show the normalization alone12for label, x in [("entering block 1", streams[0]), ("leaving the last block", streams[-1])]:13 root_mean_square = np.sqrt((x ** 2).mean(1, keepdims=True) + 1e-5) # one per position; 1e-5 is the ε14 by_hand = x / root_mean_square * gain15 print(f"root mean square {label}, before:", f"{root_mean_square.mean():.2f}")16 print(f"root mean square {label}, after:", f"{np.sqrt((by_hand ** 2).mean(1)).mean():.2f}")17 assert np.allclose(by_hand, transformer.rms_norm(x, gain), atol=1e-4) # the library's function, to 4 decimal places18 print(f"same as lib.transformer.rms_norm, {label}:", "yes")19 # layer normalization would subtract the mean first; RMSNorm doesn't, so this mean need not be zero20 print(f"largest mean of a position's numbers after, {label}:", f"{np.abs(by_hand.mean(1)).max():.2f}")
python rms-norm-by-hand.pyroot mean square entering block 1, before: 0.10
root mean square entering block 1, after: 1.00
same as lib.transformer.rms_norm, entering block 1: yes
largest mean of a position's numbers after, entering block 1: 0.23
root mean square leaving the last block, before: 1.16
root mean square leaving the last block, after: 1.00
same as lib.transformer.rms_norm, leaving the last block: yes
largest mean of a position's numbers after, leaving the last block: 0.26
正規化使兩端達到相同的尺度。均方根在進入第一個區塊時約為 0.10,離開最後一個區塊時為 1.16,正規化之後兩端都是 1.00;手算的結果在兩端都與函式庫的結果相符。RMSNorm 只縮放而不置中:某個位置的數值在正規化之後平均仍可能偏離零,此處最多可達 0.26。這就是它相較於層正規化所省下的步驟。
因此,雖然殘差流在各層之間變長了許多倍,每一個注意力層與前饋層讀到的向量,尺度都相同,再乘上各自學到的增益。
實用原則: 在區塊的每個部分之前進行正規化,而不是之後,並在輸出層之前再對殘差流正規化一次。除非要與使用層正規化的模型相符,否則請使用 RMSNorm。載入其他模型的權重時,請先確認它使用哪一種正規化、用在何處,以及 ε 為多少,再做其他事:細節不一致的範例說明了小小的不一致會造成什麼代價。
此部分尚未翻譯,因此以英文呈現。
Transform each position with a gated feed-forward layer
Attention moves information between positions; the feed-forward layer transforms it, at each position separately. It's the same idea as chapter 5's hidden layer: widen the vector, apply a function that isn't a straight line, and narrow the result back to the stream's width. The store's model widens 128 numbers to 352.
The form the LLaMA models use is gated. A gated feed-forward layer computes two widened vectors from the input, passes one through a function called SiLU, and multiplies the two number by number, so the first decides how much of the second gets through (Shazeer, 2020):
FFN(x) = (SiLU(x W_gate) ⊙ (x W_up)) W_down
where ⊙ multiplies number by number, and SiLU(z) = z · σ(z), with σ the logistic function from chapter 4. SiLU passes large positive numbers almost unchanged and turns large negative ones to nearly zero, smoothly. Shazeer compared several gated forms with the plain one, at the same number of parameters, and found the gated ones predicted better; this form is called SwiGLU.
What does the gate shut? Compute SiLU at three points, then run the first block's real feed-forward layer on the same validation product and count, at each position, how many of its hidden units the gate nearly shuts:
1"""Computes SiLU at three points, then runs block 1's feed-forward layer on one validation product and counts the hidden2units its gate nearly shuts at each position. The prose's point rests on the three SiLU values and on that count."""3import numpy as np45from lib import transformer6from lib.products import product_texts78print("silu(-3), silu(0), silu(3):", ", ".join(f"{v:.2f}" for v in transformer.silu(np.array([-3.0, 0.0, 3.0]))))910ids, _ = product_texts("validation", 1)[0] # the first validation product; its <sep> position isn't needed11heads = transformer.load()[2]["settings"]["heads"] # load returns (weights, vocabulary, manifest): index 2 is the manifest12first = transformer.layer_weights(0) # block 1's weights13x = transformer.load()[0]["embedding"][np.array(ids)] # index 0 is the weights: the embeddings, the stream entering block 114attended, _ = transformer.attention(transformer.rms_norm(x, first["norm1"]), first, heads)15normalized = transformer.rms_norm(x + attended, first["norm2"]) # what block 1's feed-forward layer reads16gate = transformer.silu(normalized @ first["gate"]) # how much of each hidden unit to let through17up = normalized @ first["up"] # what each hidden unit carries18output = (gate * up) @ first["down"] # gated, then projected back to the stream's width19shut = (np.abs(gate) < 0.05).sum(1) # per position; 0.05 is the threshold for "nearly shut"20print("hidden units:", gate.shape[1])21print("hidden units nearly shut at a position (gate under 0.05 in size), on average:", f"{shut.mean():.0f}")22print("fewest and most nearly shut at a position:", f"{shut.min()} and {shut.max()}")23print("positions:", len(ids))24print("output shape:", output.shape)
python gated-feed-forward.pysilu(-3), silu(0), silu(3): -0.14, 0.00, 2.86
hidden units: 352
hidden units nearly shut at a position (gate under 0.05 in size), on average: 49
fewest and most nearly shut at a position: 33 and 67
positions: 133
output shape: (133, 128)
SiLU of −3, 0 and 3 gives -0.14, 0.00, 2.86: nearly zero for the negative number, nearly unchanged for the positive one.
Which units the gate shuts depends on the input. At a typical position it nearly shuts 49 of the first block's 352 hidden units, and the count ranges between 33 and 67 over the positions. So each position uses its own mix of them. The output has one vector per position at the stream's width again, (133, 128), ready to add.
The feed-forward layers hold most of a block's parameters: in the store's model, three matrices of 128 by 352, against attention's four of 128 by 128. Geva and colleagues found that in trained models these layers work like stores of patterns: each hidden unit responds to particular patterns in the text, and pushes the prediction towards particular next tokens (Geva et al., 2021).
Working rule: Use a gated feed-forward layer with SiLU. Set its width so its three matrices hold as many parameters as a plain layer four times the stream's width, the original transformer's ratio, 2,048 for 512 (Vaswani et al., 2017, section 3.3): that's about 8/3 times the width. SmolLM2-360M uses exactly 8/3, 2,560 for 960. The store's model rounds 8/3 of its width up to a multiple of 32, a rule of thumb that keeps matrix sizes convenient for hardware. When you load another model, check whether its feed-forward layer is gated, since a gated layer has three matrices where a plain one has two.
此部分尚未翻譯,因此以英文呈現。
Give attention positions with rotations
A weighted average ignores the order of what it averages, as chapter 6 showed, so attention gets at most a weak sense of position from the causal mask alone. The original transformer added a vector for each position to each token's embedding (Vaswani et al., 2017, section 3.5). The LLaMA models instead use rotary position embeddings, RoPE, which work inside attention, at each layer (Su et al., 2021; Touvron et al., 2023, section 2.2). Before scoring, RoPE rotates each query and key by angles that grow with its position. The vector's d numbers are taken in pairs, number i with number i + d/2 in the store's model, and each pair (a, b) is treated as a point on a plane and turned around the origin. At position p, pair i turns by the angle p·θᵢ, where θᵢ = 10000^(−i/(d/2)):
(a, b) → (a cos pθᵢ − b sin pθᵢ, a sin pθᵢ + b cos pθᵢ)
So the first pairs turn fast, a full radian per position, and the last pairs slowly. The line angles = ... in lib.transformer.rope computes p·θᵢ for every position and pair, and the line after it applies the rotation.
Rotation has two properties that suit attention. It doesn't change a vector's length, so it adds position without distorting content. And if a query at position m is turned by mθᵢ and a key at position n by nθᵢ, their dot product depends only on the difference, (m − n)θᵢ: the score depends on how far apart they are, not on where they sit in the text. A head can learn to look 3 tokens back, and it works the same at the start of a name and deep in a description.
Rotate a random query and key of 32 numbers, the size of the store's heads, for positions 0 to 19, and compare scaled scores, as chapter 6 defines them, at three pairs, which tests that:
1"""Rotates one random query and key for each of 20 positions, checks that the rotation keeps their lengths, and scores2three pairs of positions. The prose's point rests on the two scores for pairs 3 apart, which match, and the score for3the pair 6 apart, which doesn't."""4import numpy as np56from lib.transformer import rope78rng = np.random.default_rng(2026) # a fixed seed: the same random query and key on every run9size = 32 # 32 numbers: the store's width of 128 split across 4 heads10query, key = rng.normal(0, 1, size), rng.normal(0, 1, size)11positions = 2012rotated_q = rope(np.tile(query, (positions, 1))) # the same query, rotated for each position 0 to 1913rotated_k = rope(np.tile(key, (positions, 1)))14kept = np.allclose(np.linalg.norm(rotated_q, axis=1), np.linalg.norm(query)) and \15 np.allclose(np.linalg.norm(rotated_k, axis=1), np.linalg.norm(key)) # rotation turns a vector, never changes its length16assert kept17print("length kept by the rotation, for the query and the key:", "yes")18for i, j in [(2, 5), (12, 15), (2, 8)]: # (key position, query position): the first two are 3 apart, the third 619 print(f"score with the key at {i} and the query at {j}:", f"{rotated_q[j] @ rotated_k[i] / np.sqrt(size):.3f}") # scaled, as chapter 6 defines it
python rope-by-hand.pylength kept by the rotation, for the query and the key: yes
score with the key at 2 and the query at 5: 0.217
score with the key at 12 and the query at 15: 0.217
score with the key at 2 and the query at 8: -0.765
Pairs the same distance apart score the same, wherever they sit. The key at position 2 and the query at 5 score 0.217, and the key at 12 and the query at 15 score 0.217, the same, because both pairs are 3 apart. The key at 2 and the query at 8, 6 apart, score -0.765: the same query and key, scored differently because they're further apart. The rotation keeps both vectors' lengths, which the example asserts.
RoPE has no learned parameters: the angles are fixed, and the model learns queries and keys that use them. The store's model applies it to the queries and keys of every head in every layer, which gives chapter 6's previous-token heads a direct way to find the previous token, alongside the weak signal the mask provides.
Working rule: Use rotary position embeddings, applied to queries and keys, never values. When you load another model's weights, check how its RoPE pairs the numbers and what base it uses. Some implementations pair neighbouring numbers; others pair the first half with the second, as lib.transformer.rope does and as SmolLM2's configuration states with "rope_interleaved": false. SmolLM2 uses a base of 100,000 where the store's model uses 10,000.
將輸出層與嵌入綁定
所有元件都已建構完成,整個模型短到可以一次讀完。lib.transformer.block 是單一的區塊,與本章前面的描述完全一致:
def block(x: np.ndarray, layer_weights: dict[str, np.ndarray], heads: int, positions=rope):
"""One transformer block: attention, then the gated feed-forward layer, each added to the residual stream."""
attended, weights = attention(rms_norm(x, layer_weights["norm1"]), layer_weights, heads, positions)
x = x + attended
normalized = rms_norm(x, layer_weights["norm2"])
x = x + (silu(normalized @ layer_weights["gate"]) * (normalized @ layer_weights["up"])) @ layer_weights["down"]
return x, weights
lib.transformer.run 依序讓每個詞元的嵌入通過各個區塊,而 lib.transformer.forward 中的這幾行則把最後的資料流轉換成詞彙表中每個詞元各一個 logit,也就是第 5 章的 softmax 再轉成機率的那組分數:
streams, kept = run(ids, name, positions, skip)
weights, _, _ = load(name)
logits = rms_norm(streams[-1], weights["final_norm"]) @ weights["embedding"].T
最後一行還包含另一個概念,即權重綁定(weight tying):輸出層不另外學習自己的矩陣,而是重用嵌入表,只是把它轉置過來。某個詞元的 logit,就是最終向量與該詞元嵌入的內積。Press 與 Wolf 發現,綁定除了節省輸出層的參數之外,也能降低困惑度(Press and Wolf, 2017)。SmolLM2 同樣採用綁定(其設定檔中為 "tie_word_embeddings": true),因為對這種規模的模型而言,另設一個輸出層會占整體相當大的比例。參數量顯示了綁定替商店模型省下多少參數。
實用原則: 在小型模型中將輸出層與嵌入綁定;載入他人的模型時,先確認它是否已綁定,再去尋找獨立的輸出矩陣。讓自己的前向傳遞保持簡短,短到能整段讀完,就像 lib.transformer 那樣,如此後續各節要測試的每個細節都看得見。
對照 PyTorch 檢查模型
PyTorch 已在 recordings/transformer.py 中訓練過商店的模型一次,做法與訓練第 5 章的模型相同(參見以 PyTorch 訓練),而且該腳本也儲存了 PyTorch 自己對某一個驗證商品在每個位置算出的 logits。訓練腳本使用的是 PyTorch 自己的 RMSNorm、因果注意力(causal attention)與 SiLU,因此將它們與 lib.transformer 比較,檢查的是 NumPy 程式碼對照函式庫的實作,而不是對照它自己的第二份副本。RoPE 是例外:PyTorch 沒有內建版本,所以兩個程式都是自行寫出,這項檢查只能確認兩者一致。rope-by-hand 範例則改為檢查它的各項性質。
為了測試整個前向傳遞,請在 NumPy 中重新計算該商品的 logits,並將每個位置與 PyTorch 的結果比較:
1"""Recomputes the store's model's logits in NumPy for the validation product the training script saved, and checks every2one against PyTorch's to four decimal places. The prose's point rests on the count of logits compared, and on the3assertion that they all match."""4import numpy as np56from lib import transformer78_, _, manifest = transformer.load()9check = np.load("data/recorded/transformer/check.npz") # one validation product's ids, and PyTorch's logits1011ours = transformer.forward(list(check["ids"])) # forward takes a list of token ids, not an array12print("PyTorch version:", manifest["torch"])13print("positions compared:", len(check["ids"]))14print("logits compared:", f"{ours.size:,}")15assert np.abs(ours - check["logits"]).max() < 1e-4 # 1e-4 is four decimal places16print("same as PyTorch to 4 decimal places:", "yes")
python transformer-same-as-pytorch.pyPyTorch version: 2.14.1+cpu
positions compared: 133
logits compared: 550,886
same as PyTorch to 4 decimal places: yes
NumPy 模型重現了 PyTorch 全部 550,886 個 logits,涵蓋 133 個位置,精確到小數點後四位,所用的 PyTorch 版本為 2.14.1+cpu。由於這項檢查涵蓋整段文字,因此除了正規化(norm)、閘門(gate)與共用權重的輸出層之外,也涵蓋了依賴位置的部分,即遮罩與旋轉。
實用原則: 請以完整文字、而非單一位置,對照訓練函式庫的輸出來檢查自己的前向傳遞。函式庫對某個部分有自己的實作時,就用它來訓練,讓檢查比較的是兩個獨立的實作;函式庫沒有時,例如此處的 RoPE,則以手動方式測試該部分的性質。
拿掉一個區塊,看它做了什麼
模型已經檢查過了;接下來衡量各個區塊的貢獻。由於每個區塊都是把結果加到殘差流上,拿掉任何一個區塊,模型仍然可以運作:殘差流原封不動地穿過該處,下一個區塊照常讀取。把某個部分拿掉,可以進行一個簡單的實驗,稱為消融(ablation):移除一個部分,再衡量模型損失了多少。這個實驗顯示每個區塊對已訓練模型的預測貢獻多少,這是光看權重無法得知的。
拿掉哪個區塊的傷害最大?先自己猜一個,再對前 100 筆驗證集商品的名稱與描述評分,總共 7,624 次預測,分別用完整模型,以及依序各拿掉一個區塊的模型:
1"""Scores the names and descriptions of the first 100 validation products with the full model, and with each block left2out in turn, reporting perplexity and the top token's average probability. The prose's point rests on the perplexity3with each block left out, set against the full model's."""4import numpy as np56from lib import transformer7from lib.products import product_texts89layers = transformer.load()[2]["settings"]["layers"] # load returns (weights, vocabulary, manifest): index 2 is the manifest10texts = [ids for ids, _ in product_texts("validation", 100)] # the <sep> positions aren't needed: each text is scored whole111213def measure(skip=None):14 """Perplexity per token, and the average probability of the top token, with one block left out."""15 surprise, top, count = 0.0, 0.0, 016 for ids in texts:17 p = transformer.softmax(transformer.forward(ids, skip=skip)[:-1]) # [:-1] drops the last position: nothing follows it18 surprise -= np.log(p[np.arange(len(ids) - 1), ids[1:]]).sum() # position i predicts token i + 119 top += p.max(1).sum()20 count += len(ids) - 121 return np.exp(surprise / count), top / count # perplexity: exp of the average surprise per token222324print("tokens scored, names and descriptions:", f"{sum(len(ids) - 1 for ids in texts):,}")25for skip in [None] + list(range(layers)): # None keeps all four blocks: the full model26 label = "all blocks" if skip is None else f"without block {skip + 1}" # skip counts blocks from 0, so block skip + 1 is left out27 perplexity, top = measure(skip)28 print(f"{label}:", f"{perplexity:,.1f}")29 print(f"{label}, average probability of the top token:", f"{top:.2f}")
python skip-a-block.pytokens scored, names and descriptions: 7,624
all blocks: 24.3
all blocks, average probability of the top token: 0.42
without block 1: 17,896.9
without block 1, average probability of the top token: 0.21
without block 2: 77.8
without block 2, average probability of the top token: 0.29
without block 3: 64.3
without block 3, average probability of the top token: 0.34
without block 4: 77.3
without block 4, average probability of the top token: 0.37
拿掉區塊 2、3 或 4,每個詞元的困惑度都會增加一倍以上,從四個區塊齊全時的 24.3,變成 77.8、64.3 與 77.3:後面的每個區塊都貢獻了模型能力的一份,沒有一個是可有可無的。這個困惑度同時涵蓋名稱與描述,因此不能拿來與第 3 章的基準線或第 5 章的視窗模型相比,那兩者只對驗證集的名稱評分。第 8 章會進行這項比較。
拿掉區塊 1 的失敗方式則不同。困惑度升高到 17,896.9,比在全部 4,142 個詞元之間平均猜測還差,平均猜測的分數會是 4,142。
原因在於後面的區塊預期看到什麼。它們是在區塊 1 已經寫入的殘差流上訓練的,少了區塊 1,它們讀到的向量就和訓練時見過的都不一樣。它們的預測變得不那麼有把握,最高機率詞元的平均機率從 0.42 降到 0.21,而且把這些機率放在錯誤的詞元上。區塊 1 把原始嵌入轉換成後續一切所依賴的形式,做法之一是從目前為止的文字中蒐集上下文,就像第 6 章中第一層的三個注意力頭所做的那樣。已訓練模型中的區塊並不是彼此獨立的模組:每一個都依賴前面的區塊寫入了什麼。
實用原則: 想知道模型的某個部分做了什麼,就把它移除並加以衡量,而不是從權重去推斷。預期第一個區塊最重要,因為後面的一切都建立在它之上;除此之外就要實際衡量,因為在這裡,第二個與最後一個區塊的重要性大致相當,而且都高於第三個。解讀移除某個部分所造成的小幅損失時要謹慎:其餘部分是與它一起訓練出來的,所以這項測量說明的是已訓練模型依賴什麼,而不是這個部分單獨能做什麼。
量測細節不一致的代價
區塊的每一項細節都必須與其所載入的權重相符,而不一致並不會自行示警:模型照樣會產生機率。載入其他模型的權重時,最常出錯的細節在 RoPE,因為各種實作在數字的配對方式與基底(base)上各不相同,例如 SmolLM2 的基底就與本店模型的不同。
一次只弄錯 RoPE 的一項細節,便能看出各自的代價:以同樣的 100 筆驗證集商品,分別用訓練所得的權重、改以相鄰數字配對而非前後兩半配對、基底改為 SmolLM2 所用的 100,000 而非本店的 10,000,以及完全不使用 RoPE 來計分:
1"""Scores the same 100 validation products with the trained weights, changing one detail of RoPE at a time: how it pairs2numbers, its base, and whether it is applied at all. The prose's point rests on each variant's perplexity, printed3beside the trained model's."""4import numpy as np56from lib import transformer7from lib.products import product_texts89texts = [ids for ids, _ in product_texts("validation", 100)] # the <sep> positions aren't needed: each text is scored whole101112def neighbours(x, base=10000.0):13 """RoPE that pairs neighbouring numbers, (0, 1), (2, 3) and so on, instead of the first half with the second."""14 half = x.shape[-1] // 215 rotated = transformer.rope(np.concatenate([x[..., 0::2], x[..., 1::2]], axis=-1), base) # even numbers, then odd: rope pairs halves, so these are neighbours16 out = np.empty_like(x)17 out[..., 0::2], out[..., 1::2] = rotated[..., :half], rotated[..., half:] # each rotated number back where its dimension was18 return out192021for label, positions in [("as trained", transformer.rope),22 ("RoPE pairing neighbouring numbers", neighbours),23 ("RoPE with base 100,000", lambda x: transformer.rope(x, base=100000.0)), # 100,000 is SmolLM2's base24 ("no RoPE", lambda x: x)]: # no RoPE: the queries and keys pass through unrotated25 print(f"perplexity, {label}:", f"{transformer.perplexity(texts, positions=positions):,.1f}")
python mismatched-details.pyperplexity, as trained: 24.3
perplexity, RoPE pairing neighbouring numbers: 112.2
perplexity, RoPE with base 100,000: 26.4
perplexity, no RoPE: 182.0
其中兩種不一致,只要看第一段生成的文字就會被發現。改以相鄰數字配對而非前後兩半配對,會使困惑度(perplexity)由 24.3 升至 112.2;完全省略 RoPE,則升至 182.0。
基底的錯誤最危險。基底為 100,000 時,困惑度為 26.4,只比 24.3 略差,既沒有錯誤訊息,輸出看起來也流暢,這是可能被直接上線的錯誤。
與 SmolLM2 的另一項差異是分組查詢注意力(grouped-query attention):它的 15 個查詢頭共用 5 組鍵與值,因此鍵矩陣與值矩陣的大小只有查詢矩陣的三分之一(Ainslie et al., 2023)。把這樣的權重載入期望每個頭都有各自鍵與值的模型,會因形狀不符而報錯,這正是人們希望遇到的那種不一致。
**實用原則:**載入其他模型的權重時,要逐行對照其設定與你的程式碼:正規化及其 ε、RoPE 的配對方式與基底、前饋層的閘控、輸出權重是否共用,以及有多少個頭共用鍵與值。接著用你熟悉的文字計分,並與原始實作的輸出比對;損失只是略為增加而沒有任何錯誤,才是最該提防的情況。
此部分尚未翻譯,因此以英文呈現。
Count where the parameters go
With every part built, count what the model learns, and where. A model's parameters are its capacity to learn, and where they sit says what it spends that capacity on. The trained model's count, by part:
1"""Counts the store's model's parameters by part from its saved weights, and prints its settings. The prose's point2rests on the total, the feed-forward count and the embedding's share of it."""3from lib import transformer45weights, vocabulary, manifest = transformer.load()6settings = manifest["settings"]7# one table serves as input and output, so the embedding is counted once; a weight's group is the last part of its name8groups = {"embedding (shared with the output)": ["embedding"], "attention": ["q", "k", "v", "o"],9 "feed-forward": ["gate", "up", "down"], "normalization": ["norm1", "norm2", "final_norm"]}10for group, parts in groups.items():11 count = sum(v.size for k, v in weights.items() if k.split(".")[-1] in parts) # a weight's name ends in its part: layer3.q is attention's q12 print(f"{group}:", f"{count:,}")13print("total:", f"{sum(v.size for v in weights.values()):,}")14print("embedding's share of the total:", f"{weights['embedding'].size / sum(v.size for v in weights.values()):.0%}")15print("layers:", settings["layers"])16print("width:", settings["dimensions"]) # the width of the residual stream, which the manifest calls dimensions17print("heads:", settings["heads"])18print("feed-forward width:", settings["feed_forward"])19print("longest text, in tokens:", settings["longest"]) # longer texts were cut to this many tokens20print("vocabulary:", f"{len(vocabulary):,}") # the store tokenizer's tokens, plus the four markers
python transformer-parameters.pyembedding (shared with the output): 530,176
attention: 262,144
feed-forward: 540,672
normalization: 1,152
total: 1,334,144
embedding's share of the total: 40%
layers: 4
width: 128
heads: 4
feed-forward width: 352
longest text, in tokens: 160
vocabulary: 4,142
The model has 1,334,144 parameters, slightly fewer than the 1,394,732 of chapter 5's window model. The two are about the same size but spend their parameters differently. The window model put about three quarters of its parameters in its output layer, and could see four tokens back. The transformer puts most of its parameters in its blocks, 540,672 of them in feed-forward layers alone. It shares its embeddings with its output, and can see every earlier token in a text of up to 160 tokens.
The embedding table, shared with the output layer, holds 530,176 parameters, 40% of the total, because it has a row of 128 numbers for each of the 4,142 tokens. Those are the store tokenizer's vocabulary plus four markers: start, separator, end, and padding, which fills out short texts in a training batch. Chapter 8 explains how the model was trained, and measures it against the baseline.
Working rule: Count parameters by part whenever you design or load a model. At small sizes, the embedding table is a large share of the total, so tie it to the output layer, and choose the vocabulary size with that cost in mind, as chapter 2's choice of merges did.
正式環境中會出什麼問題
Transformer 的各個部件必須完全吻合,而多數問題出在某個「幾乎吻合」的部件上:
- 細節不一致可能悄悄造成失誤。 正規化層(norm)放在哪裡、RoPE 如何配對數值及其基底(base)、前饋層是否採用閘控(gated)、輸出是否與嵌入共用權重(tied),全都必須與訓練好的權重一致。有些不一致會立刻產生無意義的輸出;也有些不一致,例如 RoPE 的基底(此處已有量測),只會損失一點準確度,而且不會出現任何錯誤。請以參考輸出來檢查。
- 超出訓練長度的文本未經測試。 RoPE 能為任何位置給出角度,但若模型是以最長 160 個 token 的文本訓練而成,它從未學過如何運用超出這個範圍的角度。在你實際量測過超出訓練長度的表現之前,請將訓練長度視為模型的上限。
- 較低精度下,大數值可能溢位。 位元數較少的格式在各個一般量級上維持相同的相對精度,但 float16 無法表示大於 65,504 的值;而在 8 位元格式中,少數幾個很大的離群值就會破壞其他所有值的精度(Dettmers et al., 2022)。就此處量測的產品而言,商店模型的串流(stream)最大值為 50.6,遠低於 float16 的上限;在以較低精度提供服務之前,請先檢查你的模型串流中的最大值。
- 將參考輸出與權重一併保存。 如第 4 章所說明,結果會因機器而異,因此請將權重、參考輸出與確切的訓練腳本放在一起,並在程式碼或函式庫有任何變動後重新執行檢查。
- 每個部件在每個位置都會執行。 前饋層在每個位置的成本相同,而注意力的成本則隨長度的平方成長,如第 6 章所示。在最佳化其中任何一項之前,請先針對你的文本量測哪一項占主要部分。
本章的決定
- **本書商店的 Transformer 採用 LLaMA 風格的區塊,**規模為玩具等級:每個部分之前先做 RMSNorm,注意力使用旋轉位置編碼,前饋層採用閘控 SiLU,輸出層與嵌入共用權重。它有 4 層,寬度為 128,有 4 個注意力頭,前饋寬度為 352。SmolLM2-360M 的組成部分種類相同,但規模較大,並使用分組查詢注意力(grouped-query attention),RoPE 基底為 100,000。
- **
lib.transformer中的 NumPy 前向傳遞是本書的參考實作,**並已在一項驗證用商品的每個位置上,與 PyTorch 自身的 RMSNorm、注意力和 SiLU 比對,結果一致到小數點後四位。之後每個使用此模型的範例都執行這個實作。 - **每個區塊都對殘差流做加法,而每個部分讀取的都是殘差流經過正規化的副本,**因此殘差流可以隨層數增長,而各部分看到的向量尺度仍然固定。
- **評斷各部分的方法,是量測模型在移除或改動該部分後的表現。**移除任何一個區塊,都會使本書商店的模型明顯變差,移除第一個區塊則使模型完全無用,因此本書把訓練完成的模型各區塊視為一個整體系統。RoPE 基底設錯只會造成小幅損失,也不會引發任何錯誤,所以載入的權重一律要以參考輸出檢查。
- **這個模型的大小與第 5 章的視窗模型相近,只是參數花在各區塊上,而不是輸出層。**與輸出層共用的嵌入表仍占整個模型的 40%,所以詞彙表大小依然是本書持續關注的成本。
本章,於本版次中
版次 2026.11.16範例於 2026年11 月 19 通過測試本章 8 個範例 個範例中有 8 個通過
譯自英文書,版次 2026.11.16。