The transformer block

읽는 데 약 50분분 걸린다, 11절에 8개 예제이(가) 있음.

2026.11.16판 예제 통과 2026년 11월 19 86개 예제 중 86개 통과 변경 사항

AI로 번역함. 영어 원문을 사람이 검증함. 우리가 책을 만드는 방식

영어판 2026.11.16판을 번역한 것이다. 원문 읽기

이 부분은 아직 번역되지 않았으므로 영어로 표시된다.

By the end of this chapter you can assemble a transformer block from its parts, joined by residual connections: attention, normalization, a gated feed-forward layer and position information. You can then stack blocks into a complete language model. You'll check each part on the store's trained model, check the whole model against PyTorch's own implementations, measure what each block and each detail contributes, and count where the parameters go.

Chapter 6 built attention: each position looks back at earlier ones and takes a weighted average of what it finds. Attention alone isn't a language model, and the store, an online furniture and home-goods shop, needs more than attention. It moves information between positions but transforms little of it, and it has almost no sense of word order. Layers of it stacked directly also degrade: without the other parts of a block, the output of pure attention tends towards the same vector at every position, faster with every layer (Dong et al., 2021). The transformer block adds parts around attention that transform each position, give it word order and keep a deep stack from degrading (Vaswani et al., 2017).

This chapter builds the block in the form the LLaMA models popularized: normalization of each part's input with RMSNorm, a gated feed-forward layer and rotary position embeddings in every layer (Touvron et al., 2023, section 2.2). SmolLM2, whose tokenizer chapter 2 read, is built the same way. Its 360-million-parameter model has 960 dimensions, 32 layers and 15 heads, and differs from the store's model in two details covered below: its heads share keys and values, and its rotary positions turn more slowly (configuration).

The store's model, trained on the store's product text, is 4 of these blocks between an embedding table and an output layer, and chapter 6 read its heads. This book calls each block a layer of the model. At every position, each block does two things in order:

  1. normalize the vector, run attention, and add the result back to the vector;
  2. normalize again, run a feed-forward layer, and add that result back too.

이 부분은 아직 번역되지 않았으므로 영어로 표시된다.

Add each part's output to a residual stream

마지막으로 변경된 판: 2026.11.16판

In chapter 5's window model, each layer replaced its input: the hidden layer's output was all the output layer saw. A transformer block adds to its input instead. Each part computes something from the current vector and adds its output to it: x ← x + attention(x), then x ← x + feed-forward(x). Adding a part's output to its input, instead of replacing the input, is a residual connection. It comes from image models, where very deep stacks of layers trained well only once each layer added to its input instead of replacing it (He et al., 2016).

Two consequences follow. With the additions, the signal can pass "directly from one block to any other block", forwards and, as gradients, backwards, so a deep stack trains far more reliably than it would without them (He et al., 2016b). And every position's vector becomes a shared workspace that each part reads from and writes to: the token's embedding goes in at the bottom, each part adds what it computes, and the output layer reads the sum at the top. Elhage and colleagues call this running sum, each position's vector with every part's output added to it, the residual stream (Elhage et al., 2021).

Run the store's model on one validation product and, at every layer, measure the length of what attention adds, of what the feed-forward layer adds, and of the vectors after both additions, averaged over positions. A vector's length here is the square root of the sum of its squared numbers:

residual-stream.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 통과
"""Runs the store's model on one validation product, one block at a time, and prints how much attention and thefeed-forward layer add to the residual stream at each layer. The prose's point rests on the two lengths printedfirst and last: the vectors entering layer 1 and leaving the last layer."""import numpy as npfrom lib import transformerfrom lib.products import product_textsweights, _, manifest = transformer.load()ids, sep = product_texts("validation", 1)[0]          # the first validation product; sep, its <sep> position, isn't needed herex = weights["embedding"][np.array(ids)]               # the stream entering block 1: each token's embeddingprint("length of the vectors entering layer 1:", f"{np.linalg.norm(x, axis=1).mean():.2f}")for layer in range(manifest["settings"]["layers"]):    layer_weights = transformer.layer_weights(layer)    added_by_attention, _ = transformer.attention(transformer.rms_norm(x, layer_weights["norm1"]), layer_weights, manifest["settings"]["heads"])    x = x + added_by_attention    normalized = transformer.rms_norm(x, layer_weights["norm2"])    added_by_feed_forward = (transformer.silu(normalized @ layer_weights["gate"]) * (normalized @ layer_weights["up"])) @ layer_weights["down"]    x = x + added_by_feed_forward    print(f"layer {layer + 1}: attention adds {np.linalg.norm(added_by_attention, axis=1).mean():.2f}, "          f"feed-forward adds {np.linalg.norm(added_by_feed_forward, axis=1).mean():.2f}, "          f"vectors now {np.linalg.norm(x, axis=1).mean():.2f}")print("length of the vectors leaving the last layer:", f"{np.linalg.norm(x, axis=1).mean():.2f}")print("largest number in the stream:", f"{np.abs(x).max():.1f}")
출력python residual-stream.py
length of the vectors entering layer 1: 1.17
layer 1: attention adds 2.90, feed-forward adds 3.00, vectors now 4.65
layer 2: attention adds 2.05, feed-forward adds 2.90, vectors now 6.14
layer 3: attention adds 2.26, feed-forward adds 4.12, vectors now 8.46
layer 4: attention adds 4.08, feed-forward adds 6.33, vectors now 13.16
length of the vectors leaving the last layer: 13.16
largest number in the stream: 50.6

The stream grows through the layers, because each addition piles onto the last. It goes from an average length of 1.17 entering the first layer to 13.16 leaving the last. Every layer adds to it, and what attention adds is comparable in size to what the feed-forward layer adds, though the feed-forward layer adds more at every layer, and by more in the later ones: both parts do real work at every layer. A layer reading the long vectors near the top would need very differently scaled weights from one reading the short ones at the bottom, unless something rescales what each layer reads; the next section does.

Working rule: Write every block as additions to the residual stream, never as replacements, and keep the stream's width the same through the whole model so the additions line up. When you debug a deep model, measure what each part adds: a part that adds almost nothing is doing almost nothing.

각 부분 앞에서 정규화하라

마지막으로 변경된 판: 2026.11.11판

잔차 스트림은 이전 절에서 측정한 대로 층을 지날수록 커지므로, 각 부분은 스트림의 크기를 조정한 사본을 읽어야 한다. 정규화(normalization)는 부분이 벡터를 읽기 전에 벡터를 표준 크기로 조정하여, 스트림이 짧든 길든 그 부분의 가중치가 똑같이 동작하게 한다. 원래의 트랜스포머는 덧셈이 끝날 때마다 층 정규화(layer normalization)로 정규화했는데, 이는 각 벡터의 평균을 빼고 표준편차로 나눈다(Ba et al., 2016). Xiong과 동료들은 대신 각 부분 앞에서 정규화하면 학습이 더 안정된다는 것을 보였다. 각 부분이 읽는 사본만 크기가 조정되고 스트림 자체는 조정되지 않기 때문이다(Xiong et al., 2020). 그리고 RMSNorm은 평균을 없앤다. 각 벡터를 제곱평균제곱근, 곧 각 수의 제곱을 평균한 값의 제곱근으로 나누고, 각 차원에 학습된 이득(gain)을 곱한다(Zhang and Sennrich, 2019).

RMSNorm(x) = x / √(mean(x²) + ε) · g

여기서 ε은 나눗셈을 안전하게 하는 아주 작은 수로 이 책에서는 10⁻⁵이고, g는 이득이며 1에서 시작한다. Zhang과 Sennrich는 RMSNorm이 층 정규화와 거의 같은 정확도를 내면서 더 빠르다는 것을 발견했다. 상점의 모델은 어텐션 앞, 피드포워드 층 앞, 그리고 출력층 앞에서 한 번 더 RMSNorm을 사용한다.

같은 검증 상품의 스트림을 모델의 양 끝, 곧 첫 번째 블록에 들어가는 지점과 마지막 블록을 나오는 지점에서 이득을 1로 둔 채 직접 정규화하고, 모든 위치의 벡터를 lib.transformer.rms_norm의 결과와 비교하라.

rms-norm-by-hand.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 통과
"""Normalizes one validation product's residual stream by hand, entering the first block and leaving the last, andasserts that the result matches the shared module's rms_norm. The prose's point rests on the root mean square beforeand after normalizing, and on the largest mean of a position's numbers, which RMSNorm does not remove."""import numpy as npfrom lib import transformerfrom lib.products import product_textsids, _ = product_texts("validation", 1)[0]            # the first validation product; its <sep> position isn't neededstreams, _ = transformer.run(ids)                      # the residual stream entering block 1 and leaving each blockgain = np.ones(streams[0].shape[1])                    # the learned gain set to 1, to show the normalization alonefor label, x in [("entering block 1", streams[0]), ("leaving the last block", streams[-1])]:    root_mean_square = np.sqrt((x ** 2).mean(1, keepdims=True) + 1e-5)   # one per position; 1e-5 is the ε    by_hand = x / root_mean_square * gain    print(f"root mean square {label}, before:", f"{root_mean_square.mean():.2f}")    print(f"root mean square {label}, after:", f"{np.sqrt((by_hand ** 2).mean(1)).mean():.2f}")    assert np.allclose(by_hand, transformer.rms_norm(x, gain), atol=1e-4)   # the library's function, to 4 decimal places    print(f"same as lib.transformer.rms_norm, {label}:", "yes")    # layer normalization would subtract the mean first; RMSNorm doesn't, so this mean need not be zero    print(f"largest mean of a position's numbers after, {label}:", f"{np.abs(by_hand.mean(1)).max():.2f}")
출력python rms-norm-by-hand.py
root mean square entering block 1, before: 0.10
root mean square entering block 1, after: 1.00
same as lib.transformer.rms_norm, entering block 1: yes
largest mean of a position's numbers after, entering block 1: 0.23
root mean square leaving the last block, before: 1.16
root mean square leaving the last block, after: 1.00
same as lib.transformer.rms_norm, leaving the last block: yes
largest mean of a position's numbers after, leaving the last block: 0.26

정규화는 양 끝을 같은 크기로 맞춘다. 제곱평균제곱근은 평균적으로 첫 번째 블록에 들어갈 때 0.10, 마지막 블록을 나올 때 1.16이며, 정규화한 뒤에는 양쪽 모두 1.00이다. 직접 계산한 결과는 양 끝에서 라이브러리의 결과와 일치한다. RMSNorm은 크기를 조정할 뿐 중심을 맞추지는 않는다. 그래서 한 위치의 수들의 평균이 정규화 뒤에도 0에서 벗어날 수 있으며, 여기서는 최대 0.26까지 벗어난다. 이것이 층 정규화에 비해 RMSNorm이 덜어 낸 단계이다.

따라서 스트림이 층을 지나며 몇 배로 길어졌더라도, 모든 어텐션 층과 피드포워드 층은 같은 크기의 벡터를 읽었고, 그 벡터에 자신이 학습한 이득이 곱해진다.

작업 규칙: 블록의 각 부분 뒤가 아니라 앞에서 정규화하고, 출력층 앞에서 스트림을 한 번 더 정규화하라. 층 정규화를 쓰는 모델에 맞추는 경우가 아니라면 RMSNorm을 사용하라. 다른 모델의 가중치를 불러올 때는 무엇보다 먼저 그 모델이 어떤 정규화를 어디에서 어떤 ε으로 사용하는지 확인하라. 불일치한 세부 사항 예제는 작은 불일치가 어떤 대가를 치르게 하는지 보여 준다.

이 부분은 아직 번역되지 않았으므로 영어로 표시된다.

Transform each position with a gated feed-forward layer

마지막으로 변경된 판: 2026.11.16판

Attention moves information between positions; the feed-forward layer transforms it, at each position separately. It's the same idea as chapter 5's hidden layer: widen the vector, apply a function that isn't a straight line, and narrow the result back to the stream's width. The store's model widens 128 numbers to 352.

The form the LLaMA models use is gated. A gated feed-forward layer computes two widened vectors from the input, passes one through a function called SiLU, and multiplies the two number by number, so the first decides how much of the second gets through (Shazeer, 2020):

FFN(x) = (SiLU(x W_gate) ⊙ (x W_up)) W_down

where ⊙ multiplies number by number, and SiLU(z) = z · σ(z), with σ the logistic function from chapter 4. SiLU passes large positive numbers almost unchanged and turns large negative ones to nearly zero, smoothly. Shazeer compared several gated forms with the plain one, at the same number of parameters, and found the gated ones predicted better; this form is called SwiGLU.

What does the gate shut? Compute SiLU at three points, then run the first block's real feed-forward layer on the same validation product and count, at each position, how many of its hidden units the gate nearly shuts:

gated-feed-forward.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 통과
"""Computes SiLU at three points, then runs block 1's feed-forward layer on one validation product and counts the hiddenunits its gate nearly shuts at each position. The prose's point rests on the three SiLU values and on that count."""import numpy as npfrom lib import transformerfrom lib.products import product_textsprint("silu(-3), silu(0), silu(3):", ", ".join(f"{v:.2f}" for v in transformer.silu(np.array([-3.0, 0.0, 3.0]))))ids, _ = product_texts("validation", 1)[0]            # the first validation product; its <sep> position isn't neededheads = transformer.load()[2]["settings"]["heads"]    # load returns (weights, vocabulary, manifest): index 2 is the manifestfirst = transformer.layer_weights(0)                   # block 1's weightsx = transformer.load()[0]["embedding"][np.array(ids)]  # index 0 is the weights: the embeddings, the stream entering block 1attended, _ = transformer.attention(transformer.rms_norm(x, first["norm1"]), first, heads)normalized = transformer.rms_norm(x + attended, first["norm2"])   # what block 1's feed-forward layer readsgate = transformer.silu(normalized @ first["gate"])    # how much of each hidden unit to let throughup = normalized @ first["up"]                          # what each hidden unit carriesoutput = (gate * up) @ first["down"]                   # gated, then projected back to the stream's widthshut = (np.abs(gate) < 0.05).sum(1)                    # per position; 0.05 is the threshold for "nearly shut"print("hidden units:", gate.shape[1])print("hidden units nearly shut at a position (gate under 0.05 in size), on average:", f"{shut.mean():.0f}")print("fewest and most nearly shut at a position:", f"{shut.min()} and {shut.max()}")print("positions:", len(ids))print("output shape:", output.shape)
출력python gated-feed-forward.py
silu(-3), silu(0), silu(3): -0.14, 0.00, 2.86
hidden units: 352
hidden units nearly shut at a position (gate under 0.05 in size), on average: 49
fewest and most nearly shut at a position: 33 and 67
positions: 133
output shape: (133, 128)

SiLU of −3, 0 and 3 gives -0.14, 0.00, 2.86: nearly zero for the negative number, nearly unchanged for the positive one.

Which units the gate shuts depends on the input. At a typical position it nearly shuts 49 of the first block's 352 hidden units, and the count ranges between 33 and 67 over the positions. So each position uses its own mix of them. The output has one vector per position at the stream's width again, (133, 128), ready to add.

The feed-forward layers hold most of a block's parameters: in the store's model, three matrices of 128 by 352, against attention's four of 128 by 128. Geva and colleagues found that in trained models these layers work like stores of patterns: each hidden unit responds to particular patterns in the text, and pushes the prediction towards particular next tokens (Geva et al., 2021).

Working rule: Use a gated feed-forward layer with SiLU. Set its width so its three matrices hold as many parameters as a plain layer four times the stream's width, the original transformer's ratio, 2,048 for 512 (Vaswani et al., 2017, section 3.3): that's about 8/3 times the width. SmolLM2-360M uses exactly 8/3, 2,560 for 960. The store's model rounds 8/3 of its width up to a multiple of 32, a rule of thumb that keeps matrix sizes convenient for hardware. When you load another model, check whether its feed-forward layer is gated, since a gated layer has three matrices where a plain one has two.

이 부분은 아직 번역되지 않았으므로 영어로 표시된다.

Give attention positions with rotations

마지막으로 변경된 판: 2026.11.16판

A weighted average ignores the order of what it averages, as chapter 6 showed, so attention gets at most a weak sense of position from the causal mask alone. The original transformer added a vector for each position to each token's embedding (Vaswani et al., 2017, section 3.5). The LLaMA models instead use rotary position embeddings, RoPE, which work inside attention, at each layer (Su et al., 2021; Touvron et al., 2023, section 2.2). Before scoring, RoPE rotates each query and key by angles that grow with its position. The vector's d numbers are taken in pairs, number i with number i + d/2 in the store's model, and each pair (a, b) is treated as a point on a plane and turned around the origin. At position p, pair i turns by the angle p·θᵢ, where θᵢ = 10000^(−i/(d/2)):

(a, b) → (a cos pθᵢ − b sin pθᵢ, a sin pθᵢ + b cos pθᵢ)

So the first pairs turn fast, a full radian per position, and the last pairs slowly. The line angles = ... in lib.transformer.rope computes p·θᵢ for every position and pair, and the line after it applies the rotation.

Rotation has two properties that suit attention. It doesn't change a vector's length, so it adds position without distorting content. And if a query at position m is turned by mθᵢ and a key at position n by nθᵢ, their dot product depends only on the difference, (m − n)θᵢ: the score depends on how far apart they are, not on where they sit in the text. A head can learn to look 3 tokens back, and it works the same at the start of a name and deep in a description.

Rotate a random query and key of 32 numbers, the size of the store's heads, for positions 0 to 19, and compare scaled scores, as chapter 6 defines them, at three pairs, which tests that:

rope-by-hand.py Python 3.12.15 · numpy 2.5.5 통과
"""Rotates one random query and key for each of 20 positions, checks that the rotation keeps their lengths, and scoresthree pairs of positions. The prose's point rests on the two scores for pairs 3 apart, which match, and the score forthe pair 6 apart, which doesn't."""import numpy as npfrom lib.transformer import roperng = np.random.default_rng(2026)                             # a fixed seed: the same random query and key on every runsize = 32                                                      # 32 numbers: the store's width of 128 split across 4 headsquery, key = rng.normal(0, 1, size), rng.normal(0, 1, size)positions = 20rotated_q = rope(np.tile(query, (positions, 1)))              # the same query, rotated for each position 0 to 19rotated_k = rope(np.tile(key, (positions, 1)))kept = np.allclose(np.linalg.norm(rotated_q, axis=1), np.linalg.norm(query)) and \    np.allclose(np.linalg.norm(rotated_k, axis=1), np.linalg.norm(key))   # rotation turns a vector, never changes its lengthassert keptprint("length kept by the rotation, for the query and the key:", "yes")for i, j in [(2, 5), (12, 15), (2, 8)]:                       # (key position, query position): the first two are 3 apart, the third 6    print(f"score with the key at {i} and the query at {j}:", f"{rotated_q[j] @ rotated_k[i] / np.sqrt(size):.3f}")   # scaled, as chapter 6 defines it
출력python rope-by-hand.py
length kept by the rotation, for the query and the key: yes
score with the key at 2 and the query at 5: 0.217
score with the key at 12 and the query at 15: 0.217
score with the key at 2 and the query at 8: -0.765

Pairs the same distance apart score the same, wherever they sit. The key at position 2 and the query at 5 score 0.217, and the key at 12 and the query at 15 score 0.217, the same, because both pairs are 3 apart. The key at 2 and the query at 8, 6 apart, score -0.765: the same query and key, scored differently because they're further apart. The rotation keeps both vectors' lengths, which the example asserts.

RoPE has no learned parameters: the angles are fixed, and the model learns queries and keys that use them. The store's model applies it to the queries and keys of every head in every layer, which gives chapter 6's previous-token heads a direct way to find the previous token, alongside the weak signal the mask provides.

Working rule: Use rotary position embeddings, applied to queries and keys, never values. When you load another model's weights, check how its RoPE pairs the numbers and what base it uses. Some implementations pair neighbouring numbers; others pair the first half with the second, as lib.transformer.rope does and as SmolLM2's configuration states with "rope_interleaved": false. SmolLM2 uses a base of 100,000 where the store's model uses 10,000.

이 부분은 아직 번역되지 않았으므로 영어로 표시된다.

Tie the output layer to the embeddings

마지막으로 변경된 판: 2026.11.16판

With all the parts built, the whole model is short enough to read in one piece. lib.transformer.block is one block, exactly as the chapter has described it:

def block(x: np.ndarray, layer_weights: dict[str, np.ndarray], heads: int, positions=rope):
    """One transformer block: attention, then the gated feed-forward layer, each added to the residual stream."""
    attended, weights = attention(rms_norm(x, layer_weights["norm1"]), layer_weights, heads, positions)
    x = x + attended
    normalized = rms_norm(x, layer_weights["norm2"])
    x = x + (silu(normalized @ layer_weights["gate"]) * (normalized @ layer_weights["up"])) @ layer_weights["down"]
    return x, weights

lib.transformer.run passes each token's embedding through the blocks in turn, and these lines of lib.transformer.forward turn the final stream into one logit per token in the vocabulary, the scores chapter 5's softmax turns into probabilities:

    streams, kept = run(ids, name, positions, skip)
    weights, _, _ = load(name)
    logits = rms_norm(streams[-1], weights["final_norm"]) @ weights["embedding"].T

The last line holds one more idea, weight tying: the output layer reuses the embedding table, turned on its side, instead of learning a matrix of its own. A token's logit is the dot product of the final vector with that token's embedding. Press and Wolf found that tying lowers perplexity as well as saving the output layer's parameters (Press and Wolf, 2017). SmolLM2 ties them too ("tie_word_embeddings": true in its configuration), since for a model of its size a separate output layer would be a large share of the whole. The parameter count shows what tying saves in the store's model.

Working rule: Tie the output layer to the embeddings in a small model, and check whether a model you load ties them before you look for a separate output matrix. Keep your own forward pass short enough to read whole, as lib.transformer is, so that every detail the next sections test is visible.

이 부분은 아직 번역되지 않았으므로 영어로 표시된다.

Check the model against PyTorch

마지막으로 변경된 판: 2026.11.11판

PyTorch trained the store's model once, in recordings/transformer.py, as it trained chapter 5's (see Train with PyTorch), and the script saved PyTorch's own logits for every position of one validation product. The training script uses PyTorch's own RMSNorm, causal attention and SiLU, so comparing them with lib.transformer checks the NumPy code against a library's, not against a second copy of itself. RoPE is the exception: PyTorch has no built-in version, so both programs write it out, and the check confirms only that they agree. The rope-by-hand example checks its properties instead.

To test the whole forward pass, recompute the saved product's logits in NumPy and compare every position with PyTorch's:

transformer-same-as-pytorch.py Python 3.12.15 · numpy 2.5.5 통과
"""Recomputes the store's model's logits in NumPy for the validation product the training script saved, and checks everyone against PyTorch's to four decimal places. The prose's point rests on the count of logits compared, and on theassertion that they all match."""import numpy as npfrom lib import transformer_, _, manifest = transformer.load()check = np.load("data/recorded/transformer/check.npz")      # one validation product's ids, and PyTorch's logitsours = transformer.forward(list(check["ids"]))               # forward takes a list of token ids, not an arrayprint("PyTorch version:", manifest["torch"])print("positions compared:", len(check["ids"]))print("logits compared:", f"{ours.size:,}")assert np.abs(ours - check["logits"]).max() < 1e-4           # 1e-4 is four decimal placesprint("same as PyTorch to 4 decimal places:", "yes")
출력python transformer-same-as-pytorch.py
PyTorch version: 2.14.1+cpu
positions compared: 133
logits compared: 550,886
same as PyTorch to 4 decimal places: yes

The NumPy model reproduces all 550,886 of PyTorch's logits, for 133 positions, to four decimal places, with PyTorch 2.14.1+cpu. Because the check covers a whole text, it covers the parts that depend on position, the mask and the rotations, as well as the norms, gates and tied output.

Working rule: Check your own forward pass against the training library's outputs on a full text, not a single position. Where the library has its own implementation of a part, train with it, so the check compares two independent implementations; where it doesn't, as with RoPE here, test the part's properties by hand.

이 부분은 아직 번역되지 않았으므로 영어로 표시된다.

Leave out a block to see what it does

마지막으로 변경된 판: 2026.11.16판

The model is checked; now measure what its blocks contribute. Because every block adds to the residual stream, a block can be left out and the model still runs: the stream passes through it unchanged, and the next block reads it as before. Leaving a part out allows a simple experiment, called an ablation: remove one part, and measure what the model loses. It shows how much each block contributes to the trained model's predictions, which reading its weights can't.

Which block's removal hurts most? Predict one, then score the names and descriptions of the first 100 validation products, 7,624 predictions in all, with the full model and with each block left out in turn:

skip-a-block.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 통과
"""Scores the names and descriptions of the first 100 validation products with the full model, and with each block leftout in turn, reporting perplexity and the top token's average probability. The prose's point rests on the perplexitywith each block left out, set against the full model's."""import numpy as npfrom lib import transformerfrom lib.products import product_textslayers = transformer.load()[2]["settings"]["layers"]     # load returns (weights, vocabulary, manifest): index 2 is the manifesttexts = [ids for ids, _ in product_texts("validation", 100)]   # the <sep> positions aren't needed: each text is scored wholedef measure(skip=None):    """Perplexity per token, and the average probability of the top token, with one block left out."""    surprise, top, count = 0.0, 0.0, 0    for ids in texts:        p = transformer.softmax(transformer.forward(ids, skip=skip)[:-1])   # [:-1] drops the last position: nothing follows it        surprise -= np.log(p[np.arange(len(ids) - 1), ids[1:]]).sum()       # position i predicts token i + 1        top += p.max(1).sum()        count += len(ids) - 1    return np.exp(surprise / count), top / count                             # perplexity: exp of the average surprise per tokenprint("tokens scored, names and descriptions:", f"{sum(len(ids) - 1 for ids in texts):,}")for skip in [None] + list(range(layers)):                # None keeps all four blocks: the full model    label = "all blocks" if skip is None else f"without block {skip + 1}"   # skip counts blocks from 0, so block skip + 1 is left out    perplexity, top = measure(skip)    print(f"{label}:", f"{perplexity:,.1f}")    print(f"{label}, average probability of the top token:", f"{top:.2f}")
출력python skip-a-block.py
tokens scored, names and descriptions: 7,624
all blocks: 24.3
all blocks, average probability of the top token: 0.42
without block 1: 17,896.9
without block 1, average probability of the top token: 0.21
without block 2: 77.8
without block 2, average probability of the top token: 0.29
without block 3: 64.3
without block 3, average probability of the top token: 0.34
without block 4: 77.3
without block 4, average probability of the top token: 0.37

Leaving out block 2, 3 or 4 more than doubles the perplexity per token, from 24.3 with all four blocks to 77.8, 64.3 and 77.3: each later block contributes a share of the model's skill, and none is dispensable. This perplexity covers names and descriptions together, so it can't be set beside the baseline of chapter 3 or the window model of chapter 5, which are scored on validation names alone. Chapter 8 makes that comparison.

Leaving out block 1 fails differently. Perplexity rises to 17,896.9, worse than guessing evenly among all 4,142 tokens, which would score 4,142.

The reason is in what the later blocks expect. They were trained on a stream that block 1 had already written to, and without it they read vectors unlike any they saw in training. Their predictions become less confident, with the top token's average probability falling from 0.42 to 0.21, and they put that probability on the wrong tokens. Block 1 turns raw embeddings into the form everything after it is built on, partly by gathering context from the text so far, as three of chapter 6's first-layer heads did. Blocks in a trained model aren't independent modules: each depends on what the earlier ones write.

Working rule: To learn what a part of a model does, remove it and measure, rather than inferring from its weights. Expect the first block to matter most, since everything later is built on it; beyond that, measure, because here the second and last blocks mattered about equally, and more than the third. Read a small loss from removing a part with care: the remaining parts were trained alongside it, so the measurement says what the trained model relies on, not what the part could do alone.

이 부분은 아직 번역되지 않았으므로 영어로 표시된다.

Measure what a mismatched detail costs

마지막으로 변경된 판: 2026.11.16판

Every detail of a block has to match the weights it runs with, and a mismatch doesn't announce itself: the model still produces probabilities. The details most often wrong when loading another model's weights are in RoPE, because implementations differ in how they pair numbers and in their base, as SmolLM2's differs from the store's model's.

One wrong detail of RoPE at a time shows what each costs: score the same 100 validation products with the trained weights, pairing neighbouring numbers instead of halves, a base of 100,000, SmolLM2's, instead of the store's 10,000, and no RoPE at all:

mismatched-details.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 통과
"""Scores the same 100 validation products with the trained weights, changing one detail of RoPE at a time: how it pairsnumbers, its base, and whether it is applied at all. The prose's point rests on each variant's perplexity, printedbeside the trained model's."""import numpy as npfrom lib import transformerfrom lib.products import product_textstexts = [ids for ids, _ in product_texts("validation", 100)]   # the <sep> positions aren't needed: each text is scored wholedef neighbours(x, base=10000.0):    """RoPE that pairs neighbouring numbers, (0, 1), (2, 3) and so on, instead of the first half with the second."""    half = x.shape[-1] // 2    rotated = transformer.rope(np.concatenate([x[..., 0::2], x[..., 1::2]], axis=-1), base)   # even numbers, then odd: rope pairs halves, so these are neighbours    out = np.empty_like(x)    out[..., 0::2], out[..., 1::2] = rotated[..., :half], rotated[..., half:]                 # each rotated number back where its dimension was    return outfor label, positions in [("as trained", transformer.rope),                         ("RoPE pairing neighbouring numbers", neighbours),                         ("RoPE with base 100,000", lambda x: transformer.rope(x, base=100000.0)),   # 100,000 is SmolLM2's base                         ("no RoPE", lambda x: x)]:          # no RoPE: the queries and keys pass through unrotated    print(f"perplexity, {label}:", f"{transformer.perplexity(texts, positions=positions):,.1f}")
출력python mismatched-details.py
perplexity, as trained: 24.3
perplexity, RoPE pairing neighbouring numbers: 112.2
perplexity, RoPE with base 100,000: 26.4
perplexity, no RoPE: 182.0

Two mismatches would be caught by the first sample of generated text. Pairing neighbouring numbers instead of halves raises the perplexity from 24.3 to 112.2, and leaving RoPE out raises it to 182.0.

The base is the dangerous one. At 100,000 the perplexity is 26.4, slightly worse than 24.3, with no error and fluent-looking output, a mistake that could ship.

The other difference from SmolLM2 is grouped-query attention: its 15 query heads share 5 sets of keys and values, so its key and value matrices are a third of the size of its query matrix (Ainslie et al., 2023). Loading such weights into a model that expects a key and value for every head fails with a shape error, which is the kind of mismatch you want.

Working rule: Check another model's configuration against your code, line by line, when you load its weights: normalization and its ε, RoPE's pairing and base, the feed-forward layer's gating, tied outputs, and how many heads share keys and values. Then score text you know against the original implementation's outputs; a small loss with no error is the case to fear.

이 부분은 아직 번역되지 않았으므로 영어로 표시된다.

Count where the parameters go

마지막으로 변경된 판: 2026.11.11판

With every part built, count what the model learns, and where. A model's parameters are its capacity to learn, and where they sit says what it spends that capacity on. The trained model's count, by part:

transformer-parameters.py Python 3.12.15 · numpy 2.5.5 통과
"""Counts the store's model's parameters by part from its saved weights, and prints its settings. The prose's pointrests on the total, the feed-forward count and the embedding's share of it."""from lib import transformerweights, vocabulary, manifest = transformer.load()settings = manifest["settings"]# one table serves as input and output, so the embedding is counted once; a weight's group is the last part of its namegroups = {"embedding (shared with the output)": ["embedding"], "attention": ["q", "k", "v", "o"],          "feed-forward": ["gate", "up", "down"], "normalization": ["norm1", "norm2", "final_norm"]}for group, parts in groups.items():    count = sum(v.size for k, v in weights.items() if k.split(".")[-1] in parts)   # a weight's name ends in its part: layer3.q is attention's q    print(f"{group}:", f"{count:,}")print("total:", f"{sum(v.size for v in weights.values()):,}")print("embedding's share of the total:", f"{weights['embedding'].size / sum(v.size for v in weights.values()):.0%}")print("layers:", settings["layers"])print("width:", settings["dimensions"])                  # the width of the residual stream, which the manifest calls dimensionsprint("heads:", settings["heads"])print("feed-forward width:", settings["feed_forward"])print("longest text, in tokens:", settings["longest"])  # longer texts were cut to this many tokensprint("vocabulary:", f"{len(vocabulary):,}")             # the store tokenizer's tokens, plus the four markers
출력python transformer-parameters.py
embedding (shared with the output): 530,176
attention: 262,144
feed-forward: 540,672
normalization: 1,152
total: 1,334,144
embedding's share of the total: 40%
layers: 4
width: 128
heads: 4
feed-forward width: 352
longest text, in tokens: 160
vocabulary: 4,142

The model has 1,334,144 parameters, slightly fewer than the 1,394,732 of chapter 5's window model. The two are about the same size but spend their parameters differently. The window model put about three quarters of its parameters in its output layer, and could see four tokens back. The transformer puts most of its parameters in its blocks, 540,672 of them in feed-forward layers alone. It shares its embeddings with its output, and can see every earlier token in a text of up to 160 tokens.

The embedding table, shared with the output layer, holds 530,176 parameters, 40% of the total, because it has a row of 128 numbers for each of the 4,142 tokens. Those are the store tokenizer's vocabulary plus four markers: start, separator, end, and padding, which fills out short texts in a training batch. Chapter 8 explains how the model was trained, and measures it against the baseline.

Working rule: Count parameters by part whenever you design or load a model. At small sizes, the embedding table is a large share of the total, so tie it to the output layer, and choose the vocabulary size with that cost in mind, as chapter 2's choice of merges did.

운영 환경에서 잘못되는 것들

마지막으로 변경된 판: 2026.11.11판

트랜스포머의 각 부분은 정확하게 맞물려야 하며, 잘못되는 일의 대부분은 거의 들어맞는 부분에서 생긴다.

  • 세부 사항이 어긋나도 조용히 실패할 수 있다. 정규화의 위치, RoPE가 수를 짝짓는 방식과 그 기저값, 피드포워드 층의 게이트 여부, 출력의 가중치 공유 여부는 모두 학습된 가중치와 일치해야 한다. 어떤 불일치는 곧바로 무의미한 출력을 내지만, 여기서 측정한 RoPE 기저값처럼 오류 없이 정확도를 조금 떨어뜨리는 불일치도 있다. 참조 출력과 대조하여 확인한다.
  • 학습 길이를 넘는 길이는 검증되지 않았다. RoPE는 어떤 위치에든 각도를 부여하지만, 최대 160토큰 길이의 텍스트로 학습한 모델은 그 너머의 각도를 사용하는 법을 배운 적이 없다. 그 너머를 측정해 보기 전까지는 학습 길이를 모델의 한계로 간주한다.
  • 큰 값은 낮은 정밀도에서 넘칠 수 있다. 비트 수가 적은 형식도 일반적인 크기 범위에서는 상대 정밀도를 똑같이 유지하지만, float16은 65,504보다 큰 값을 담을 수 없고, 8비트 형식에서는 소수의 큰 이상값이 다른 모든 값의 정밀도를 망가뜨린다(Dettmers et al., 2022). 여기서 측정한 제품에 대해 스토어 모델의 스트림은 최댓값이 50.6로, float16의 한계에서 멀지만, 더 낮은 정밀도로 서비스하기 전에 모델의 스트림에서 가장 큰 값을 확인한다.
  • 참조 출력을 가중치와 함께 보관한다. 4장에서 설명하듯 결과는 기계에 따라 달라지므로, 가중치와 참조 출력과 정확한 학습 스크립트를 함께 보관하고, 코드나 라이브러리를 바꿀 때마다 검사를 다시 실행한다.
  • 모든 부분이 모든 위치에서 실행된다. 피드포워드 층은 모든 위치에서 비용이 같고, 어텐션의 비용은 6장에서 보았듯 길이의 제곱에 따라 커진다. 둘 중 어느 쪽을 최적화하기 전에, 어느 쪽이 지배적인지 대상 텍스트로 측정한다.

이 장에서 결정한 것

  • 저장소의 트랜스포머는 LLaMA 방식의 블록을 장난감 크기로 사용한다. 각 부분 앞에 RMSNorm을 두고, 어텐션에는 회전 위치 인코딩을 쓰며, 피드포워드 층은 게이트가 있는 SiLU 방식이고, 출력층은 임베딩과 가중치를 공유한다. 층은 4개, 너비는 128, 헤드는 4개, 피드포워드 너비는 352이다. SmolLM2-360M은 같은 종류의 부분으로 이루어져 있으나 크기가 더 크고, 그룹 쿼리 어텐션과 100,000의 RoPE 기준값을 쓴다.
  • lib.transformer의 NumPy 순전파가 이 책의 기준 구현이다. 이 구현은 검증용 상품 하나의 모든 위치에서 PyTorch의 RMSNorm, 어텐션, SiLU와 비교하여 소수점 아래 넷째 자리까지 일치하는지 확인했다. 이후 이 모델을 사용하는 모든 예제는 이 구현을 실행한다.
  • 모든 블록은 잔차 스트림에 더하고, 모든 부분은 스트림을 정규화한 사본을 읽는다. 그래서 스트림은 층을 거치며 커질 수 있고, 각 부분은 크기가 일정한 벡터를 본다.
  • 부분의 역할은 그 부분을 뺀 모델이나 바꾼 모델을 측정하여 판단한다. 블록을 하나라도 빼면 저장소의 모델은 크게 나빠지고, 첫 블록을 빼면 쓸모가 없어진다. 따라서 이 책은 학습된 모델의 블록들을 하나의 체계로 다룬다. RoPE 기준값이 틀리면 성능이 조금 떨어지지만 오류는 발생하지 않으므로, 불러온 가중치는 항상 기준 출력과 대조하여 확인한다.
  • 이 모델의 크기는 5장의 윈도 모델과 비슷하지만, 매개변수를 출력층이 아니라 블록에 쓴다. 출력과 공유하는 임베딩 테이블은 여전히 전체의 40%를 차지하므로, 어휘 크기는 이 책이 계속 살피는 비용으로 남는다.

이 판의 이 장

2026.11.16판예제 통과 2026년 11월 19이 장에서 8개 예제개 중 8개가 통과한다

영어판 2026.11.16판을 번역한 것이다.

이 절 인용

형식

각 형식은 이 번역이 따르는 영문 원서를 출처로 표기한다.

모든 형식은 이 판을 명시한다. 링크는 현재 판을 열며, 여기서 절의 제목은 본문이 마지막으로 바뀐 시점을 알려 준다. 접근 날짜와 검색 날짜는 이 자료를 여는 날짜다.

이 페이지의 내용

11장 중 7장: The transformer block

  1. Add each part's output to a residual stream
  2. 각 부분 앞에서 정규화하라
  3. Transform each position with a gated feed-forward layer
  4. Give attention positions with rotations
  5. Tie the output layer to the embeddings
  6. Check the model against PyTorch
  7. Leave out a block to see what it does
  8. Measure what a mismatched detail costs
  9. Count where the parameters go
  10. 운영 환경에서 잘못되는 것들
  11. 이 장에서 결정한 것

맨 위로 돌아가기전체 장