어텐션
읽는 데 약 44분분 걸린다, 10절에 8개 예제이(가) 있음.
AI로 번역함. 영어 원문을 사람이 검증함. 우리가 책을 만드는 방식
영어판 2026.11.16판을 번역한 것이다. 원문 읽기
이 장을 마치면 어텐션을 손으로 계산할 수 있다. 각 토큰의 쿼리가 이전 모든 토큰의 키와 얼마나 잘 맞는지 점수를 매기고, 그 점수를 가중치로 바꾸고, 그 가중치가 고른 값들을 평균한다. 언어 모델이 미래를 보지 못하게 하는 인과 마스크, 가중치가 한 토큰에 몰리지 않게 하는 스케일링, 서로 다른 것을 동시에 찾는 여러 헤드를 더한다. 그런 다음 학습된 모델이 무엇에 어텐션을 두는지 읽고, 무엇을 복사하는지 시험하며, 얼마나 먼 과거까지 보는지 측정한다.
5장의 윈도 모델은 나란히 이어 붙인 앞선 네 토큰만으로 가게의 이름과 설명에 나오는 다음 토큰을 예측했고, 각 자리마다 고유한 가중치를 두었다. 이 설계에는 두 가지 한계가 있다. 다섯 토큰 앞은 아무리 중요해도 볼 수 없고, 첫 번째 자리에 놓인 토큰과 세 번째 자리에 놓인 같은 토큰을 따로 학습한다. 이 가게는 가구와 생활용품을 온라인으로 판매하는데, 첫 번째 한계가 왜 문제인지는 설명문이 보여 준다. 설명문은 제품 이름에 나온 단어를 수십 토큰 뒤에 다시 쓰는 경우가 많다. protect your outdoor patio furniture from harsh elements with this outdoor patio furniture cover가 그 예다. 이름을 되돌아볼 수 있는 모델은 다음에 올 단어가 무엇인지 안다. 마지막 네 토큰인 this outdoor patio furniture만 보는 모델은 다른 제품에서 이 단어들 뒤에 이어지는 모든 것 가운데서 짐작해야 한다.
어텐션(attention)은 모든 위치가 이전의 모든 위치를 바라보고, 그 내용으로부터 각 위치가 얼마나 중요한지 정하게 한다. 어텐션은 번역 모델이 번역 중인 문장의 단어들을 되돌아보게 하려고 도입되었다(Bahdanau et al., 2015). 이것은 트랜스포머(Vaswani et al., 2017)의 핵심이며, 트랜스포머는 이 책이 10장에서 읽는 공개 모델들의 설계이다. 이 장은 어텐션을 단독으로 구성한 다음, 이 책의 작은 트랜스포머 안에서 읽는다. 이 책은 그것을 가게의 모델이라 부른다. 가게의 텍스트로 한 번 학습시킨 트랜스포머이며, 7장에서 부분별로 조립한다.
이 부분은 아직 번역되지 않았으므로 영어로 표시된다.
Look back with a weighted average
Attention replaces chapter 5's fixed slots with a weighted average. At each position, the model computes a new vector as an average of vectors from the positions it can see, with weights it chooses from their content: high for the positions that matter here, low for the rest. The weights come from three vectors that each token's embedding is turned into, by three learned matrices:
- a query, a vector for what this position is looking for,
- a key, a vector for what this position offers to the positions that look at it,
- a value, a vector for what this position passes on when it's looked at.
The score of position j for position i is the dot product of i's query with j's key, divided by the square root of d, the size of each query and key, meaning how many numbers it holds. Softmax turns each row of scores into weights that are positive and sum to one, and the output at position i is the weighted sum of the values. In matrix form, for all positions at once:
Attention(Q, K, V) = softmax(QKᵀ / √d) V
where Q, K and V hold every position's query, key and value as rows (Vaswani et al., 2017, section 3.2.1). The division by √d is the subject of a later section.
To watch the arithmetic on something small, take four tokens of the store's text, the start marker and sled coffee table, a table whose base is shaped like a sled's runners. Give each a random vector of 8 numbers, as an untrained embedding table would, and compute attention with random 8-by-4 query, key and value matrices:
1"""Computes one attention layer by hand for four tokens, with random matrices. The prose's point rests on the2printed weight matrix: each row says which positions one position draws from, and how much."""3import numpy as np45tokens = ["<s>", "sled</w>", "coffee</w>", "table</w>"]6rng = np.random.default_rng(2026) # a fixed seed, so the same numbers come out on every run7x = rng.normal(0, 1, (4, 8)) # one 8-number vector per token, as an embedding table gives8w_query, w_key, w_value = (rng.normal(0, 0.5, (8, 4)) for _ in range(3))910queries, keys, values = x @ w_query, x @ w_key, x @ w_value # what each token looks for, offers, and passes on11scores = queries @ keys.T / np.sqrt(4) # how well each query matches each key, scaled12weights = np.exp(scores - scores.max(1, keepdims=True)) # subtracting each row's largest score keeps exp from overflowing13weights /= weights.sum(1, keepdims=True) # softmax along each row: each row sums to 114output = weights @ values # each token: a weighted average of all the values1516np.set_printoptions(precision=2, suppress=True)17print("attention weights (rows attend to columns):")18for token, row in zip(tokens, weights):19 print(f" {token:<11}", row)20assert np.allclose(weights.sum(1), 1) # each row sums to 1, which the prose relies on21print("output shape:", output.shape)
python attention-by-hand.pyattention weights (rows attend to columns):
<s> [0.02 0.97 0.01 0. ]
sled</w> [0.21 0.52 0.16 0.11]
coffee</w> [0.04 0. 0.03 0.94]
table</w> [0.16 0. 0.03 0.81]
output shape: (4, 4)
Each row of the printed weights is one position's average: which positions it draws from, and how much. Every row sums to one, which the example asserts, and the output has one 4-number vector per position. The weights are random, because the matrices are, but they already show attention's freedom: coffee puts most of its weight on table, a token after it, and the start marker puts most of its weight on sled. Nothing in the arithmetic stops a position from looking ahead. For predicting the next token, that freedom is a bug, as the next section shows.
Three properties of this arithmetic matter for everything that follows. The weights depend on content, through queries and keys, so the same token can attend to different places in different texts. The number of positions doesn't matter: the same three matrices work for a name of 4 tokens or a description of 150, so the model learns one way of looking back, not one per slot. And a weighted average ignores the order of what it averages. Without a mask, attention gives bar the same output in bar stool as in stool bar. The causal mask of the next section lets a little order back in, because each token then sees a different number of earlier tokens, and Haviv and colleagues found that language models can infer positions from that alone (Haviv et al., 2022). Models still add explicit position information, as chapter 7 does.
Working rule: Write out attention's four lines once, as this example does, and print the weight matrix: its rows say exactly where each position looks. Keep queries, keys and values as separate matrices while you're learning. Libraries often keep them in one matrix, as PyTorch's MultiheadAttention does with its in_proj_weight (source), and split the result, which computes the same thing.
이 부분은 아직 번역되지 않았으므로 영어로 표시된다.
Hide the future with a causal mask
In the first section, coffee drew most of its average from table, a later token. A language model can't be allowed to do that, because it predicts each token from the tokens before it. If position i could attend to position i + 1, it would see the answer: training would reward copying the next token, and the model would learn nothing it could use when generating text, where the next token doesn't exist yet. The fix is the causal mask: before softmax, set every score for a later position to minus infinity. The exponential of minus infinity is zero, so those positions get a weight of zero, and each position averages only over itself and the positions before it.
The mask also makes training efficient. With it, one pass over a text predicts every position at once, each from its own past, where chapter 5's window model needed a separate window for every prediction.
A change to the last token can reach coffee only if coffee is allowed to look ahead. Apply the mask to the same four tokens' scores, replace table with a different vector, and measure how far the output at coffee moves, without the mask and with it:
1"""Applies the causal mask to the same four tokens, then changes the last token and measures how far the output at2coffee moves, with and without the mask. The prose's point rests on the last printed line: with the mask, that change is zero."""3import numpy as np45tokens = ["<s>", "sled</w>", "coffee</w>", "table</w>"]6rng = np.random.default_rng(2026)7x = rng.normal(0, 1, (4, 8))8w_query, w_key, w_value = (rng.normal(0, 0.5, (8, 4)) for _ in range(3)) # the same draws as attention-by-hand9allowed = np.tril(np.ones((4, 4), dtype=bool)) # position i may look at positions 0 to i only101112def attend(x, mask):13 scores = (x @ w_query) @ (x @ w_key).T / np.sqrt(4)14 if mask:15 scores = np.where(allowed, scores, -np.inf) # a score of minus infinity becomes a weight of 016 weights = np.exp(scores - scores.max(1, keepdims=True))17 weights /= weights.sum(1, keepdims=True)18 return weights, weights @ (x @ w_value)192021weights, _ = attend(x, mask=True)22np.set_printoptions(precision=2, suppress=True)23print("weights with the causal mask:")24for token, row in zip(tokens, weights):25 print(f" {token:<11}", row)26print("weight on later tokens:", float(weights[~allowed].sum())) # ~allowed marks the positions above the diagonal2728changed = x.copy()29changed[3] = rng.normal(0, 1, 8) # a different last token in place of `table`30for mask in [False, True]:31 before, after = attend(x, mask)[1][2], attend(changed, mask)[1][2] # attend returns (weights, output); 2 is coffee32 change = float(np.abs(after - before).max())33 print(f"largest change in coffee's output, {'with' if mask else 'without'} the mask: {change:.2f}")34 assert (change == 0) == mask # the mask's job: with it, nothing after coffee reaches coffee, so the change is zero
python causal-mask.pyweights with the causal mask:
<s> [1. 0. 0. 0.]
sled</w> [0.29 0.71 0. 0. ]
coffee</w> [0.58 0.01 0.41 0. ]
table</w> [0.16 0. 0.03 0.81]
weight on later tokens: 0.0
largest change in coffee's output, without the mask: 4.69
largest change in coffee's output, with the mask: 0.00
The weight matrix is now lower-triangular, with zeros above the diagonal, so the total weight on later tokens is 0.0. The start marker can attend only to itself, so its row is a single 1. sled splits its weight between the start marker and itself. coffee, which put almost all its weight on the later table before the mask, now spreads it over the positions it's allowed, because softmax divides only among the positions left. The last row, table, is unchanged, since it had no later positions to lose.
The last two lines show the leak the mask closes. Without the mask, changing the last token moves the output at coffee by as much as 4.69. With the mask, coffee's output can't depend on what comes later, and the example asserts that the change is zero. A model trained without the mask can learn to copy each answer from the next position, so its training loss can fall close to zero while it learns nothing it can use to generate text. Nothing crashes along the way, which is what makes a wrong mask hard to notice.
Working rule: Apply the mask to scores before softmax, never to weights after it, so each row still sums to one. Test it the way this example does: check that the weight on later positions is exactly zero, and that changing a later token leaves earlier outputs unchanged. If a model's training loss falls far faster than expected, check the mask first.
이 부분은 아직 번역되지 않았으므로 영어로 표시된다.
Scale the scores
Attention's formula divides every score by √d, and the reason is in the dot product. A dot product of two vectors adds up one product per number. If the numbers in queries and keys are independent and random, with mean 0 and variance 1, a score has variance d, so queries and keys of a larger size give larger scores (Vaswani et al., 2017, section 3.2.1, footnote 4). Softmax turns large differences between scores into extreme weights: one position gets nearly all the weight, and the rest nearly none. Vaswani and colleagues suspected that a model in that state learns slowly, because softmax's gradients are tiny when one weight is close to 1. Dividing by √d brings the scores back to variance 1, whatever the size.
The store's model makes this concrete: its heads work with queries and keys of 32 numbers. Draw random queries and keys of three sizes, 4, 32 and 256, score each query against 16 keys with and without the division, and report the largest weight in each row, averaged over 1,000 queries. Guess first: with 256 numbers and no division, how large is a row's biggest weight on average?
1"""Measures the largest softmax weight in each row, at three sizes of query and key, with and without dividing by the2square root of the size. The prose's point rests on the unscaled averages growing with the size, and the scaled ones staying level."""3import numpy as np45rng = np.random.default_rng(2026) # a fixed seed, so the printed averages are the same on every run6for size in [4, 32, 256]:7 # numbers with mean 0 and variance 1, the case the scaling argument assumes; 1,000 queries against 16 keys8 queries, keys = rng.normal(0, 1, (1000, size)), rng.normal(0, 1, (16, size))9 for divide in [False, True]:10 scores = queries @ keys.T / (np.sqrt(size) if divide else 1)11 weights = np.exp(scores - scores.max(1, keepdims=True))12 weights /= weights.sum(1, keepdims=True)13 label = "scaled" if divide else "unscaled"14 print(f"size {size}, {label}, average largest weight in a row: {weights.max(1).mean():.2f}")
python scaling-matters.pysize 4, unscaled, average largest weight in a row: 0.45
size 4, scaled, average largest weight in a row: 0.25
size 32, unscaled, average largest weight in a row: 0.78
size 32, scaled, average largest weight in a row: 0.24
size 256, unscaled, average largest weight in a row: 0.92
size 256, scaled, average largest weight in a row: 0.25
Without the division, the largest weight grows with the size: 0.45 with 4 numbers, 0.78 with 32, the size of the store's model's heads, and 0.92 with 256. At the larger sizes, each query attends mostly to one key before any training has happened. With the division, the largest weight stays at 0.25 at every size: spread over several keys, though not evenly, since 16 equal weights would be about 0.06 each.
That's the state training should start from. Every position looks at several others, so the gradients reach all of them, and training sharpens the weights where sharp weights help. A trained head can still put most of its weight on one position, as some of the store's model's heads do; the scaling only keeps heads from starting there.
Working rule: Divide attention scores by the square root of the query's size, as the transformer paper does. A few models, such as T5, leave the division out and fold the factor into how the query weights start instead (source), so check which a model does before you load its weights. If you change how queries and keys are computed or initialized, print the average largest weight of an untrained model and expect it to be well below 1.
여러 헤드가 서로 다른 것을 찾는다
지금까지의 각 어텐션 계산은 위치마다 가중치 한 벌을 만들며, 이 가중치는 한 가지 종류의 대상만 찾을 수 있다. 예를 들어 바로 앞의 토큰이나 제품의 이름이다. 그러나 모델은 대개 여러 가지를 동시에 찾아야 한다. 멀티헤드 어텐션(multi-head attention)은 헤드(head)라고 부르는 여러 어텐션 계산을 나란히 실행한다. 각 헤드는 자신의 쿼리, 키, 값 행렬을 가지며, 벡터의 수 일부를 맡아 작업한다. 수가 8개이고 헤드가 2개라면 각 헤드는 4개의 수로 작업한다. 헤드들의 출력은 옆으로 이어 붙인 뒤, 학습되는 행렬 하나를 더 거쳐 다시 섞는다. 이 행렬이 출력 투영(output projection)이다(Vaswani et al., 2017, 3.2.2절).
수를 나누면 전체 수에 대해 헤드 하나를 쓸 때와 비용이 거의 같으면서, 각 헤드가 자기만의 가중치 패턴을 갖게 된다. 이 상점의 모델은 각 층에 헤드가 4개 있고, 각 헤드는 32개의 수로 작업한다.
무작위 벡터 네 위치에 대해 인과 마스크를 적용하고 헤드 두 개를 실행해, 마지막 위치가 두 헤드에서 같은 곳을 보는지 확인해 보자.
1"""Runs two heads of causal attention over four positions, each head working in four of the eight numbers, then mixes2the heads with an output projection. The prose's point rests on the last position's weights differing between the heads,3and on the output keeping the input's shape."""4import numpy as np56rng = np.random.default_rng(2026) # a fixed seed, so the printed weights are the same on every run7positions, dims, heads = 4, 8, 28size = dims // heads # each head works in 4 of the 8 dimensions9x = rng.normal(0, 1, (positions, dims))10w_query, w_key, w_value, output_projection = (rng.normal(0, 0.5, (dims, dims)) for _ in range(4))111213def split(m): # (positions, dims) -> (heads, positions, size)14 return m.reshape(positions, heads, size).transpose(1, 0, 2) # head 1 takes the first four numbers of each vector151617queries, keys, values = split(x @ w_query), split(x @ w_key), split(x @ w_value)18scores = queries @ keys.transpose(0, 2, 1) / np.sqrt(size) # one score matrix per head19scores = np.where(np.tril(np.ones((positions, positions), dtype=bool)), scores, -np.inf) # the causal mask, for both heads20weights = np.exp(scores - scores.max(-1, keepdims=True))21weights /= weights.sum(-1, keepdims=True)22joined = (weights @ values).transpose(1, 0, 2).reshape(positions, dims) # the heads' outputs side by side23output = joined @ output_projection # mixed back together by one more matrix2425print("weights shape (heads, positions, positions):", weights.shape)26print("what the last token attends to, head 1:", np.round(weights[0, -1], 2)) # -1 is the last position, which sees all four27print("what the last token attends to, head 2:", np.round(weights[1, -1], 2))28print("output shape:", output.shape)
python attention-heads.pyweights shape (heads, positions, positions): (2, 4, 4)
what the last token attends to, head 1: [0.41 0.18 0.29 0.11]
what the last token attends to, head 2: [0.04 0.32 0.14 0.5 ]
output shape: (4, 8)
이제 가중치는 헤드마다 행렬이 하나씩 있으며, 그 모양은 (2, 4, 4)이다. 두 헤드가 같은 입력을 읽었는데도 마지막 위치의 행은 헤드마다 다르다. 헤드 1은 가장 큰 가중치를 첫 번째 위치에 두고, 헤드 2는 마지막 위치, 곧 자기 자신에 둔다. 행렬이 무작위이므로 이 차이에는 아직 의미가 없다. 그러나 이것이 학습이 활용하는 여지다. 각 헤드는 서로 다른 방식으로 되돌아보는 법을 배울 수 있고, 출력 투영은 그 결과를 결합한다. 출력은 다시 입력과 같은 모양인 (4, 8)이므로, 7장에서 하듯 어텐션 층을 쌓을 수 있다.
작업 규칙: 경험칙으로, 각 헤드의 너비는 최소 32개의 수로 유지한다. 트랜스포머 논문의 실험에서 32개씩 16개 헤드는 64개씩 8개 헤드만큼 좋은 결과를 냈지만, 16개씩 32개 헤드는 더 나빴으며, 키의 크기를 줄이면 성능이 떨어졌다(Vaswani et al., 2017, 표 3). 모델을 바꿀 때마다 가중치 텐서의 모양, 즉 헤드 수 × 위치 수 × 위치 수를 확인한다. 헤드 축이 잘못된 자리에 있으면 오류 없이 헤드들이 섞인다.
이 부분은 아직 번역되지 않았으므로 영어로 표시된다.
Read what trained heads attend to
Random matrices show the arithmetic; a trained model shows what attention is for. The store's model has 4 layers, each with 4 heads of attention. It was trained once, in recordings/transformer.py, on the store's training names and descriptions, each product as one text: the start marker, the name, a separator token <sep>, the description and the end marker. Each layer's attention reads the output of the layer before, so later layers can attend on the basis of what earlier ones found. As in chapter 5, the examples read the saved weights and recompute the model in NumPy, with lib.transformer, which also returns every head's weights.
The question is where the model looks for cover in a validation product it never trained on, double chaise patio furniture cover, whose description repeats the end of its name. Stop at the description's second furniture, where the next token is cover, and print the model's probability for cover, and each head's three largest weights in the first and last layers. current: marks the position being predicted from, name: a token in the name, and description: an earlier token in the description:
1"""Runs the store's trained model on one validation product, up to the description's second furniture, and prints where2its heads attend there. The prose's point rests on the last layer's head with the most weight on the name's cover, and on3the probability it gives cover."""4import numpy as np56from lib import transformer7from lib.store import load_products89_, vocabulary, manifest = transformer.load() # forward reads the weights itself, so only the vocabulary and manifest are kept10settings = manifest["settings"]11index = {t: i for i, t in enumerate(vocabulary)}12product = load_products("validation").query("name == 'double chaise patio furniture cover'").iloc[0]13ids = transformer.product_ids(product["name"], product["description"], index, settings["longest"])14sep = ids.index(index[transformer.SEP])15position = ids.index(ids[sep - 1], sep) - 1 # just before the description first repeats the name's last token16logits, attention = transformer.forward(ids, keep=True)17probabilities = transformer.softmax(logits[position])181920def label(i):21 """Where a token sits: the current position, the name, the description, or a marker."""22 if i == position:23 return f"current:{vocabulary[ids[i]]}"24 if i == 0 or i == sep:25 return vocabulary[ids[i]]26 return f"{'name' if i < sep else 'description'}:{vocabulary[ids[i]]}"272829print("layers:", settings["layers"])30print("heads per layer:", settings["heads"])31print("dimensions per head:", settings["dimensions"] // settings["heads"])32print("name:", " ".join(vocabulary[i] for i in ids[1:sep]))33# after <sep>, up to and including the current position34print("description so far:", " ".join(vocabulary[i] for i in ids[sep + 1:position + 1]))35# position + 1 is the token that really comes next in the text36print(f"P(next token is {vocabulary[ids[position + 1]]}): {probabilities[ids[position + 1]]:.2f}")37for layer in [0, len(attention) - 1]:38 for head, weights in enumerate(attention[layer]):39 top = np.argsort(-weights[position], kind="stable")[:3]40 print(f"layer {layer + 1}, head {head + 1}: " + ", ".join(f"{label(i)} {weights[position, i]:.2f}" for i in top))4142name_cover = sep - 1 # the name's last token, `cover`43earlier = ids.index(ids[position], sep + 1) # the description's earlier `furniture`44last = attention[-1][:, position] # the last layer's heads, at the current position45head = int(last[:, name_cover].argmax())46print("last-layer head with the most weight on the name's cover:", f"head {head + 1}")47print("its weight on the name's cover:", f"{last[head, name_cover]:.2f}")48print(f"its weight on {vocabulary[ids[earlier + 1]].removesuffix('</w>')}, after the description's earlier furniture:", # earlier + 1: the token after the first furniture49 f"{last[head, earlier + 1]:.2f}")
python trained-attention.pylayers: 4
heads per layer: 4
dimensions per head: 32
name: double</w> chaise</w> patio</w> furniture</w> cover</w>
description so far: protect</w> your</w> outdoor</w> patio</w> furniture</w> from</w> har sh</w> elements</w> with</w> this</w> outdoor</w> patio</w> furniture</w>
P(next token is cover</w>): 0.97
layer 1, head 1: current:furniture</w> 0.80, description:furniture</w> 0.11, description:elements</w> 0.03
layer 1, head 2: current:furniture</w> 0.31, name:furniture</w> 0.13, description:furniture</w> 0.11
layer 1, head 3: current:furniture</w> 0.18, description:patio</w> 0.18, <sep> 0.12
layer 1, head 4: description:protect</w> 0.17, current:furniture</w> 0.14, description:patio</w> 0.11
layer 4, head 1: name:cover</w> 0.48, name:double</w> 0.18, name:furniture</w> 0.13
layer 4, head 2: name:cover</w> 0.37, <s> 0.13, name:chaise</w> 0.11
layer 4, head 3: description:patio</w> 0.27, current:furniture</w> 0.25, description:outdoor</w> 0.21
layer 4, head 4: description:with</w> 0.38, description:from</w> 0.21, description:elements</w> 0.06
last-layer head with the most weight on the name's cover: head 1
its weight on the name's cover: 0.48
its weight on from, after the description's earlier furniture: 0.00
The model gives cover a probability of 0.97, and the heads show where it may have looked. In the first layer, one head attends mostly to the current token, furniture, and the other three spread their weight over the description and the name so far. In the last layer, head 1 puts 0.48 of its weight on cover in the name, many tokens back: the token that followed furniture in the name. The description's own earlier furniture was followed by from, and that head gives from only 0.00. A second last-layer head also puts its largest weight on the name's cover.
That pattern has a name. Olsson and colleagues found pairs of heads that work this way in small models made only of attention layers, and heads with the same behaviour in large models. They call the later head of each pair an induction head: a head in an earlier layer copies each token's predecessor into its position, and the induction head uses that to find earlier positions that followed the current token, and attends to them (Olsson et al., 2022). The weights above are consistent with that behaviour, but weights show where a head drew its average from, not what the model used, so the next section tests it.
Working rule: Inspect a trained model by starting with specific tokens in texts you understand, where you know what the model should look at. Treat what the weights suggest as a hypothesis, and test it by changing the input, as the next section does.
모델이 복사하는 것을 시험하기
모델이 cover을(를) 이름에서 복사하여 예측한다면, 이름을 바꾸면 예측도 바뀌어야 한다. 이런 종류의 설명이 흔히 그렇게 이어지기 때문에 cover을(를) 예측한다면, 이름은 그다지 중요하지 않을 것이다. 입력을 바꾸고 출력을 관찰하는 시험은 이 물음에 답할 수 있지만, 어텐션 가중치로는 답할 수 없다.
설명은 그대로 두고 이름만 바꾼다. 판매되는 그대로의 이름, cover을(를) 뺀 이름, set으로 끝나는 이름을 사용한다. 한 위치에서 cover과(와) set의 확률을 보면 예측이 이름을 따르는지 알 수 있다. 그런 다음 훈련 설명에서 this outdoor patio furniture 뒤에 무엇이 오는지 센다. 이 네 토큰은 윈도 모델이 보게 되는 범위이다.
1"""Changes the name of one validation product and reads the model's probabilities for cover and set at the same position,2then counts what follows this outdoor patio furniture in the training descriptions. The prose's point rests on P(cover)3falling when cover leaves the name, and on how rarely this outdoor patio furniture is followed by cover in the training4descriptions, which is all the four tokens alone give."""5from lib import transformer6from lib.store import load_products78_, vocabulary, manifest = transformer.load() # forward reads the weights itself, so only the vocabulary and manifest are kept9index = {t: i for i, t in enumerate(vocabulary)}10product = load_products("validation").query("name == 'double chaise patio furniture cover'").iloc[0]11cover, other = index["cover</w>"], index["set</w>"]1213for name, label in [("double chaise patio furniture cover", "name as sold"),14 ("double chaise patio furniture", "name without cover"),15 ("double chaise patio furniture set", "name ending in set")]:16 ids = transformer.product_ids(name, product["description"], index, manifest["settings"]["longest"])17 sep = ids.index(index[transformer.SEP])18 furniture = [i for i in range(sep + 1, len(ids)) if ids[i] == index["furniture</w>"]]19 p = transformer.softmax(transformer.forward(ids)[furniture[1]]) # at the description's second `furniture`20 print(f"{label}, P(cover): {p[cover]:.2f}")21 print(f"{label}, P(set): {p[other]:.2f}")2223phrase = "this outdoor patio furniture " # the trailing space makes the phrase end at a word boundary24# the word after the phrase, in each description that has one; a phrase at the very end has no next word and is skipped25after = [d.split(phrase, 1)[1].split()[0] for d in load_products("train")["description"]26 if isinstance(d, str) and phrase in d and d.split(phrase, 1)[1].split()]27print("training descriptions with 'this outdoor patio furniture':", len(after))28print("followed by cover:", after.count("cover"))
python copy-from-the-name.pyname as sold, P(cover): 0.97
name as sold, P(set): 0.02
name without cover, P(cover): 0.00
name without cover, P(set): 0.29
name ending in set, P(cover): 0.00
name ending in set, P(set): 0.70
training descriptions with 'this outdoor patio furniture': 6
followed by cover: 2
예측을 결정하는 것은 이름이다. 판매되는 그대로의 이름이면 모델은 cover에 0.97을(를) 부여하고, 이름에서 cover을(를) 빼면 0.00(으)로 떨어진다. 대신 이름을 set으로 끝내면 모델은 set에 0.70을(를), cover에 0.00을(를) 부여한다.
네 토큰만으로는 cover을(를) 약하게만 가리킨다. this outdoor patio furniture을(를) 포함하는 훈련 설명 6개 가운데 cover(으)로 이어지는 것은 2개이다.
이는 앞 절의 가중치가 시사한 행동이며, 이제 그 효과로 확인된 셈이다. 모델은 여러 토큰 앞에 있는 제품 이름의 단어를 복사하는데, 네 토큰 윈도에 제한된 모델은 이렇게 할 수 없다. 제품 하나만으로는 모델이 이런 일을 얼마나 자주 하는지 측정할 수 없다. 다음 두 절에서는 많은 제품에 걸쳐 어텐션 헤드를 측정한다.
작업 규칙: 모델이 무엇을 사용하는지에 관한 주장은 해당 입력을 바꾸고 출력을 측정하여 시험하며, 출력에 영향을 주지 않아야 하는 변경도 함께 포함한다. 어텐션 가중치는 가설을 세우는 데 읽고, 결론을 내리는 데는 이와 같은 시험을 사용한다.
일부 헤드는 이전 토큰에 주목한다
텍스트 하나는 오해를 낳을 수 있다. 많은 텍스트에 대한 평균은 헤드의 습관을 보여 준다. 측정할 가치가 있는 습관 하나는 이전 토큰에 주목하는 것이다. 이렇게 하는 헤드는 각 토큰의 직전 토큰을 해당 위치로 가져오므로, 이후 층은 토큰의 쌍을 볼 수 있다. 1장의 바이그램 빈도가 그랬고, 앞선 절들의 복사도 이에 의존한다. 앞서 furniture 다음에 무엇이 왔는지 찾으려면, 헤드는 앞선 각 위치가 자신의 직전 토큰을 알고 있어야 한다.
처음 300개의 검증용 제품에 대해 각 헤드가 이전 토큰에 두는 가중치를 평균하고, 이전 토큰에 가중치를 가장 많이 두는 헤드를 균등 분포와 비교한다. 균등 분포는 위치 i가 볼 수 있는 i + 1개의 토큰 각각에 1 / (i + 1)을 둔다.
1"""Averages each head's weight on the previous token over 300 validation products, and sets the largest average beside2an even spread. The prose's point rests on the head with the most weight on the previous token, and on the even spread beside it."""3import numpy as np45from lib import transformer6from lib.products import product_texts78texts = product_texts("validation", 300) # the first 300 validation products, which the model never trained on9# with keep=True, forward's second value is every layer's weights; the head count is in the manifest's settings10layers, heads = len(transformer.forward(texts[0][0], keep=True)[1]), transformer.load()[2]["settings"]["heads"]11on_previous = np.zeros((layers, heads))12for ids, _ in texts:13 _, attention = transformer.forward(ids, keep=True)14 for layer, weights in enumerate(attention): # weights: (heads, positions, positions)15 on_previous[layer] += weights[:, np.arange(1, len(ids)), np.arange(len(ids) - 1)].mean(1) # position i's weight on i - 1, for i after the start16on_previous /= len(texts)17even = np.mean([np.mean(1 / np.arange(2, len(ids) + 1)) for ids, _ in texts]) # position i sees i + 1 tokens, so 1 / (i + 1) each1819print("average weight on the previous token (rows: layers, columns: heads):")20for layer, row in enumerate(on_previous, start=1):21 print(f" layer {layer}: " + " ".join(f"{w:.2f}" for w in row))22layer, head = np.unravel_index(on_previous.argmax(), on_previous.shape)23print("head with the most weight on the previous token:", f"layer {layer + 1}, head {head + 1}")24print("its weight on the previous token:", f"{on_previous.max():.2f}")25print("an even spread's weight on the previous token:", f"{even:.2f}")
python previous-token-heads.pyaverage weight on the previous token (rows: layers, columns: heads):
layer 1: 0.03 0.06 0.13 0.10
layer 2: 0.32 0.13 0.27 0.24
layer 3: 0.13 0.33 0.09 0.07
layer 4: 0.11 0.07 0.35 0.27
head with the most weight on the previous token: layer 4, head 3
its weight on the previous token: 0.35
an even spread's weight on the previous token: 0.07
이전 토큰에 가중치를 가장 많이 두는 헤드는 layer 4, head 3이며, 평균 0.35를 둔다. 같은 방식으로 평균한 균등 분포라면 그 자리에 0.07를 둘 것이므로, 이 헤드는 이전 토큰을 약 다섯 배 선호한다. 후반 층의 여러 헤드는 가중치의 4분의 1 이상을 이전 토큰에 두며, 첫 번째 층의 헤드들은 훨씬 적게 둔다. 학습된 어텐션 예제에서는 그중 하나가 주로 현재 토큰에 주목했다.
순수하게 이전 토큰만 보는 헤드는 없다. 각 헤드는 이 습관에 다른 습관을 섞는다. 헤드의 평균은 그 헤드가 가장 많이 하는 일을 말해 줄 뿐 하는 일 전부를 말해 주지는 않으므로, 헤드에 하나의 역할만 있다고 설명하면 지나치게 단순화하는 것이다.
작업 규칙: 학습된 헤드는 수백 개의 텍스트에 걸쳐 측정한 평균으로 기술한다. 예를 들어 이전 토큰, 현재 토큰, 시작 표시에 두는 가중치가 그러하다. 각각을 같은 방식으로 평균한 균등 분포의 값과 비교해야, 0.3 같은 수치가 큰지 작은지 판단할 수 있다.
윈도 너머를 보기
어텐션이 약속한 것은 5장의 네 토큰 윈도, 곧 현재 토큰과 그 앞의 세 토큰으로 예측하는 윈도 너머를 보는 것이었다. 스토어의 모델이 실제로 그러한지는 측정할 수 있다. 윈도 너머에 얼마나 많은 가중치가 놓이는가? 검증 제품 처음 300개의 모든 설명 토큰에 대해, 끝 표시는 제외하고, 각 헤드가 4칸 이상 앞에 있는 토큰, 즉 네 토큰 윈도가 볼 수 없는 가장 가까운 위치까지를 포함해 두는 가중치를 합산하고, 별도로 두 표시 <s>와 <sep>에 두는 가중치도 합산한다. 이를 토큰, 헤드, 층에 걸쳐 평균하고, 각 토큰이 볼 수 있는 위치들에 고르게 분산했을 때의 값과 비교한다.
1"""Measures how much attention a description token puts on tokens four or more positions back, and on the two markers,2against an even spread, over 300 validation products. The prose's point rests on the all-layers share four or more back,3and on the count of descriptions that repeat a token of the name."""4import numpy as np56from lib import transformer7from lib.products import product_texts89_, vocabulary, manifest = transformer.load()10end = vocabulary.index(transformer.END)11texts = product_texts("validation", 300) # the first 300 validation products, which the model never trained on12layers = manifest["settings"]["layers"]13far, markers, even_far, even_markers, counted = np.zeros(layers), np.zeros(layers), 0.0, 0.0, 014for ids, sep in texts:15 # the description's tokens: after <sep>, so not the name, and without the end marker16 queries = np.array([i for i in range(sep + 1, len(ids)) if ids[i] != end])17 if len(queries) == 0:18 continue # a product with no description19 _, attention = transformer.forward(ids, keep=True)20 stacked = np.stack(attention)[:, :, queries] # (layers, heads, description tokens, positions)21 positions = np.arange(len(ids))22 is_marker = np.isin(positions, [0, sep]) # <s> and <sep>23 beyond = (queries[:, None] - positions[None, :] >= 4) & ~is_marker # 4 positions back is the nearest a four-token window can't see24 far += (stacked * beyond).sum(-1).mean(1).sum(-1)25 markers += (stacked * is_marker).sum(-1).mean(1).sum(-1)26 even_far += (beyond.sum(1) / (queries + 1)).sum() # an even spread over the positions each token can see27 even_markers += (2 / (queries + 1)).sum() # two markers among the query's i + 1 visible positions28 counted += len(queries)2930print("description tokens measured:", f"{counted:,}")31print("average attention weight of a description token, by layer:")32for layer in range(layers):33 print(f" layer {layer + 1}: 4 or more tokens back {far[layer] / counted:.2f}, on <s> and <sep> {markers[layer] / counted:.2f}")34print("all layers, weight 4 or more tokens back:", f"{far.mean() / counted:.2f}")35print("an even spread's weight 4 or more tokens back:", f"{even_far / counted:.2f}")36print("all layers, weight on <s> and <sep>:", f"{markers.mean() / counted:.2f}")37print("an even spread's weight on <s> and <sep>:", f"{even_markers / counted:.2f}")38# the name without <s>, and what follows <sep>; a product counts as described if anything but the end marker follows39described = [(ids[1:sep], ids[sep + 1:]) for ids, sep in texts if any(i != end for i in ids[sep + 1:])]40print("products with a description:", len(described))41print("descriptions that repeat a token of the name:", sum(bool(set(n) & set(d)) for n, d in described))
python attention-reaches-back.pydescription tokens measured: 20,571
average attention weight of a description token, by layer:
layer 1: 4 or more tokens back 0.42, on <s> and <sep> 0.12
layer 2: 4 or more tokens back 0.39, on <s> and <sep> 0.04
layer 3: 4 or more tokens back 0.58, on <s> and <sep> 0.13
layer 4: 4 or more tokens back 0.47, on <s> and <sep> 0.13
all layers, weight 4 or more tokens back: 0.46
an even spread's weight 4 or more tokens back: 0.85
all layers, weight on <s> and <sep>: 0.11
an even spread's weight on <s> and <sep>: 0.05
products with a description: 262
descriptions that repeat a token of the name: 244
헤드는 다음 단어를 예측하는 데 예상되는 대로 가까운 토큰을 선호하지만, 그럼에도 이 책의 이전 모델 중 어느 것도 사용할 수 없던 윈도 너머에 어텐션의 상당 부분을 보낸다. 모든 층을 통틀어 설명 토큰은 어텐션의 0.46를 4칸 이상 앞의 토큰에 두는데, 고른 분산이라면 0.85를 둘 것이다. 설명 토큰이 볼 수 있는 대부분이 멀리 있기 때문이다.
이는 이 텍스트에서 중요하다. 설명이 있는 제품 262개 가운데 244개는 이름의 토큰을 하나 이상 반복하는 설명을 가진다.
표시 토큰은 어텐션의 0.11를 받는데, 고른 분산에서는 0.05이다. 우연의 약 두 배이지만, 표시 토큰이 스스로 어떤 단어도 예측하지는 않는다. 한 가지 그럴듯한 이유는 더 큰 모델에서 발견된 패턴이다. 소프트맥스는 가중치의 각 행이 합해서 1이 되게 하므로, 어떤 위치에서 쓸모 있는 것을 찾지 못한 헤드도 가중치를 어딘가에는 두어야 한다. Xiao와 동료들은 언어 모델이 텍스트의 첫 토큰들이 무엇이든 그 토큰에 큰 가중치를 둔다는 것을 발견하고 이를 어텐션 싱크(attention sink)라고 불렀다(Xiao et al., 2024). 스토어의 모델에서는 <s>가 항상 맨 앞에 오고 <sep>는 모든 텍스트에 들어 있으므로 둘 다 싱크 역할을 할 수 있다. <sep>는 설명이 시작되는 곳도 표시하는데, 이것 자체로도 유용하다. 이 예제로는 둘을 구별할 수 없다.
작업 규칙: 컨텍스트가 얼마나 길어야 하는지 정하기 전에, 학습된 모델이 얼마나 멀리 앞을 어텐션하는지 측정하고, 그 수치가 의미를 갖도록 고른 분산과 비교한다. 표시 토큰의 가중치가 장거리 어텐션처럼 보이지 않도록 별도로 보고하며, 공간을 아끼려고 모델의 컨텍스트를 자를 때는 첫 토큰들을 남겨 둔다.
이 부분은 아직 번역되지 않았으므로 영어로 표시된다.
What goes wrong in production
Attention is the part of a language model whose cost grows fastest with text length, and the part most often misread:
- Cost grows with the square of the length. Every position scores every earlier one, so a text twice as long costs four times as much attention. Set a maximum length, as the store's model does at 160 tokens, and check how many of your texts the limit cuts.
- A wrong mask fails silently. A model that can see the future can reach a low training loss and still be useless for generating text. Test that the weight on later positions is zero, and that changing a later token leaves earlier outputs alone, in the training code and the generation code.
- Attention weights aren't explanations. A high weight says where a head drew its average from, not why the model predicted what it did. In the same model, very different attention weights can give the same prediction (Jain and Wallace, 2019, on recurrent and convolutional classifiers). Use weights to form hypotheses, and test them by changing the input, as this chapter did with the product's name.
- Marker weight looks like importance. Heavy weight on the first token or a separator may mean a head has nothing to do there. Don't remove those tokens, and don't read their weight as meaning.
- Softmax overflows without care. Subtract each row's largest score before exponentiating, as
lib.transformer.softmaxdoes. Make sure every row keeps at least one unmasked position, or the row becomes NaN: a causal mask always leaves each position itself, but a mask that hides padding can hide a whole row.
이 부분은 아직 번역되지 않았으므로 영어로 표시된다.
What this chapter decided
- The book's models use causal, scaled, multi-head attention, as Vaswani et al. defined it: scores divided by the square root of the size of each head's queries and keys, minus infinity on every later position before softmax, and heads that each work in a share of the numbers, mixed by an output projection.
- The store's model has 4 layers of 4 heads, each head 32 numbers wide, trained once in
recordings/transformer.pyon names and descriptions joined by a<sep>marker, and recomputed in NumPy bylib.transformer. - Claims about what a model uses are tested by changing its input. Attention weights suggest hypotheses; changing the product's name showed that the model copies
coverfrom it. - Trained heads are described by measured averages over many texts, compared with an even spread: the weight on the previous token, the weight on tokens 4 or more back, and the weight on markers, counted separately.
- On the store's descriptions, attention reaches well past chapter 5's window. Description tokens put 0.46 of their attention on tokens 4 or more back, and the model copies from the name, which a fixed window can't.
- Attention needs explicit position information. A weighted average ignores order, and the causal mask restores only a little of it, so chapter 7 adds rotary positions, as the store's model does inside every head.
이 판의 이 장
2026.11.16판예제 통과 2026년 11월 19이 장에서 8개 예제개 중 8개가 통과한다
영어판 2026.11.16판을 번역한 것이다.