오픈 모델 들여다보기
읽는 데 약 38분분 걸린다, 10절에 7개 예제이(가) 있음.
AI로 번역함. 영어 원문을 사람이 검증함. 우리가 책을 만드는 방식
영어판 2026.11.16판을 번역한 것이다. 원문 읽기
이 장을 마치면 이 책이 지금까지 만든 도구로 오픈 언어 모델을 읽을 수 있다. 파라미터가 어디에 쓰이는지 세고, 모델을 실행하는 데 드는 비용을 계산하고, 토크나이저가 다른 모델과 공정하게 비교되도록 이 모델로 텍스트에 점수를 매기고, 6장에서 본 어텐션 패턴을 이 모델의 헤드에서 찾고, 모델이 무엇을 복사하는지 시험하고, 모델의 크기가 상점의 텍스트에서 무엇을 가져다주고 무엇을 가져다주지 못하는지 알아본다.
9장은 상점 자체 모델이 만든 설명으로 끝났는데, 그 설명은 누군가 읽기 전에 검증이 필요하다. 그 모델은 6장부터 9장까지 하나의 가구·생활용품 카탈로그에서 만들고, 학습시키고, 사용한 작은 트랜스포머이다. Hugging Face가 Apache 2.0 라이선스로 공개한 오픈 모델 SmolLM2-360M은 7장에서 보았듯이 같은 부품으로 만들어졌다. 이 모델은 파라미터가 200배 넘게 많고, 웹 텍스트, 코드, 수학, 합성 교과서로 이루어진 4조 토큰(모델 카드)으로 학습되었다(Allal et al., 2025, 6절). 이 장이 던지는 질문은, 그렇게 훨씬 더 많은 텍스트로 학습한 무료 다운로드 가능 모델이 상점 자체 모델의 일, 곧 검색창에서 다음 단어를 제안하고 설명을 예측하는 일을 해낼 수 있는가이다. 2장에서는 SmolLM2의 토크나이저를 읽었고, 이 장에서는 모델을 읽는다.
SmolLM2를 실행하려면 PyTorch와 약 0.7GB의 다운로드가 필요하며, 가중치는 16비트 숫자로 저장된다. 그래서 이 책의 다른 기록된 결과처럼 한 번만 실행했다. 스크립트 recordings/open_model.py는 상점의 모델처럼 다음 토큰을 예측하도록만 학습된 베이스 모델을, 저장소의 고정된 버전인 특정 리비전에서 불러왔다. 이 스크립트는 상점의 텍스트로 모델을 실행하고 이 장에서 읽을 내용을 저장했다. 저장된 내용은 모든 가중치의 형태, 상점의 검증용 이름과 설명에 대한 점수, 각 헤드의 어텐션 습관, 6장의 복사 시험, 8장의 검색창 접두사이다. 두 번째 스크립트 recordings/open_model_family.py는 SmolLM2의 세 가지 크기에 대한 설정 파일만 역시 고정된 리비전에서 내려받았다. 예제는 두 기록을 examples/data/recorded/에서 lib.open_model를 통해 읽는다.
이 부분은 아직 번역되지 않았으므로 영어로 표시된다.
Count the parameters at scale
The parts are familiar, so the first question is where the parameters sit. Chapter 7 counted the store's model by part: 1,334,144 parameters, 40% of them in the embedding table it shares with its output.
Run the same count on SmolLM2's recorded weights, and watch which part now holds the most:
1"""Counts SmolLM2-360M's recorded weights by part, with the embedding table shared with the output counted once.2The point that the feed-forward layers hold most of the parameters rests on the feed-forward line."""3import math45from lib import open_model67shapes = {name: shape for name, shape in open_model.read("parameters").items()8 if name != "lm_head.weight"} # the output layer is tied to the embeddings: the same weights, counted once9# A part is matched by name: a weight counts toward the part whose key appears in the weight's name.10parts = {"embedding (shared with the output)": ["embed_tokens"], "attention": ["q_proj", "k_proj", "v_proj", "o_proj"],11 "feed-forward": ["gate_proj", "up_proj", "down_proj"], "normalization": ["layernorm", "model.norm"]}12total = sum(math.prod(shape) for shape in shapes.values())13for part, keys in parts.items():14 count = sum(math.prod(shape) for name, shape in shapes.items() if any(key in name for key in keys))15 print(f"{part}:", f"{count:,}", f"({count / total:.0%})")16print("total:", f"{total:,}")17config = open_model.read("manifest")["config"] # the configuration recorded with the weights18print("layers:", config["num_hidden_layers"])19print("width:", config["hidden_size"])20print("query heads:", config["num_attention_heads"])21print("key and value heads:", config["num_key_value_heads"])22print("feed-forward width:", config["intermediate_size"])23print("vocabulary:", f"{config['vocab_size']:,}")24# Layer 0 stands for every layer: each layer's weights have the same shapes.25print("one layer's query weights:", shapes["model.layers.0.self_attn.q_proj.weight"])26print("one layer's key weights:", shapes["model.layers.0.self_attn.k_proj.weight"])
python open-model-parameters.pyembedding (shared with the output): 47,185,920 (13%)
attention: 78,643,200 (22%)
feed-forward: 235,929,600 (65%)
normalization: 62,400 (0%)
total: 361,821,120
layers: 32
width: 960
query heads: 15
key and value heads: 5
feed-forward width: 2560
vocabulary: 49,152
one layer's query weights: [960, 960]
one layer's key weights: [320, 960]
The balance has shifted from the store's model. The feed-forward layers hold 235,929,600 (65%) of SmolLM2's 361,821,120 parameters. The embedding table, still shared with the output, holds 47,185,920 (13%), attention 78,643,200 (22%), and the normalization gains almost nothing. The blocks grow with the number of layers times the square of the width, and the embedding table, for a fixed vocabulary, only with the width, so as a model grows, the blocks take over.
Attention holds fewer parameters than 15 full heads would, because of grouped-query attention, which shares each key and value head among several query heads. Each layer has 15 query heads but only 5 heads of keys and values, each shared by 3 query heads (Ainslie et al., 2023). The weights show it. One layer's query weights have the shape [960, 960], and its key weights [320, 960], a third as many numbers. Per layer, with the head size being the width divided by the number of query heads:
attention parameters = 2 × width² + 2 × width × key and value heads × head size
for the query and output weights, then the key and value weights. During generation, a model keeps every layer's keys and values for every earlier token, so sharing them shrinks that store, as a later section measures; Ainslie and colleagues found it costs little in quality.
Working rule: Count an open model's parameters by part from its weights before you use it, and read the shapes for its design choices: tied embeddings, grouped keys and values, a gated feed-forward layer. Expect the blocks, not the embeddings, to hold most of a large model's parameters.
한 모델 계열의 크기 비교
SmolLM2는 세 가지 크기로 나오며, 각 크기는 같은 부품으로 만들어진다. 이 모델들의 구성 파일을 보면 제작자가 매개변수를 어디에 더 쓰는지 알 수 있다. 각 구성 파일에는 층 수, 너비, 헤드 수, 피드포워드 너비가 적혀 있고, 이 값만 있으면 7장의 계산과 위의 공식으로 모든 매개변수를 부품별로 셀 수 있다. lib.open_model.parameters가 이 일을 한다.
기록된 구성 파일로 세 가지 크기를 모두 세고, 360M 모델의 개수는 첫 번째 기록에서 저장한 가중치와 대조한다.
1"""Counts all three SmolLM2 sizes from their recorded configurations, part by part, and checks the 360M model's count2against its recorded weights. The claim that the counts match the weights rests on the 360M parameters line and the last line."""3import math45from lib import open_model67for name in open_model.SIZES:8 config = open_model.config(name)9 parts = open_model.parameters(config) # the count by part, from the configuration alone, grouped keys and values included10 print(f"{name}, layers:", config["num_hidden_layers"])11 print(f"{name}, width:", f"{config['hidden_size']:,}")12 print(f"{name}, query heads:", config["num_attention_heads"])13 print(f"{name}, query heads per key and value head:", config["num_attention_heads"] // config["num_key_value_heads"]) # query heads sharing each key and value head14 print(f"{name}, feed-forward width:", f"{config['intermediate_size']:,}")15 print(f"{name}, RoPE base:", f"{config['rope_theta']:,}") # the rotary base: a larger base makes positions turn more slowly16 print(f"{name}, parameters:", f"{parts['total']:,}")17 print(f"{name}, embedding's share:", f"{parts['embedding'] / parts['total']:.0%}")18# The output layer is tied to the embeddings, so its weights are skipped: the shared weights are counted once.19recorded = sum(math.prod(shape) for key, shape in open_model.read("parameters").items() if key != "lm_head.weight")20print("SmolLM2-360M, parameters in the recorded weights:", f"{recorded:,}")
python open-model-family.pySmolLM2-135M, layers: 30
SmolLM2-135M, width: 576
SmolLM2-135M, query heads: 9
SmolLM2-135M, query heads per key and value head: 3
SmolLM2-135M, feed-forward width: 1,536
SmolLM2-135M, RoPE base: 100,000
SmolLM2-135M, parameters: 134,515,008
SmolLM2-135M, embedding's share: 21%
SmolLM2-360M, layers: 32
SmolLM2-360M, width: 960
SmolLM2-360M, query heads: 15
SmolLM2-360M, query heads per key and value head: 3
SmolLM2-360M, feed-forward width: 2,560
SmolLM2-360M, RoPE base: 100,000
SmolLM2-360M, parameters: 361,821,120
SmolLM2-360M, embedding's share: 13%
SmolLM2-1.7B, layers: 24
SmolLM2-1.7B, width: 2,048
SmolLM2-1.7B, query heads: 32
SmolLM2-1.7B, query heads per key and value head: 1
SmolLM2-1.7B, feed-forward width: 8,192
SmolLM2-1.7B, RoPE base: 130,000
SmolLM2-1.7B, parameters: 1,711,376,384
SmolLM2-1.7B, embedding's share: 6%
SmolLM2-360M, parameters in the recorded weights: 361,821,120
360M 모델의 구성 파일로 센 개수는 기록된 가중치와 정확히 일치한다. 센 값은 361,821,120이고 기록된 값은 361,821,120이다. 가장 큰 모델의 추가 매개변수는 대부분 깊이가 아니라 너비에 들어갔다. SmolLM2-1.7B의 층 수는 24으로, 더 작은 두 모델의 30과 32보다 적다. 반면 너비는 2,048로, 두 모델의 576과 960보다 훨씬 넓다.
임베딩 테이블이 차지하는 비중은 모델이 커질수록 21%, 13%, 6%으로 줄어든다. 세 모델이 하나의 어휘를 공유하는 반면 블록은 커지기 때문이다. 또한 가장 큰 모델은 크기 외에도 다른 점이 있다. 이 모델은 키와 값을 묶지 않아서 키·값 헤드 하나당 질의 헤드가 1개이지만, 더 작은 두 모델은 키·값 헤드 하나를 질의 헤드 3개가 공유한다. 피드포워드 층의 너비는 8,192으로 너비의 네 배인 반면, 360M 모델은 2,560으로 너비의 8/3이다. 회전 위치 인코딩도 더 느리게 돌아서, 기준값이 130,000이고 360M 모델은 100,000이다.
작업 규칙: 모델의 매개변수는 구성 파일로 세고, 그 개수를 가중치와 대조한다. 두 값이 맞지 않으면 세부 사항을 잘못 읽은 것이다. 한 계열의 크기를 비교하면 제작자가 설계를 어떻게 확장했는지 알 수 있다. 큰 모델이 작은 모델과 크기만 다르다고 가정해서는 안 된다.
이 부분은 아직 번역되지 않았으므로 영어로 표시된다.
Count what it costs to run
Parameters set what a model can learn; they also set what it costs to run. Its weights have to sit in memory, at two bytes each in 16-bit formats, the form SmolLM2's weights are stored in (model card). And during generation, as chapter 9 noted, every layer keeps a key and a value for every earlier token, so each new token doesn't redo the work for the text before it. The key and value cache, every layer's keys and values for the earlier tokens, grows with the length of the text:
cache per token, in bytes = layers × 2 × key and value heads × head size × 2
counting a key and a value per layer, and two bytes per number.
Budgeting needs, for each size, the weights, the cache per token and the cache for the longest text the models are set up for, 8,192 tokens, with and without grouped keys and values:
1"""Works out each size's weights and key and value cache, with and without grouped keys and values, in decimal gigabytes2and kilobytes. The point that grouping cuts the 360M model's cache to a third rests on its two cache lines for the longest text."""3from lib import open_model45longest = open_model.config("SmolLM2-360M")["max_position_embeddings"] # the longest text the models are set up for6print("longest text, in tokens:", f"{longest:,}")7for name in open_model.SIZES:8 config = open_model.config(name)9 head_size = config["hidden_size"] // config["num_attention_heads"] # the width shared out over the query heads, grouped or not10 weights = open_model.parameters(config)["total"] * 2 # 16-bit numbers: 2 bytes each11 cache = config["num_hidden_layers"] * 2 * config["num_key_value_heads"] * head_size * 2 # a key and a value per layer, 2 bytes each12 # The same cache with every query head keeping its own key and value.13 ungrouped = config["num_hidden_layers"] * 2 * config["num_attention_heads"] * head_size * 214 # Decimal units: a gigabyte is 1e9 bytes and a kilobyte 1e3 bytes.15 print(f"{name}, weights in 16-bit numbers, in GB:", f"{weights / 1e9:.1f}")16 print(f"{name}, key and value cache per token, in KB:", f"{cache / 1e3:.0f}")17 print(f"{name}, cache for the longest text, in GB:", f"{cache * longest / 1e9:.2f}")18 print(f"{name}, cache for the longest text without grouping, in GB:", f"{ungrouped * longest / 1e9:.2f}")
python open-model-memory.pylongest text, in tokens: 8,192
SmolLM2-135M, weights in 16-bit numbers, in GB: 0.3
SmolLM2-135M, key and value cache per token, in KB: 23
SmolLM2-135M, cache for the longest text, in GB: 0.19
SmolLM2-135M, cache for the longest text without grouping, in GB: 0.57
SmolLM2-360M, weights in 16-bit numbers, in GB: 0.7
SmolLM2-360M, key and value cache per token, in KB: 41
SmolLM2-360M, cache for the longest text, in GB: 0.34
SmolLM2-360M, cache for the longest text without grouping, in GB: 1.01
SmolLM2-1.7B, weights in 16-bit numbers, in GB: 3.4
SmolLM2-1.7B, key and value cache per token, in KB: 197
SmolLM2-1.7B, cache for the longest text, in GB: 1.61
SmolLM2-1.7B, cache for the longest text without grouping, in GB: 1.61
Grouping keys and values keeps the 360M model's cache small. It takes 41 kilobytes per token, so one text of the longest length needs 0.34 gigabytes, a third of the 1.01 it would need without grouping.
A service writing many descriptions at once keeps one cache per text, so with long texts the caches can outgrow the weights. The 1.7B model, which doesn't group keys and values, needs 197 kilobytes per token, and 1.61 gigabytes for one long text, about half its 3.4 gigabytes of weights.
Working rule: Budget a model's memory as its weights plus one cache per text in progress, computed from its configuration. Prefer models that group keys and values when you serve long texts or many at once, and cap the text length you allow to what you've budgeted for.
이 부분은 아직 번역되지 않았으므로 영어로 표시된다.
Score the store's text with a general model
A model with two hundred times the parameters, trained on more than a million times as many tokens as the store's catalog holds, might be expected to predict the store's text better than the store's own small model. Testing that takes care, because the two models use different tokenizers: SmolLM2's vocabulary of 49,152 tokens splits text differently from the store's, so their perplexities per token count different things. What they share is the text. A perplexity per word divides a model's total log-probability of a text by the number of words in it, instead of by its number of tokens. In symbols, for a text of W words whose tokens the model gives probabilities p₁ to pₙ, however many tokens that takes:
perplexity per word = exp(−(log p₁ + … + log pₙ) / W)
Dividing each model's total by the same number of words puts both on one scale, however many tokens each spent.
Each model scores the same events. For a name, the store's model starts from <s> and SmolLM2 from its own start token, <|endoftext|>, and both predict the name's words and then its end: <sep> for the store's model, a newline for SmolLM2. For a description, both are given the name and predict the description's words and its end. The names are the first 500 validation names, and the descriptions are chapter 8's 446 products. The recording also scored each name after a label, Product name: , to test whether telling SmolLM2 what it's reading helps.
Which model scores the store's names better per word, and by how much? Decide before reading on. Then compute the store's model's scores in NumPy, read SmolLM2's from the recording, and compare them per token and per word:
1"""Scores the store's model and SmolLM2-360M on the same names and the same descriptions, per word, the unit the two share.2The point that the small model wins on both kinds of text rests on the per-word lines for names and for descriptions."""3import math45from lib import bpe, open_model, transformer6from lib.products import product_texts7from lib.store import load_products89_, vocabulary, _ = transformer.load() # load() returns the weights, the vocabulary and the manifest; only the vocabulary is used10index = {token: i for i, token in enumerate(vocabulary)}11scores, tokenizer = open_model.read("scores"), bpe.store_tokenizer()12per = lambda log_p, count: f"{math.exp(-log_p / count):,.1f}" # perplexity: e to the average negative log-probability per count1314# Names: each name's tokens and its end (<sep> for the store's model, a newline for SmolLM2), from the start token.15names = list(load_products("validation")["name"])[:len(scores["names"])] # as many names as SmolLM2 was scored on, from the top of the split16# The start marker, the name's tokens and <sep>: the model scores everything after the start marker.17encoded = [[index[transformer.START]] + [index[t] for t in tokenizer.encode(name)] + [index[transformer.SEP]] for name in names]18store = sum(transformer.total_log_probability(ids, 1) for ids in encoded) # scores ids[1:], every token after <s>19smollm2 = sum(row["log_p"] for row in scores["names"])20# Words are whitespace-separated, as the recording counted them; both models are divided by the same count.21words = sum(row["words"] for row in scores["names"])22print("validation names scored:", len(names))23print("names, per-token perplexity, the store's model:", per(store, sum(len(ids) - 1 for ids in encoded))) # all but <s>24print("names, per-token perplexity, SmolLM2-360M:", per(smollm2, sum(row["tokens"] for row in scores["names"]))) # SmolLM2's own count25print("names, per-word perplexity, the store's model:", per(store, words))26print("names, per-word perplexity, SmolLM2-360M:", per(smollm2, words))27# The label is context for the recording and is not scored; the same words count.28print("names after a label, per-word perplexity, SmolLM2-360M:", per(sum(r["log_p"] for r in scores["labelled names"]), words))2930# Descriptions: chapter 8's products, each description's tokens and its end, given the name.31end = index[transformer.END] # the end marker closes a description32# The first 500 validation products. Keeps those whose text ends rather than being cut at the length limit, and that have33# at least one description token after <sep>. Scores from sep + 1: the description and its end, given everything up to <sep>.34store = sum(transformer.total_log_probability(ids, sep + 1) for ids, sep in product_texts("validation", 500)35 if ids[-1] == end and sep < len(ids) - 2)36words = sum(row["words"] for row in scores["descriptions"])37print("descriptions scored:", len(scores["descriptions"]))38print("descriptions, per-word perplexity, the store's model:", per(store, words))39print("descriptions, per-word perplexity, SmolLM2-360M:", per(sum(r["log_p"] for r in scores["descriptions"]), words))
python open-model-on-the-store.pyvalidation names scored: 500
names, per-token perplexity, the store's model: 25.8
names, per-token perplexity, SmolLM2-360M: 564.8
names, per-word perplexity, the store's model: 199.3
names, per-word perplexity, SmolLM2-360M: 54,352.8
names after a label, per-word perplexity, SmolLM2-360M: 67,101.5
descriptions scored: 446
descriptions, per-word perplexity, the store's model: 37.0
descriptions, per-word perplexity, SmolLM2-360M: 61.2
The small model wins on both kinds of text, by a wide margin on names. Per word, the store's model scores 199.3 on the names against 54,352.8 for SmolLM2, and 37.0 on the descriptions against 61.2. Per token, on these names, the store's model scores 25.8 and SmolLM2 564.8. Per token hides the gap, because SmolLM2 spends more tokens on each of the store's words.
Telling SmolLM2 it's reading a product name doesn't close the gap. After the label, it scores 67,101.5 per word, no better than without. So the gap isn't that SmolLM2 doesn't know a name is coming; it's the names themselves. Our judgement, untested here, is that two features of the store's text account for much of it: many names begin with a product line's made-up name, which the store's model has seen across its catalog, and the store's text is lowercased with spaces around its punctuation, unlike most of what SmolLM2 learned from.
The lesson isn't that small models are better. A model predicts best the kind of text it learned from: SmolLM2 knows far more English than the store's model, and little of the store's habits. Teaching it those habits takes further training on the store's text, which chapter 11 describes.
Working rule: Compare models with different tokenizers per word, or per character, on exactly the same text and events, never per token. Before choosing a large general model over a small one trained on your own text, measure both on your text; size doesn't make up for text unlike anything the model learned from.
이 부분은 아직 번역되지 않았으므로 영어로 표시된다.
Find the same heads at scale
Chapter 6 measured the store's model's heads by their habits, against an even spread, which puts 1 / (i + 1) of a position's weight on each of the i + 1 tokens position i can see. The same measures work on any model, and SmolLM2 has 480 heads to measure.
Look at each head's recorded average weight on the previous token and on the first token, over the first 50 of the 446 products scored above, each text a name, a newline and its description. The even spread is averaged the same way, so the two can be set side by side:
1"""Reads SmolLM2's recorded head averages over the recorded texts, and compares them with an even spread averaged the same way.2The point that a few heads specialize and many put their weight on the first token rests on the head counts and the first-token averages."""3import numpy as np4from tokenizers import Tokenizer56from lib import open_model, transformer7from lib.products import product_texts8from lib.store import load_products910heads = open_model.read("heads")11previous, first = np.array(heads["previous"]), np.array(heads["first"]) # (layers, heads)12print("texts averaged:", heads["texts"])13print("heads in all:", previous.size)14print("heads with more than half their weight on the previous token:", int((previous > 0.5).sum())) # a position's weights sum to 1, so above 0.5 is most of it15print("the most weight a head puts on the previous token:", f"{previous.max():.2f}")16print("average weight on the first token, layers 1 to 4:", f"{first[:4].mean():.2f}") # rows count layers from 0: rows 0 to 317print("average weight on the first token, layers 5 to 32:", f"{first[4:].mean():.2f}") # rows from 4 on are layers 5 to 3218print("heads with more than half their weight on the first token:", int((first > 0.5).sum()))1920# An even spread, averaged the same way: each position i shares its weight equally among the i + 1 tokens it sees.21smollm2 = Tokenizer.from_file("data/smollm2-tokenizer/tokenizer.json")22end = transformer.load()[1].index(transformer.END) # [1] is the vocabulary: load() returns weights, vocabulary, manifest23rows = load_products("validation").head(500) # the first 500 validation products, the ones product_texts reads24# The same products as product_texts, with their names and descriptions, filtered the same way. Each text is a name, a newline25# and a description, as the recording wrote it.26texts = [f"{name}\n{description}\n" for (ids, sep), name, description in27 zip(product_texts("validation", 500), rows["name"], rows["description"]) if ids[-1] == end and sep < len(ids) - 2]28# Only the texts the recording averaged. The +1 is the start token the recording put before every text.29lengths = [len(smollm2.encode(text, add_special_tokens=False).ids) + 1 for text in texts[:heads["texts"]]]30# A position with k tokens to see gives an even spread 1 / k on each of them. k runs from 2 to the text's length.31print("an even spread's weight on the previous token, and on the first:",32 f"{np.mean([np.mean(1 / np.arange(2, n + 1)) for n in lengths]):.2f}")
python open-model-heads.pytexts averaged: 50
heads in all: 480
heads with more than half their weight on the previous token: 3
the most weight a head puts on the previous token: 0.79
average weight on the first token, layers 1 to 4: 0.13
average weight on the first token, layers 5 to 32: 0.61
heads with more than half their weight on the first token: 286
an even spread's weight on the previous token, and on the first: 0.05
A few heads specialize sharply. 3 heads put more than half their weight on the previous token, and the strongest puts 0.79 there, against an even spread's 0.05. The store's model's most previous-token head, measured the same way, put 0.35 on the previous token. With hundreds of heads, a model can afford to give a few of them one narrow job.
The first token's weight is the larger surprise. In the first four layers, heads put 0.13 of their weight on it on average; from layer 5 on, 0.61, against an even spread's 0.05, and 286 heads put more than half there. That's the attention sink that Xiao and colleagues found in large models (Xiao et al., 2024). They explain it by softmax: a head with nothing useful to attend to still has to put its weight somewhere, and the first token is always there. Chapter 6 found the pattern only weakly in the store's model, whose description tokens put 0.11 of their attention on its two markers, against an even spread's 0.05.
Working rule: Describe a large model's heads by the same measured averages you'd use on a small one, compared with an even spread. Expect a few specialized heads and many that put their weight on the first token, and keep that token when you cut a model's context to save memory.
이 부분은 아직 번역되지 않았으므로 영어로 표시된다.
Test copying at scale
Chapter 6 showed the store's model copying cover from a product's name into its description, by changing the name and watching the prediction follow. The same test asks whether SmolLM2 does it too. Whether its training text held this product's text can't be checked, and the test doesn't need it: if SmolLM2 predicted cover from memory of the product, taking cover out of the name wouldn't move it.
Check SmolLM2's recorded probabilities of cover and set at one point in the description of double chaise patio furniture cover, after this outdoor patio furniture, with the name as sold, without cover, and ending in set:
1"""Prints SmolLM2's recorded probabilities of cover and set after the same words of a description, under three versions of the name.2The point that each word follows its own name rests on the cover lines for the name as sold and without cover, and the set line for the name ending in set."""3from lib import open_model45copying = open_model.read("copying") # keyed by the version of the name, each with the probabilities of the tokens after the same words6for label, probabilities in copying.items():7 for token, p in probabilities.items():8 print(f"SmolLM2-360M, {label}, P({token.strip()}):", f"{p:.2f}") # the recorded tokens carry the space before them; strip removes it
python open-model-copying.pySmolLM2-360M, name as sold, P(cover): 0.75
SmolLM2-360M, name as sold, P(set): 0.00
SmolLM2-360M, name without cover, P(cover): 0.02
SmolLM2-360M, name without cover, P(set): 0.02
SmolLM2-360M, name ending in set, P(cover): 0.01
SmolLM2-360M, name ending in set, P(set): 0.77
Both models copy the next word of an earlier phrase when the phrase repeats. With the name as sold, SmolLM2 gives cover 0.75. Take cover out of the name and it falls to 0.02; end the name in set and set gets 0.77.
The store's model behaved the same way, with 0.97, 0.00 and 0.70 for the name ending in set. That copying is the behaviour of induction heads, heads that find an earlier copy of the current token and attend to what followed it. Olsson and colleagues found them in every model they studied with more than one layer (Olsson et al., 2022).
So copying isn't something scale adds; the store's 4-layer model learned it too. For a store like this one, the habit is useful and risky at once: a model that copies words from the product's name stays on the product, and one that copies from the wrong product's text spreads its mistake.
Working rule: Test a behaviour by changing the input and watching the output, and run the same test on every model you compare. When two models of very different sizes share a behaviour, it's a basic one; look for what differs elsewhere.
이 부분은 아직 번역되지 않았으므로 영어로 표시된다.
Suggest the next word at scale
The search box is where the store's text matters most. Chapter 8 found the store's model alone giving stool a probability of 0.50 after outdoor bar, and the mix of the store's model and its counts, the model the search box serves, 0.28.
Here are SmolLM2's three likeliest next tokens after the same five prefixes, each given with nothing before it, the way the store's model got them:
1"""Prints SmolLM2's three likeliest next tokens after five search-box prefixes, and its probability of stool after outdoor bar.2The point that its suggestions are the wrong text rests on the top token after each prefix, and on the stool line."""3from lib import open_model45for prefix, recorded in open_model.read("prefixes").items():6 shown = ", ".join(f"{token!r} {p:.2f}" for token, p in recorded["top"][:3]) # the three likeliest tokens, as recorded7 print(f"{prefix:<12} -> {shown}")8# The recorded key keeps the space before stool. Four places: at two, this probability would print as 0.00.9print("P(stool | outdoor bar), SmolLM2-360M:", f"{open_model.read('prefixes')['outdoor bar']['P( stool)']:.4f}")
python open-model-autocomplete.pybar -> 'ry' 0.05, 'ley' 0.05, 'b' 0.04
patio bar -> '\n' 0.21, 'be' 0.06, ',' 0.06
outdoor bar -> '\n' 0.28, ',' 0.06, '\n\n' 0.04
kitchen bar -> '\n' 0.37, '\n\n' 0.03, ',' 0.02
wine bar -> '\n' 0.20, ',' 0.09, '\n\n' 0.03
P(stool | outdoor bar), SmolLM2-360M: 0.0002
SmolLM2's suggestions read like general English, not like the store's search box. After outdoor bar, patio bar, kitchen bar and wine bar, its likeliest next token is a newline: it takes each prefix as a complete line, likely because short lines like these end in a newline in much of the text it learned from. Its probability of stool after outdoor bar is 0.0002. After bar alone, it suggests pieces of longer words, such as ry and ley, as in barry or barley. None of that is wrong English; it's the wrong text. A search box needs the store's habits, and the store's own models have them. A search box also has to answer within a keystroke, and the store's model runs a small fraction of SmolLM2's arithmetic per token.
Working rule: Don't put a general model in front of shoppers without measuring its suggestions on your own prefixes against what you have. Where a model trained on your text suggests better, keep it, and measure a general model on the jobs where its broader knowledge could matter.
두 모델을 나란히 놓고 보기
모든 측정이 끝났으므로 두 모델을 하나의 표에서 비교할 수 있다. 상점의 모델은 이 책에서 직접 만든 모델로, 6장부터 8장까지에서 측정했다. SmolLM2-360M은 이 장에서 그 모델의 기록을 읽어 측정했다.
| 상점의 모델 | SmolLM2-360M | |
|---|---|---|
| 파라미터 | 1,334,144 | 361,821,120 |
| 층 수와 너비 | 4층, 너비 128 | 32층, 너비 960 |
| 층마다 쿼리 헤드 수와 키·값 헤드 수 | 4개와 4개 | 15개와 5개 |
| 어휘 | 4,142 | 49,152 |
| 텍스트(토큰 단위): 카탈로그 또는 학습에 쓴 토큰 | 2,843,222, 상점의 카탈로그 | 4조, 대부분 일반 텍스트 |
| 상점 상품명에 대한 단어당 퍼플렉서티 | 199.3 | 54,352.8 |
| 상점 상품 설명에 대한 단어당 퍼플렉서티 | 37.0 | 61.2 |
설명이 이름을 반복할 때의 P(cover) |
0.97 | 0.75 |
| 한 헤드가 이전 토큰에 두는 가장 큰 가중치 | 0.35 | 0.79 |
열을 따라 내려가며 읽으면 이 장의 흐름이 드러난다. SmolLM2는 크기를 재는 모든 척도에서 훨씬 크고, 두 모델은 복사하기와 이전 토큰에 주의를 두기라는 같은 기본 동작을 학습했다. 한편 상점 자체의 텍스트에 관한 모든 측정은 작은 모델에 유리하다. 작은 모델은 그 텍스트만 학습했기 때문이다. 상점이 내릴 선택은 작은 모델과 큰 모델 중 하나를 고르는 것이 아니라, 각 모델에 어떤 일을 맡기느냐이다.
작업 규칙: 고르려는 모델을 모두 나란히 놓고, 작업에 중요한 척도로 비교한다. 이때 모든 열에서 같은 텍스트와 같은 테스트를 사용한다. 모든 일에 모델 하나를 쓰지 말고, 일마다 모델을 고른다.
운영 환경에서 무엇이 잘못되는가
오픈 모델은 만든 쪽의 선택을 함께 가져오며, 그 선택은 스스로 드러나지 않는다.
- 모델이 학습한 텍스트는 여러분의 텍스트가 아니다. SmolLM2는 상점의 이름을 상점 자체의 작은 모델보다 훨씬 나쁘게 채점했고, 레이블도 도움이 되지 않았다. 어떤 모델이든 믿기 전에 여러분의 텍스트로 측정해야 한다.
- 모델의 tokenizers는 여러분의 것과 다르다. 서로 다른 토크나이저에서 나온 토큰당 수치는 비교할 수 없다. 같은 이벤트에 대해 단어당 또는 문자당으로 채점해야 한다.
- 모델의 세부 사항은 그것을 실행하는 코드와 일치해야 한다. 그룹화된 키와 값, 100,000의 RoPE 기준값, 묶인 임베딩은 모두 SmolLM2-360M의 설정에 들어 있다. 7장에서는 잘못된 RoPE 기준값의 대가를 측정했다. 정확도가 조금 떨어질 뿐 오류는 나지 않는다. 모델은 만든 쪽이 공개한 코드로, 고정된 버전에서 불러오거나, 모든 세부 사항을 그 코드와 대조해야 한다.
- 리비전은 바뀐다. 모델의 저장소는 같은 이름으로 갱신될 수 있다.
recordings/open_model.py가 하듯이 리비전을 고정하고, 모든 결과와 함께 기록해야 한다. - 첫 번째 토큰이 중요하다. SmolLM2의 이후 헤드 대부분은 첫 번째 토큰에 기댄다. 컨텍스트를 줄일 때 그 토큰을 남겨야 하며, 파이프라인이 그것을 조용히 버리게 두어서는 안 된다.
- 크기가 속도를 정한다. 생성되는 모든 토큰은 모든 층을 거치므로, 수백 배 큰 모델은 토큰당 수백 배의 연산이 든다. 응답 시간을 약속하기 전에, 서비스할 머신에서 서비스할 텍스트 길이로 토큰당 시간을 측정해야 한다.
- 라이선스와 약관은 모델을 따라다닌다. SmolLM2는 Apache 2.0이지만, 다른 오픈 모델은 다른 약관을 따른다. 그 위에 무언가를 만들기 전에 해당 모델과 리비전의 라이선스를 확인해야 한다.
이 장에서 결정한 것
- 이 책의 공개 모델은 SmolLM2-360M이며, 두 가지 기록에서 읽는다. 고정된 리비전에서 가져온 기록으로, 스토어의 텍스트에 대한 실행 기록과 이 모델 계열의 구성 기록이다. 모든 예제는 PyTorch나 다운로드 없이 실행되며, 결과는 기록된 내용과 대조해 확인할 수 있다.
- 토크나이저가 다른 모델은 단어 단위로 비교한다. 같은 텍스트와 같은 사건을 대상으로 하며, 각 이름은 시작 토큰부터 이름의 단어들과 끝까지, 각 설명은 이름이 주어진 상태에서 설명의 단어들과 끝까지 계산한다.
- 스토어의 텍스트에서는 스토어의 작은 모델이 SmolLM2를 앞선다. 이름에서는 단어당 199.3 대 54,352.8이고, 설명에서는 37.0 대 61.2이다. 검색창은 8장에서 고른, 스토어 모델과 그 빈도 집계를 섞은 방식을 그대로 유지한다.
- 어텐션 헤드는 같은 척도로 모델 간에 비교하며, 각 헤드의 어텐션을 고른 분포와 비교한다. SmolLM2는 6장의 두 패턴을 모두 더 뚜렷하게 보여 준다. 직전 토큰에 특화된 소수의 헤드와, 첫 토큰에 가중치를 두는 다수의 헤드이다.
- 앞선 텍스트에서 복사하는 능력은 기본적인 능력이다. 4개 층의 모델과 32개 층의 모델이 이를 공유하며, 두 모델에서 같은 방식으로 시험한다.
- 모델의 비용은 서비스하기 전에 구성만으로 계산한다. 가중치에 더해, 처리 중인 텍스트마다 키와 값 캐시가 하나씩 필요하다. SmolLM2의 여러 크기 중에서 360M 모델은 키와 값을 그룹으로 묶어 캐시를 3분의 1로 줄인다.
이 판의 이 장
2026.11.16판예제 통과 2026년 11월 19이 장에서 7개 예제개 중 7개가 통과한다
영어판 2026.11.16판을 번역한 것이다.