Inside an open model
閱讀約需 52 分鐘 分鐘,10 節 中共有 7 個範例。
此部分尚未翻譯,因此以英文呈現。
By the end of this chapter you can read an open language model with what this book has built: count where its parameters go, work out what it costs to run, score text with it fairly against a model with a different tokenizer, find the attention patterns of chapter 6 in its heads, test what it copies, and learn what its size buys on the store's text, and what it doesn't.
Chapter 9 ended with descriptions from the store's own model that need checking before anyone reads them. That model is the small transformer that chapters 6 to 9 built, trained and used on one furniture and home-goods catalog. SmolLM2-360M, an open model released by Hugging Face under the Apache 2.0 licence, is built from the same parts, as chapter 7 showed. It has more than two hundred times as many parameters, and was trained on 4 trillion tokens (model card) of web text, code, mathematics and synthetic textbooks (Allal et al., 2025, section 6). The question this chapter asks is whether a free, downloadable model trained on that much more text could do the jobs of the store's own model: suggesting the next word in the search box, and predicting descriptions. Chapter 2 read SmolLM2's tokenizer; this chapter reads the model.
Running SmolLM2 needs PyTorch and a download of about 0.7 gigabytes, its weights stored as 16-bit numbers, so it ran once, like the book's other recorded results. The script recordings/open_model.py loaded the base model, the one trained only to predict the next token, as the store's model was, at a pinned revision, a fixed version of its repository. It ran the model on the store's text and saved what this chapter reads: every weight's shape, its scores on the store's validation names and descriptions, its heads' attention habits, chapter 6's copying test and chapter 8's search-box prefixes. A second script, recordings/open_model_family.py, downloaded only the configuration files of SmolLM2's three sizes, also at pinned revisions. The examples read both recordings, in examples/data/recorded/, through lib.open_model.
此部分尚未翻譯,因此以英文呈現。
Count the parameters at scale
The parts are familiar, so the first question is where the parameters sit. Chapter 7 counted the store's model by part: 1,334,144 parameters, 40% of them in the embedding table it shares with its output.
Run the same count on SmolLM2's recorded weights, and watch which part now holds the most:
1"""Counts SmolLM2-360M's recorded weights by part, with the embedding table shared with the output counted once.2The point that the feed-forward layers hold most of the parameters rests on the feed-forward line."""3import math45from lib import open_model67shapes = {name: shape for name, shape in open_model.read("parameters").items()8 if name != "lm_head.weight"} # the output layer is tied to the embeddings: the same weights, counted once9# A part is matched by name: a weight counts toward the part whose key appears in the weight's name.10parts = {"embedding (shared with the output)": ["embed_tokens"], "attention": ["q_proj", "k_proj", "v_proj", "o_proj"],11 "feed-forward": ["gate_proj", "up_proj", "down_proj"], "normalization": ["layernorm", "model.norm"]}12total = sum(math.prod(shape) for shape in shapes.values())13for part, keys in parts.items():14 count = sum(math.prod(shape) for name, shape in shapes.items() if any(key in name for key in keys))15 print(f"{part}:", f"{count:,}", f"({count / total:.0%})")16print("total:", f"{total:,}")17config = open_model.read("manifest")["config"] # the configuration recorded with the weights18print("layers:", config["num_hidden_layers"])19print("width:", config["hidden_size"])20print("query heads:", config["num_attention_heads"])21print("key and value heads:", config["num_key_value_heads"])22print("feed-forward width:", config["intermediate_size"])23print("vocabulary:", f"{config['vocab_size']:,}")24# Layer 0 stands for every layer: each layer's weights have the same shapes.25print("one layer's query weights:", shapes["model.layers.0.self_attn.q_proj.weight"])26print("one layer's key weights:", shapes["model.layers.0.self_attn.k_proj.weight"])
python open-model-parameters.pyembedding (shared with the output): 47,185,920 (13%)
attention: 78,643,200 (22%)
feed-forward: 235,929,600 (65%)
normalization: 62,400 (0%)
total: 361,821,120
layers: 32
width: 960
query heads: 15
key and value heads: 5
feed-forward width: 2560
vocabulary: 49,152
one layer's query weights: [960, 960]
one layer's key weights: [320, 960]
The balance has shifted from the store's model. The feed-forward layers hold 235,929,600 (65%) of SmolLM2's 361,821,120 parameters. The embedding table, still shared with the output, holds 47,185,920 (13%), attention 78,643,200 (22%), and the normalization gains almost nothing. The blocks grow with the number of layers times the square of the width, and the embedding table, for a fixed vocabulary, only with the width, so as a model grows, the blocks take over.
Attention holds fewer parameters than 15 full heads would, because of grouped-query attention, which shares each key and value head among several query heads. Each layer has 15 query heads but only 5 heads of keys and values, each shared by 3 query heads (Ainslie et al., 2023). The weights show it. One layer's query weights have the shape [960, 960], and its key weights [320, 960], a third as many numbers. Per layer, with the head size being the width divided by the number of query heads:
attention parameters = 2 × width² + 2 × width × key and value heads × head size
for the query and output weights, then the key and value weights. During generation, a model keeps every layer's keys and values for every earlier token, so sharing them shrinks that store, as a later section measures; Ainslie and colleagues found it costs little in quality.
Working rule: Count an open model's parameters by part from its weights before you use it, and read the shapes for its design choices: tied embeddings, grouped keys and values, a gated feed-forward layer. Expect the blocks, not the embeddings, to hold most of a large model's parameters.
比較同一系列的各種規模
SmolLM2 有三種規模,皆由相同的部件組成,而它們的設定顯示模型的開發者如何投入更多參數。每個設定檔都給出層數、寬度、注意力頭數與前饋層寬度,這些資料足以依據第 7 章的算法與上述公式,逐一部件計算每個參數。lib.open_model.parameters 負責這項計算。
依三種規模各自記錄的設定計算參數量,並將 360M 的計算結果與第一次記錄所儲存的權重互相核對:
1"""Counts all three SmolLM2 sizes from their recorded configurations, part by part, and checks the 360M model's count2against its recorded weights. The claim that the counts match the weights rests on the 360M parameters line and the last line."""3import math45from lib import open_model67for name in open_model.SIZES:8 config = open_model.config(name)9 parts = open_model.parameters(config) # the count by part, from the configuration alone, grouped keys and values included10 print(f"{name}, layers:", config["num_hidden_layers"])11 print(f"{name}, width:", f"{config['hidden_size']:,}")12 print(f"{name}, query heads:", config["num_attention_heads"])13 print(f"{name}, query heads per key and value head:", config["num_attention_heads"] // config["num_key_value_heads"]) # query heads sharing each key and value head14 print(f"{name}, feed-forward width:", f"{config['intermediate_size']:,}")15 print(f"{name}, RoPE base:", f"{config['rope_theta']:,}") # the rotary base: a larger base makes positions turn more slowly16 print(f"{name}, parameters:", f"{parts['total']:,}")17 print(f"{name}, embedding's share:", f"{parts['embedding'] / parts['total']:.0%}")18# The output layer is tied to the embeddings, so its weights are skipped: the shared weights are counted once.19recorded = sum(math.prod(shape) for key, shape in open_model.read("parameters").items() if key != "lm_head.weight")20print("SmolLM2-360M, parameters in the recorded weights:", f"{recorded:,}")
python open-model-family.pySmolLM2-135M, layers: 30
SmolLM2-135M, width: 576
SmolLM2-135M, query heads: 9
SmolLM2-135M, query heads per key and value head: 3
SmolLM2-135M, feed-forward width: 1,536
SmolLM2-135M, RoPE base: 100,000
SmolLM2-135M, parameters: 134,515,008
SmolLM2-135M, embedding's share: 21%
SmolLM2-360M, layers: 32
SmolLM2-360M, width: 960
SmolLM2-360M, query heads: 15
SmolLM2-360M, query heads per key and value head: 3
SmolLM2-360M, feed-forward width: 2,560
SmolLM2-360M, RoPE base: 100,000
SmolLM2-360M, parameters: 361,821,120
SmolLM2-360M, embedding's share: 13%
SmolLM2-1.7B, layers: 24
SmolLM2-1.7B, width: 2,048
SmolLM2-1.7B, query heads: 32
SmolLM2-1.7B, query heads per key and value head: 1
SmolLM2-1.7B, feed-forward width: 8,192
SmolLM2-1.7B, RoPE base: 130,000
SmolLM2-1.7B, parameters: 1,711,376,384
SmolLM2-1.7B, embedding's share: 6%
SmolLM2-360M, parameters in the recorded weights: 361,821,120
由 360M 模型的設定算出的數量與其記錄的權重完全吻合,計算值為 361,821,120,記錄值為 361,821,120。最大模型多出的參數,大部分來自寬度而非深度:SmolLM2-1.7B 有 24 層,比兩個較小規模的 30 層與 32 層都少,但寬度為 2,048,遠大於兩者的 576 與 960。
嵌入表所占的比例隨模型變大而下降,從 21% 到 13% 再到 6%,原因是三者共用同一個詞彙表,而它們的區塊卻不斷增大。此外,最大的模型不只在規模上有所不同。它沒有將鍵與值分組,每個鍵與值頭對應 1 個查詢頭,而兩個較小的模型則是由 3 個查詢頭共用一組。它的前饋層寬度為 8,192,是其寬度的四倍,而 360M 模型的前饋層寬度為 2,560,是其寬度的 8/3。它的旋轉位置編碼轉動得也較慢,基底為 130,000,相對於 100,000。
**實用原則:**依據設定計算模型的參數量,並將結果與其權重核對;若不吻合,代表你誤解了某個細節。比較同一系列的各種規模,以了解開發者如何擴展設計,不要假設較大的模型與較小的模型只有大小上的差異。
計算執行的成本
參數決定模型能學到什麼,也決定執行它要付出多少成本。模型的權重必須放在記憶體中;在 16 位元格式下,每個權重佔兩個位元組,SmolLM2 的權重即以這種形式儲存(模型卡)。此外,在生成過程中,如同第 9 章所述,每一層都會為每個先前的詞元保存一個鍵(key)與一個值(value),讓每個新詞元不必重新計算前面文字已做過的工作。鍵值快取(key and value cache)即每一層對先前詞元所保存的鍵與值,它的大小會隨文字長度增加:
每個詞元的快取大小(位元組)= 層數 × 2 × 鍵值頭數 × 頭的大小 × 2
其中每層計入一個鍵與一個值,每個數字佔兩個位元組。
要編列預算,需要針對每種規模求出權重大小、每個詞元的快取大小,以及模型所設定的最長文字(8,192 個詞元)所需的快取大小,並分別列出有分組與無分組鍵值的情形:
1"""Works out each size's weights and key and value cache, with and without grouped keys and values, in decimal gigabytes2and kilobytes. The point that grouping cuts the 360M model's cache to a third rests on its two cache lines for the longest text."""3from lib import open_model45longest = open_model.config("SmolLM2-360M")["max_position_embeddings"] # the longest text the models are set up for6print("longest text, in tokens:", f"{longest:,}")7for name in open_model.SIZES:8 config = open_model.config(name)9 head_size = config["hidden_size"] // config["num_attention_heads"] # the width shared out over the query heads, grouped or not10 weights = open_model.parameters(config)["total"] * 2 # 16-bit numbers: 2 bytes each11 cache = config["num_hidden_layers"] * 2 * config["num_key_value_heads"] * head_size * 2 # a key and a value per layer, 2 bytes each12 # The same cache with every query head keeping its own key and value.13 ungrouped = config["num_hidden_layers"] * 2 * config["num_attention_heads"] * head_size * 214 # Decimal units: a gigabyte is 1e9 bytes and a kilobyte 1e3 bytes.15 print(f"{name}, weights in 16-bit numbers, in GB:", f"{weights / 1e9:.1f}")16 print(f"{name}, key and value cache per token, in KB:", f"{cache / 1e3:.0f}")17 print(f"{name}, cache for the longest text, in GB:", f"{cache * longest / 1e9:.2f}")18 print(f"{name}, cache for the longest text without grouping, in GB:", f"{ungrouped * longest / 1e9:.2f}")
python open-model-memory.pylongest text, in tokens: 8,192
SmolLM2-135M, weights in 16-bit numbers, in GB: 0.3
SmolLM2-135M, key and value cache per token, in KB: 23
SmolLM2-135M, cache for the longest text, in GB: 0.19
SmolLM2-135M, cache for the longest text without grouping, in GB: 0.57
SmolLM2-360M, weights in 16-bit numbers, in GB: 0.7
SmolLM2-360M, key and value cache per token, in KB: 41
SmolLM2-360M, cache for the longest text, in GB: 0.34
SmolLM2-360M, cache for the longest text without grouping, in GB: 1.01
SmolLM2-1.7B, weights in 16-bit numbers, in GB: 3.4
SmolLM2-1.7B, key and value cache per token, in KB: 197
SmolLM2-1.7B, cache for the longest text, in GB: 1.61
SmolLM2-1.7B, cache for the longest text without grouping, in GB: 1.61
將鍵與值分組,能讓 360M 模型的快取維持很小。它每個詞元佔 41 KB,因此一段最長長度的文字需要 0.34 GB,只有不分組時所需的 1.01 GB 的三分之一。
同時產生大量描述的服務,會為每段文字各保留一份快取,因此在文字很長時,快取總量可能超過權重。1.7B 模型並未將鍵與值分組,每個詞元需要 197 KB,一段長文字需要 1.61 GB,約為其權重 3.4 GB 的一半。
**實用原則:**依模型的設定,將記憶體預算編列為權重加上每段進行中文字各一份快取。服務長文字或同時處理大量文字時,優先選用將鍵與值分組的模型,並把允許的文字長度限制在預算所能負擔的範圍內。
以通用模型為商店文字評分
一個參數量多達兩百倍、訓練所用詞元數超過商店目錄所含詞元數一百萬倍以上的模型,或許會被預期比商店自己的小模型更能預測商店的文字。要檢驗這一點必須謹慎,因為兩個模型使用不同的分詞器:SmolLM2 的詞彙表有 49,152 個詞元,切分文字的方式與商店的不同,所以兩者的每詞元困惑度計算的是不同的東西。兩者共有的是文字本身。每字困惑度(perplexity per word)是以文字所含的字數,而非詞元數,去除模型對該文字的總對數機率。以符號表示,對於一段含 W 個字的文字,模型給予其詞元的機率為 p₁ 到 pₙ(不論需要多少個詞元):
每字困惑度 = exp(−(log p₁ + … + log pₙ) / W)
將各模型的總和除以相同的字數,不論各自用了多少詞元,兩者都落在同一個尺度上。
兩個模型對相同的事件評分。對於名稱,商店的模型從 <s> 開始,SmolLM2 則從它自己的起始詞元 <|endoftext|> 開始,兩者都預測名稱的各個字,然後預測結尾:商店的模型是 <sep>,SmolLM2 是換行字元。對於描述,兩者都先得到名稱,再預測描述的各個字及其結尾。名稱取自前 500 個驗證名稱,描述則是第 8 章的 446 項商品。錄製的結果中還對每個名稱在標籤 Product name: 之後另外評分,以檢驗告訴 SmolLM2 它正在讀什麼是否有幫助。
哪一個模型對商店名稱的每字評分較好,差距有多大?請先自行判斷,再繼續閱讀。接著以 NumPy 計算商店模型的分數,從錄製結果讀取 SmolLM2 的分數,並分別以每詞元與每字來比較:
1"""Scores the store's model and SmolLM2-360M on the same names and the same descriptions, per word, the unit the two share.2The point that the small model wins on both kinds of text rests on the per-word lines for names and for descriptions."""3import math45from lib import bpe, open_model, transformer6from lib.products import product_texts7from lib.store import load_products89_, vocabulary, _ = transformer.load() # load() returns the weights, the vocabulary and the manifest; only the vocabulary is used10index = {token: i for i, token in enumerate(vocabulary)}11scores, tokenizer = open_model.read("scores"), bpe.store_tokenizer()12per = lambda log_p, count: f"{math.exp(-log_p / count):,.1f}" # perplexity: e to the average negative log-probability per count1314# Names: each name's tokens and its end (<sep> for the store's model, a newline for SmolLM2), from the start token.15names = list(load_products("validation")["name"])[:len(scores["names"])] # as many names as SmolLM2 was scored on, from the top of the split16# The start marker, the name's tokens and <sep>: the model scores everything after the start marker.17encoded = [[index[transformer.START]] + [index[t] for t in tokenizer.encode(name)] + [index[transformer.SEP]] for name in names]18store = sum(transformer.total_log_probability(ids, 1) for ids in encoded) # scores ids[1:], every token after <s>19smollm2 = sum(row["log_p"] for row in scores["names"])20# Words are whitespace-separated, as the recording counted them; both models are divided by the same count.21words = sum(row["words"] for row in scores["names"])22print("validation names scored:", len(names))23print("names, per-token perplexity, the store's model:", per(store, sum(len(ids) - 1 for ids in encoded))) # all but <s>24print("names, per-token perplexity, SmolLM2-360M:", per(smollm2, sum(row["tokens"] for row in scores["names"]))) # SmolLM2's own count25print("names, per-word perplexity, the store's model:", per(store, words))26print("names, per-word perplexity, SmolLM2-360M:", per(smollm2, words))27# The label is context for the recording and is not scored; the same words count.28print("names after a label, per-word perplexity, SmolLM2-360M:", per(sum(r["log_p"] for r in scores["labelled names"]), words))2930# Descriptions: chapter 8's products, each description's tokens and its end, given the name.31end = index[transformer.END] # the end marker closes a description32# The first 500 validation products. Keeps those whose text ends rather than being cut at the length limit, and that have33# at least one description token after <sep>. Scores from sep + 1: the description and its end, given everything up to <sep>.34store = sum(transformer.total_log_probability(ids, sep + 1) for ids, sep in product_texts("validation", 500)35 if ids[-1] == end and sep < len(ids) - 2)36words = sum(row["words"] for row in scores["descriptions"])37print("descriptions scored:", len(scores["descriptions"]))38print("descriptions, per-word perplexity, the store's model:", per(store, words))39print("descriptions, per-word perplexity, SmolLM2-360M:", per(sum(r["log_p"] for r in scores["descriptions"]), words))
python open-model-on-the-store.pyvalidation names scored: 500
names, per-token perplexity, the store's model: 25.8
names, per-token perplexity, SmolLM2-360M: 564.8
names, per-word perplexity, the store's model: 199.3
names, per-word perplexity, SmolLM2-360M: 54,352.8
names after a label, per-word perplexity, SmolLM2-360M: 67,101.5
descriptions scored: 446
descriptions, per-word perplexity, the store's model: 37.0
descriptions, per-word perplexity, SmolLM2-360M: 61.2
小模型在兩種文字上都勝出,在名稱上的差距尤其大。以每字計,商店的模型在名稱上得分 199.3,SmolLM2 為 54,352.8;在描述上,商店的模型為 37.0,SmolLM2 為 61.2。以每詞元計,在這些名稱上,商店的模型得分 25.8,SmolLM2 為 564.8。每詞元的算法掩蓋了這個差距,因為 SmolLM2 對商店的每個字要花上更多詞元(平均而言)。
告訴 SmolLM2 它正在讀的是商品名稱,並未縮小差距。加上標籤後,它的每字得分是 67,101.5,並不比沒有標籤時更好。因此差距並非來自 SmolLM2 不知道接下來是名稱,而是來自名稱本身。我們的判斷(此處未經檢驗)是,商店文字的兩項特徵解釋了其中大部分:許多名稱以某個產品系列虛構的名稱開頭,商店的模型在整個目錄中見過這些名稱;而且商店的文字全為小寫,標點符號前後有空格,與 SmolLM2 所學的大部分文字不同。
這裡的教訓並不是小模型比較好。模型最擅長預測的,是它學習時所見的那一類文字:SmolLM2 懂得的英文遠多於商店的模型,卻很少掌握商店的慣例。要讓它學會這些慣例,需要以商店的文字做進一步訓練,這是第 11 章所說明的內容。
實用原則: 比較分詞器不同的模型時,應在完全相同的文字與事件上,以每字或每字元來比較,絕不以每詞元來比較。在選擇大型通用模型而捨棄以自有文字訓練的小模型之前,先在你的文字上衡量兩者;規模無法彌補文字與模型所學內容截然不同的落差。
此部分尚未翻譯,因此以英文呈現。
Find the same heads at scale
Chapter 6 measured the store's model's heads by their habits, against an even spread, which puts 1 / (i + 1) of a position's weight on each of the i + 1 tokens position i can see. The same measures work on any model, and SmolLM2 has 480 heads to measure.
Look at each head's recorded average weight on the previous token and on the first token, over the first 50 of the 446 products scored above, each text a name, a newline and its description. The even spread is averaged the same way, so the two can be set side by side:
1"""Reads SmolLM2's recorded head averages over the recorded texts, and compares them with an even spread averaged the same way.2The point that a few heads specialize and many put their weight on the first token rests on the head counts and the first-token averages."""3import numpy as np4from tokenizers import Tokenizer56from lib import open_model, transformer7from lib.products import product_texts8from lib.store import load_products910heads = open_model.read("heads")11previous, first = np.array(heads["previous"]), np.array(heads["first"]) # (layers, heads)12print("texts averaged:", heads["texts"])13print("heads in all:", previous.size)14print("heads with more than half their weight on the previous token:", int((previous > 0.5).sum())) # a position's weights sum to 1, so above 0.5 is most of it15print("the most weight a head puts on the previous token:", f"{previous.max():.2f}")16print("average weight on the first token, layers 1 to 4:", f"{first[:4].mean():.2f}") # rows count layers from 0: rows 0 to 317print("average weight on the first token, layers 5 to 32:", f"{first[4:].mean():.2f}") # rows from 4 on are layers 5 to 3218print("heads with more than half their weight on the first token:", int((first > 0.5).sum()))1920# An even spread, averaged the same way: each position i shares its weight equally among the i + 1 tokens it sees.21smollm2 = Tokenizer.from_file("data/smollm2-tokenizer/tokenizer.json")22end = transformer.load()[1].index(transformer.END) # [1] is the vocabulary: load() returns weights, vocabulary, manifest23rows = load_products("validation").head(500) # the first 500 validation products, the ones product_texts reads24# The same products as product_texts, with their names and descriptions, filtered the same way. Each text is a name, a newline25# and a description, as the recording wrote it.26texts = [f"{name}\n{description}\n" for (ids, sep), name, description in27 zip(product_texts("validation", 500), rows["name"], rows["description"]) if ids[-1] == end and sep < len(ids) - 2]28# Only the texts the recording averaged. The +1 is the start token the recording put before every text.29lengths = [len(smollm2.encode(text, add_special_tokens=False).ids) + 1 for text in texts[:heads["texts"]]]30# A position with k tokens to see gives an even spread 1 / k on each of them. k runs from 2 to the text's length.31print("an even spread's weight on the previous token, and on the first:",32 f"{np.mean([np.mean(1 / np.arange(2, n + 1)) for n in lengths]):.2f}")
python open-model-heads.pytexts averaged: 50
heads in all: 480
heads with more than half their weight on the previous token: 3
the most weight a head puts on the previous token: 0.79
average weight on the first token, layers 1 to 4: 0.13
average weight on the first token, layers 5 to 32: 0.61
heads with more than half their weight on the first token: 286
an even spread's weight on the previous token, and on the first: 0.05
A few heads specialize sharply. 3 heads put more than half their weight on the previous token, and the strongest puts 0.79 there, against an even spread's 0.05. The store's model's most previous-token head, measured the same way, put 0.35 on the previous token. With hundreds of heads, a model can afford to give a few of them one narrow job.
The first token's weight is the larger surprise. In the first four layers, heads put 0.13 of their weight on it on average; from layer 5 on, 0.61, against an even spread's 0.05, and 286 heads put more than half there. That's the attention sink that Xiao and colleagues found in large models (Xiao et al., 2024). They explain it by softmax: a head with nothing useful to attend to still has to put its weight somewhere, and the first token is always there. Chapter 6 found the pattern only weakly in the store's model, whose description tokens put 0.11 of their attention on its two markers, against an even spread's 0.05.
Working rule: Describe a large model's heads by the same measured averages you'd use on a small one, compared with an even spread. Expect a few specialized heads and many that put their weight on the first token, and keep that token when you cut a model's context to save memory.
此部分尚未翻譯,因此以英文呈現。
Test copying at scale
Chapter 6 showed the store's model copying cover from a product's name into its description, by changing the name and watching the prediction follow. The same test asks whether SmolLM2 does it too. Whether its training text held this product's text can't be checked, and the test doesn't need it: if SmolLM2 predicted cover from memory of the product, taking cover out of the name wouldn't move it.
Check SmolLM2's recorded probabilities of cover and set at one point in the description of double chaise patio furniture cover, after this outdoor patio furniture, with the name as sold, without cover, and ending in set:
1"""Prints SmolLM2's recorded probabilities of cover and set after the same words of a description, under three versions of the name.2The point that each word follows its own name rests on the cover lines for the name as sold and without cover, and the set line for the name ending in set."""3from lib import open_model45copying = open_model.read("copying") # keyed by the version of the name, each with the probabilities of the tokens after the same words6for label, probabilities in copying.items():7 for token, p in probabilities.items():8 print(f"SmolLM2-360M, {label}, P({token.strip()}):", f"{p:.2f}") # the recorded tokens carry the space before them; strip removes it
python open-model-copying.pySmolLM2-360M, name as sold, P(cover): 0.75
SmolLM2-360M, name as sold, P(set): 0.00
SmolLM2-360M, name without cover, P(cover): 0.02
SmolLM2-360M, name without cover, P(set): 0.02
SmolLM2-360M, name ending in set, P(cover): 0.01
SmolLM2-360M, name ending in set, P(set): 0.77
Both models copy the next word of an earlier phrase when the phrase repeats. With the name as sold, SmolLM2 gives cover 0.75. Take cover out of the name and it falls to 0.02; end the name in set and set gets 0.77.
The store's model behaved the same way, with 0.97, 0.00 and 0.70 for the name ending in set. That copying is the behaviour of induction heads, heads that find an earlier copy of the current token and attend to what followed it. Olsson and colleagues found them in every model they studied with more than one layer (Olsson et al., 2022).
So copying isn't something scale adds; the store's 4-layer model learned it too. For a store like this one, the habit is useful and risky at once: a model that copies words from the product's name stays on the product, and one that copies from the wrong product's text spreads its mistake.
Working rule: Test a behaviour by changing the input and watching the output, and run the same test on every model you compare. When two models of very different sizes share a behaviour, it's a basic one; look for what differs elsewhere.
此部分尚未翻譯,因此以英文呈現。
Suggest the next word at scale
The search box is where the store's text matters most. Chapter 8 found the store's model alone giving stool a probability of 0.50 after outdoor bar, and the mix of the store's model and its counts, the model the search box serves, 0.28.
Here are SmolLM2's three likeliest next tokens after the same five prefixes, each given with nothing before it, the way the store's model got them:
1"""Prints SmolLM2's three likeliest next tokens after five search-box prefixes, and its probability of stool after outdoor bar.2The point that its suggestions are the wrong text rests on the top token after each prefix, and on the stool line."""3from lib import open_model45for prefix, recorded in open_model.read("prefixes").items():6 shown = ", ".join(f"{token!r} {p:.2f}" for token, p in recorded["top"][:3]) # the three likeliest tokens, as recorded7 print(f"{prefix:<12} -> {shown}")8# The recorded key keeps the space before stool. Four places: at two, this probability would print as 0.00.9print("P(stool | outdoor bar), SmolLM2-360M:", f"{open_model.read('prefixes')['outdoor bar']['P( stool)']:.4f}")
python open-model-autocomplete.pybar -> 'ry' 0.05, 'ley' 0.05, 'b' 0.04
patio bar -> '\n' 0.21, 'be' 0.06, ',' 0.06
outdoor bar -> '\n' 0.28, ',' 0.06, '\n\n' 0.04
kitchen bar -> '\n' 0.37, '\n\n' 0.03, ',' 0.02
wine bar -> '\n' 0.20, ',' 0.09, '\n\n' 0.03
P(stool | outdoor bar), SmolLM2-360M: 0.0002
SmolLM2's suggestions read like general English, not like the store's search box. After outdoor bar, patio bar, kitchen bar and wine bar, its likeliest next token is a newline: it takes each prefix as a complete line, likely because short lines like these end in a newline in much of the text it learned from. Its probability of stool after outdoor bar is 0.0002. After bar alone, it suggests pieces of longer words, such as ry and ley, as in barry or barley. None of that is wrong English; it's the wrong text. A search box needs the store's habits, and the store's own models have them. A search box also has to answer within a keystroke, and the store's model runs a small fraction of SmolLM2's arithmetic per token.
Working rule: Don't put a general model in front of shoppers without measuring its suggestions on your own prefixes against what you have. Where a model trained on your text suggests better, keep it, and measure a general model on the jobs where its broader knowledge could matter.
將兩個模型並排比較
所有量測都完成後,兩個模型就能放進同一張表格比較。商店的模型是本書自己的模型,已在第 6 章到第 8 章量測過;SmolLM2-360M 則是在本章中從它的記錄讀出數據。
| 商店的模型 | SmolLM2-360M | |
|---|---|---|
| 參數量 | 1,334,144 | 361,821,120 |
| 層數與寬度 | 4 層,寬度 128 | 32 層,寬度 960 |
| 每層的查詢頭,以及鍵與值頭 | 4 與 4 | 15 與 5 |
| 詞彙表 | 4,142 | 49,152 |
| 文字量(以 token 計):商店目錄,或訓練所用的 token | 2,843,222,商店的目錄 | 4 兆,以一般文字為主 |
| 對商店商品名稱的每詞困惑度 | 199.3 | 54,352.8 |
| 對商店商品描述的每詞困惑度 | 37.0 | 61.2 |
描述重複商品名稱時的 P(cover) |
0.97 | 0.75 |
| 單一注意力頭放在前一個 token 上的最大權重 | 0.35 | 0.79 |
沿著各欄往下讀,就能看出本章的規律。SmolLM2 在每一項規模指標上都大得多,而且兩個模型都學會了相同的基本行為:複製,以及注意前一個 token。凡是與商店自身文字有關的項目,都是小模型占優勢,因為那是它學過的全部內容。商店要做的選擇,不在於小模型或大模型,而在於各自負責哪一項工作。
實用原則: 把你要從中挑選的每個模型,依對你的工作重要的指標並排比較,每一欄都使用相同的文字與相同的測試。要依工作選擇模型,而不是用一個模型包辦所有事。
正式環境中會出什麼問題
開放模型會帶著其開發者的選擇一起進來,而這些選擇不會主動說明:
- 它的文字不是你的文字。 SmolLM2 對商店名稱的評分遠差於商店自己的小型模型,加上標籤也沒有幫助。在信任任何模型之前,先用你自己的文字衡量它。
- 它的 tokenizer 不是你的 tokenizer。 兩種 tokenizer 產生的逐 token 數值無法互相比較;請在相同的事件上,以每個字詞或每個字元計分。
- 模型的細節必須與執行它的程式碼相符。 分組的鍵與值(grouped keys and values)、100,000 的 RoPE base,以及綁定的嵌入(tied embeddings),都記載於 SmolLM2-360M 的設定中。第 7 章衡量過錯誤的 RoPE base 要付出什麼代價:準確度略為下降,而且不會出現錯誤。請使用開發者發布的程式碼載入模型,並採用鎖定版本,或對照該程式碼逐項檢查每個細節。
- 修訂版本會變動。 模型的儲存庫可以在同一個名稱下被更新。請鎖定修訂版本,如同
recordings/open_model.py的做法,並將它與每一筆結果一起記錄下來。 - 第一個 token 很重要。 SmolLM2 後面的大多數注意力頭都依賴第一個 token。裁剪上下文時要保留它,也絕不要讓處理流程在無聲無息中把它丟掉。
- 大小決定速度。 每產生一個 token 都要執行每一層,因此大了幾百倍的模型,每個 token 所需的運算量也大了幾百倍。在承諾回應時間之前,請在你將用來提供服務的機器上,以你將提供的文字長度,衡量每個 token 的耗時。
- 授權與條款會隨模型而來。 SmolLM2 採用 Apache 2.0;其他開放模型有其他條款。在以某個模型為基礎進行建置之前,請檢查該確切模型與修訂版本的授權。
本章的決定
- 本書使用的開放模型是 SmolLM2-360M,資料取自兩份記錄,兩者都固定在特定修訂版本:一份是它在商店文字上的執行結果,另一份是它所屬系列的各種設定。每個範例的執行都不需要 PyTorch,也不需要下載,結果可以與記錄相互核對。
- 使用不同 tokenizers 的模型,以每個字為單位來比較,採用相同的文字與相同的事件:每個名稱的各個字與其結尾,從起始 token 開始計算;每則描述的各個字與其結尾,以名稱為條件。
- 在商店的文字上,商店的小型模型勝過 SmolLM2:名稱上每個字為 199.3,對 54,352.8;描述上為 37.0,對 61.2。搜尋框沿用第 8 章所選定的組合,也就是商店模型與其計數的混合。
- 各模型的注意力頭以相同的度量來比較,基準是平均分布。SmolLM2 呈現出第 6 章的兩種模式,而且更為明顯:少數注意力頭專門處理前一個 token,許多注意力頭則把權重放在第一個 token 上。
- 從文字前段複製是基本能力,4 層的模型與 32 層的模型都具備,而且兩者以相同的方式測試。
- 模型的成本在提供服務之前,就能從它的設定算出:包括它的權重,加上每個進行中的文字各一份鍵與值的快取。在 SmolLM2 的各種大小中,分組的鍵與值使 360M 模型的快取減為三分之一。
本章,於本版次中
版次 2026.11.16範例於 2026年11 月 19 通過測試本章 7 個範例 個範例中有 7 個通過
譯自英文書,版次 2026.11.16。