Inside an open model

阅读约需 52分钟 分钟,10节中含7个示例。

版次 2026.11.16 示例于 2026年11月19日 通过 86个示例中有86个通过 变化内容

AI 翻译。由人工以英文核验。 我们如何制作图书

译自英文版图书,版次 2026.11.16。 阅读原文

这部分尚未翻译,因此以英文显示。

By the end of this chapter you can read an open language model with what this book has built: count where its parameters go, work out what it costs to run, score text with it fairly against a model with a different tokenizer, find the attention patterns of chapter 6 in its heads, test what it copies, and learn what its size buys on the store's text, and what it doesn't.

Chapter 9 ended with descriptions from the store's own model that need checking before anyone reads them. That model is the small transformer that chapters 6 to 9 built, trained and used on one furniture and home-goods catalog. SmolLM2-360M, an open model released by Hugging Face under the Apache 2.0 licence, is built from the same parts, as chapter 7 showed. It has more than two hundred times as many parameters, and was trained on 4 trillion tokens (model card) of web text, code, mathematics and synthetic textbooks (Allal et al., 2025, section 6). The question this chapter asks is whether a free, downloadable model trained on that much more text could do the jobs of the store's own model: suggesting the next word in the search box, and predicting descriptions. Chapter 2 read SmolLM2's tokenizer; this chapter reads the model.

Running SmolLM2 needs PyTorch and a download of about 0.7 gigabytes, its weights stored as 16-bit numbers, so it ran once, like the book's other recorded results. The script recordings/open_model.py loaded the base model, the one trained only to predict the next token, as the store's model was, at a pinned revision, a fixed version of its repository. It ran the model on the store's text and saved what this chapter reads: every weight's shape, its scores on the store's validation names and descriptions, its heads' attention habits, chapter 6's copying test and chapter 8's search-box prefixes. A second script, recordings/open_model_family.py, downloaded only the configuration files of SmolLM2's three sizes, also at pinned revisions. The examples read both recordings, in examples/data/recorded/, through lib.open_model.

这部分尚未翻译,因此以英文显示。

Count the parameters at scale

最近一次修改见于版次 2026.11.11

The parts are familiar, so the first question is where the parameters sit. Chapter 7 counted the store's model by part: 1,334,144 parameters, 40% of them in the embedding table it shares with its output.

Run the same count on SmolLM2's recorded weights, and watch which part now holds the most:

open-model-parameters.py Python 3.12.15 通过
"""Counts SmolLM2-360M's recorded weights by part, with the embedding table shared with the output counted once.The point that the feed-forward layers hold most of the parameters rests on the feed-forward line."""import mathfrom lib import open_modelshapes = {name: shape for name, shape in open_model.read("parameters").items()          if name != "lm_head.weight"}       # the output layer is tied to the embeddings: the same weights, counted once# A part is matched by name: a weight counts toward the part whose key appears in the weight's name.parts = {"embedding (shared with the output)": ["embed_tokens"], "attention": ["q_proj", "k_proj", "v_proj", "o_proj"],         "feed-forward": ["gate_proj", "up_proj", "down_proj"], "normalization": ["layernorm", "model.norm"]}total = sum(math.prod(shape) for shape in shapes.values())for part, keys in parts.items():    count = sum(math.prod(shape) for name, shape in shapes.items() if any(key in name for key in keys))    print(f"{part}:", f"{count:,}", f"({count / total:.0%})")print("total:", f"{total:,}")config = open_model.read("manifest")["config"]     # the configuration recorded with the weightsprint("layers:", config["num_hidden_layers"])print("width:", config["hidden_size"])print("query heads:", config["num_attention_heads"])print("key and value heads:", config["num_key_value_heads"])print("feed-forward width:", config["intermediate_size"])print("vocabulary:", f"{config['vocab_size']:,}")# Layer 0 stands for every layer: each layer's weights have the same shapes.print("one layer's query weights:", shapes["model.layers.0.self_attn.q_proj.weight"])print("one layer's key weights:", shapes["model.layers.0.self_attn.k_proj.weight"])
输出python open-model-parameters.py
embedding (shared with the output): 47,185,920 (13%)
attention: 78,643,200 (22%)
feed-forward: 235,929,600 (65%)
normalization: 62,400 (0%)
total: 361,821,120
layers: 32
width: 960
query heads: 15
key and value heads: 5
feed-forward width: 2560
vocabulary: 49,152
one layer's query weights: [960, 960]
one layer's key weights: [320, 960]

The balance has shifted from the store's model. The feed-forward layers hold 235,929,600 (65%) of SmolLM2's 361,821,120 parameters. The embedding table, still shared with the output, holds 47,185,920 (13%), attention 78,643,200 (22%), and the normalization gains almost nothing. The blocks grow with the number of layers times the square of the width, and the embedding table, for a fixed vocabulary, only with the width, so as a model grows, the blocks take over.

Attention holds fewer parameters than 15 full heads would, because of grouped-query attention, which shares each key and value head among several query heads. Each layer has 15 query heads but only 5 heads of keys and values, each shared by 3 query heads (Ainslie et al., 2023). The weights show it. One layer's query weights have the shape [960, 960], and its key weights [320, 960], a third as many numbers. Per layer, with the head size being the width divided by the number of query heads:

attention parameters = 2 × width² + 2 × width × key and value heads × head size

for the query and output weights, then the key and value weights. During generation, a model keeps every layer's keys and values for every earlier token, so sharing them shrinks that store, as a later section measures; Ainslie and colleagues found it costs little in quality.

Working rule: Count an open model's parameters by part from its weights before you use it, and read the shapes for its design choices: tied embeddings, grouped keys and values, a gated feed-forward layer. Expect the blocks, not the embeddings, to hold most of a large model's parameters.

比较同一系列的不同规模

最近一次修改见于版次 2026.11.11

SmolLM2 有三种规模,每种都由相同的部件构成,它们的配置展示了模型的开发者如何增加参数。每个配置文件都给出了层数、宽度、注意力头数和前馈层宽度,凭这些就足以按第 7 章的算法和上文的公式,逐部件数出每一个参数。lib.open_model.parameters 完成的就是这项工作。

根据三种规模各自记录的配置来统计参数量,并把 360M 的统计结果与第一次记录所保存的权重相核对:

open-model-family.py Python 3.12.15 通过
"""Counts all three SmolLM2 sizes from their recorded configurations, part by part, and checks the 360M model's countagainst its recorded weights. The claim that the counts match the weights rests on the 360M parameters line and the last line."""import mathfrom lib import open_modelfor name in open_model.SIZES:    config = open_model.config(name)    parts = open_model.parameters(config)           # the count by part, from the configuration alone, grouped keys and values included    print(f"{name}, layers:", config["num_hidden_layers"])    print(f"{name}, width:", f"{config['hidden_size']:,}")    print(f"{name}, query heads:", config["num_attention_heads"])    print(f"{name}, query heads per key and value head:", config["num_attention_heads"] // config["num_key_value_heads"])   # query heads sharing each key and value head    print(f"{name}, feed-forward width:", f"{config['intermediate_size']:,}")    print(f"{name}, RoPE base:", f"{config['rope_theta']:,}")      # the rotary base: a larger base makes positions turn more slowly    print(f"{name}, parameters:", f"{parts['total']:,}")    print(f"{name}, embedding's share:", f"{parts['embedding'] / parts['total']:.0%}")# The output layer is tied to the embeddings, so its weights are skipped: the shared weights are counted once.recorded = sum(math.prod(shape) for key, shape in open_model.read("parameters").items() if key != "lm_head.weight")print("SmolLM2-360M, parameters in the recorded weights:", f"{recorded:,}")
输出python open-model-family.py
SmolLM2-135M, layers: 30
SmolLM2-135M, width: 576
SmolLM2-135M, query heads: 9
SmolLM2-135M, query heads per key and value head: 3
SmolLM2-135M, feed-forward width: 1,536
SmolLM2-135M, RoPE base: 100,000
SmolLM2-135M, parameters: 134,515,008
SmolLM2-135M, embedding's share: 21%
SmolLM2-360M, layers: 32
SmolLM2-360M, width: 960
SmolLM2-360M, query heads: 15
SmolLM2-360M, query heads per key and value head: 3
SmolLM2-360M, feed-forward width: 2,560
SmolLM2-360M, RoPE base: 100,000
SmolLM2-360M, parameters: 361,821,120
SmolLM2-360M, embedding's share: 13%
SmolLM2-1.7B, layers: 24
SmolLM2-1.7B, width: 2,048
SmolLM2-1.7B, query heads: 32
SmolLM2-1.7B, query heads per key and value head: 1
SmolLM2-1.7B, feed-forward width: 8,192
SmolLM2-1.7B, RoPE base: 130,000
SmolLM2-1.7B, parameters: 1,711,376,384
SmolLM2-1.7B, embedding's share: 6%
SmolLM2-360M, parameters in the recorded weights: 361,821,120

由 360M 模型的配置得到的统计结果与其记录的权重完全一致:统计值为 361,821,120,记录值为 361,821,120。最大的模型多出来的参数,大部分来自宽度,而不是深度:SmolLM2-1.7B 有 24 层,比两个较小规模的 30 层和 32 层都少,而它的宽度为 2,048,比两者的 576 和 960 宽得多。

嵌入表所占的比例随模型增大而下降,依次为 21%、13%、6%,原因是三者共用同一个词表,而它们的各个块在不断增大。此外,最大的模型不只是尺寸不同。它不对键和值分组,每个键值头对应 1 个查询头,而两个较小的模型则让 3 个查询头共享一个键值头。它的前馈层宽度为 8,192,是其宽度的四倍,而 360M 模型的前馈层宽度为 2,560,是其宽度的 8/3。它的旋转位置编码也转得更慢,基数为 130,000,而 360M 模型为 100,000。

工作准则: 根据配置统计模型的参数量,并用权重核对统计结果;两者不符,说明你误读了某个细节。比较同一系列的不同规模,以了解开发者如何放大这一设计,不要认为较大的模型与较小的模型只有尺寸上的差别。

计算运行成本

最近一次修改见于版次 2026.11.16

参数决定模型能学到什么,也决定运行它要付出什么。模型的权重必须常驻内存,在 16 位格式中每个权重占两个字节,SmolLM2 的权重即以这种形式存储(模型卡片)。而在生成过程中,正如第 9 章所指出的,每一层都为每个先前的词元保存一个键和一个值,这样每生成一个新词元时,就不必对其前面的文本重做计算。键值缓存,即每一层针对先前词元的键和值,会随文本长度而增长:

每个词元的缓存字节数 = 层数 × 2 × 键值头数 × 头维度 × 2

其中每层计一个键和一个值,每个数占两个字节。

做预算时,需要针对每种规模给出权重大小、每个词元的缓存,以及模型所设定的最长文本(8,192 个词元)对应的缓存,并分别列出使用与不使用分组键值的情形:

open-model-memory.py Python 3.12.15 通过
"""Works out each size's weights and key and value cache, with and without grouped keys and values, in decimal gigabytesand kilobytes. The point that grouping cuts the 360M model's cache to a third rests on its two cache lines for the longest text."""from lib import open_modellongest = open_model.config("SmolLM2-360M")["max_position_embeddings"]   # the longest text the models are set up forprint("longest text, in tokens:", f"{longest:,}")for name in open_model.SIZES:    config = open_model.config(name)    head_size = config["hidden_size"] // config["num_attention_heads"]   # the width shared out over the query heads, grouped or not    weights = open_model.parameters(config)["total"] * 2          # 16-bit numbers: 2 bytes each    cache = config["num_hidden_layers"] * 2 * config["num_key_value_heads"] * head_size * 2   # a key and a value per layer, 2 bytes each    # The same cache with every query head keeping its own key and value.    ungrouped = config["num_hidden_layers"] * 2 * config["num_attention_heads"] * head_size * 2    # Decimal units: a gigabyte is 1e9 bytes and a kilobyte 1e3 bytes.    print(f"{name}, weights in 16-bit numbers, in GB:", f"{weights / 1e9:.1f}")    print(f"{name}, key and value cache per token, in KB:", f"{cache / 1e3:.0f}")    print(f"{name}, cache for the longest text, in GB:", f"{cache * longest / 1e9:.2f}")    print(f"{name}, cache for the longest text without grouping, in GB:", f"{ungrouped * longest / 1e9:.2f}")
输出python open-model-memory.py
longest text, in tokens: 8,192
SmolLM2-135M, weights in 16-bit numbers, in GB: 0.3
SmolLM2-135M, key and value cache per token, in KB: 23
SmolLM2-135M, cache for the longest text, in GB: 0.19
SmolLM2-135M, cache for the longest text without grouping, in GB: 0.57
SmolLM2-360M, weights in 16-bit numbers, in GB: 0.7
SmolLM2-360M, key and value cache per token, in KB: 41
SmolLM2-360M, cache for the longest text, in GB: 0.34
SmolLM2-360M, cache for the longest text without grouping, in GB: 1.01
SmolLM2-1.7B, weights in 16-bit numbers, in GB: 3.4
SmolLM2-1.7B, key and value cache per token, in KB: 197
SmolLM2-1.7B, cache for the longest text, in GB: 1.61
SmolLM2-1.7B, cache for the longest text without grouping, in GB: 1.61

分组键值使 360M 模型的缓存保持在较小水平。它每个词元占 41 KB,因此一条最长长度的文本需要 0.34 GB,仅为不分组时所需 1.01 GB 的三分之一。

同时生成大量描述的服务,要为每条文本各保留一份缓存,因此在文本较长时,缓存可能超过权重本身。1.7B 模型不对键值分组,每个词元需要 197 KB,一条长文本需要 1.61 GB,约为其 3.4 GB 权重的一半。

工作准则: 模型的内存预算等于其权重加上每条处理中文本各一份缓存,并依据模型配置计算得出。当需要服务长文本或大量文本时,优先选用对键值分组的模型,并把允许的文本长度限制在预算所能承受的范围内。

用通用模型为商店文本打分

最近一次修改见于版次 2026.11.16

一个参数量为商店小模型两百倍、训练所用词元数是商店目录所含词元数一百万倍以上的模型,似乎理应比商店自己的小模型更能预测商店的文本。要检验这一点需要谨慎,因为两个模型使用不同的分词器:SmolLM2 的词表有 49,152 个词元,切分文本的方式与商店的不同,所以它们的每词元困惑度统计的是不同的对象。两者共有的是文本本身。每词困惑度用模型对一段文本给出的总对数概率除以文本中的词数,而不是除以词元数。用符号表示:一段文本含 W 个词,模型给出其各词元的概率 p₁ 至 pₙ,无论这需要多少个词元:

每词困惑度 = exp(−(log p₁ + … + log pₙ) / W)

两个模型的总值都除以同一个词数,无论各自用了多少词元,二者都落在同一尺度上。

两个模型对相同的事件打分。对于名称,商店的模型从 <s> 开始,SmolLM2 从它自己的起始词元 <|endoftext|> 开始,二者都预测名称的各个词,再预测其结尾:商店的模型预测 <sep>,SmolLM2 预测换行符。对于描述,二者都先得到名称,再预测描述的各个词及其结尾。名称取前 500 个验证集名称,描述取第 8 章的 446 件商品。录制的结果还对每个名称在标签 Product name: 之后打了分,以检验告诉 SmolLM2 它在读什么是否有帮助。

哪个模型对商店名称的每词得分更好,好多少?先自己判断,再往下读。然后用 NumPy 计算商店模型的得分,从录制结果中读出 SmolLM2 的得分,分别按每词元和每词进行比较:

open-model-on-the-store.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 通过
"""Scores the store's model and SmolLM2-360M on the same names and the same descriptions, per word, the unit the two share.The point that the small model wins on both kinds of text rests on the per-word lines for names and for descriptions."""import mathfrom lib import bpe, open_model, transformerfrom lib.products import product_textsfrom lib.store import load_products_, vocabulary, _ = transformer.load()           # load() returns the weights, the vocabulary and the manifest; only the vocabulary is usedindex = {token: i for i, token in enumerate(vocabulary)}scores, tokenizer = open_model.read("scores"), bpe.store_tokenizer()per = lambda log_p, count: f"{math.exp(-log_p / count):,.1f}"    # perplexity: e to the average negative log-probability per count# Names: each name's tokens and its end (<sep> for the store's model, a newline for SmolLM2), from the start token.names = list(load_products("validation")["name"])[:len(scores["names"])]     # as many names as SmolLM2 was scored on, from the top of the split# The start marker, the name's tokens and <sep>: the model scores everything after the start marker.encoded = [[index[transformer.START]] + [index[t] for t in tokenizer.encode(name)] + [index[transformer.SEP]] for name in names]store = sum(transformer.total_log_probability(ids, 1) for ids in encoded)     # scores ids[1:], every token after <s>smollm2 = sum(row["log_p"] for row in scores["names"])# Words are whitespace-separated, as the recording counted them; both models are divided by the same count.words = sum(row["words"] for row in scores["names"])print("validation names scored:", len(names))print("names, per-token perplexity, the store's model:", per(store, sum(len(ids) - 1 for ids in encoded)))   # all but <s>print("names, per-token perplexity, SmolLM2-360M:", per(smollm2, sum(row["tokens"] for row in scores["names"])))   # SmolLM2's own countprint("names, per-word perplexity, the store's model:", per(store, words))print("names, per-word perplexity, SmolLM2-360M:", per(smollm2, words))# The label is context for the recording and is not scored; the same words count.print("names after a label, per-word perplexity, SmolLM2-360M:", per(sum(r["log_p"] for r in scores["labelled names"]), words))# Descriptions: chapter 8's products, each description's tokens and its end, given the name.end = index[transformer.END]                    # the end marker closes a description# The first 500 validation products. Keeps those whose text ends rather than being cut at the length limit, and that have# at least one description token after <sep>. Scores from sep + 1: the description and its end, given everything up to <sep>.store = sum(transformer.total_log_probability(ids, sep + 1) for ids, sep in product_texts("validation", 500)            if ids[-1] == end and sep < len(ids) - 2)words = sum(row["words"] for row in scores["descriptions"])print("descriptions scored:", len(scores["descriptions"]))print("descriptions, per-word perplexity, the store's model:", per(store, words))print("descriptions, per-word perplexity, SmolLM2-360M:", per(sum(r["log_p"] for r in scores["descriptions"]), words))
输出python open-model-on-the-store.py
validation names scored: 500
names, per-token perplexity, the store's model: 25.8
names, per-token perplexity, SmolLM2-360M: 564.8
names, per-word perplexity, the store's model: 199.3
names, per-word perplexity, SmolLM2-360M: 54,352.8
names after a label, per-word perplexity, SmolLM2-360M: 67,101.5
descriptions scored: 446
descriptions, per-word perplexity, the store's model: 37.0
descriptions, per-word perplexity, SmolLM2-360M: 61.2

小模型在两类文本上都胜出,在名称上优势悬殊。按每词计,商店的模型在名称上得分为 199.3,SmolLM2 为 54,352.8;在描述上,商店的模型为 37.0,SmolLM2 为 61.2。按每词元计,在这些名称上,商店的模型为 25.8,SmolLM2 为 564.8。每词元的比较掩盖了差距,因为 SmolLM2 对商店的每个词要花更多词元。

告诉 SmolLM2 它读的是商品名称,并不能缩小差距。加上标签之后,它的每词得分为 67,101.5,并不比不加时更好。所以差距不在于 SmolLM2 不知道接下来是名称,而在于名称本身。我们的判断(此处未经检验)是,商店文本的两个特点在很大程度上解释了这一差距:许多名称以某个产品线虚构的名字开头,商店的模型在其目录中见过这些名字;而且商店的文本全部小写,标点两侧有空格,不同于 SmolLM2 所学的大部分文本。

这里的结论并不是小模型更好。模型最擅长预测的是它学习过的那类文本:SmolLM2 懂得的英语远多于商店的模型,却几乎不了解商店的这些习惯。要让它学会这些习惯,需要在商店的文本上继续训练,第 11 章对此有所说明。

工作准则: 比较分词器不同的模型时,应在完全相同的文本和事件上按每词或每字符比较,绝不按每词元比较。在选用大型通用模型而不用在自己文本上训练的小模型之前,先在自己的文本上测量两者;规模无法弥补文本与模型所学内容毫不相像的缺陷。

这部分尚未翻译,因此以英文显示。

Find the same heads at scale

最近一次修改见于版次 2026.11.16

Chapter 6 measured the store's model's heads by their habits, against an even spread, which puts 1 / (i + 1) of a position's weight on each of the i + 1 tokens position i can see. The same measures work on any model, and SmolLM2 has 480 heads to measure.

Look at each head's recorded average weight on the previous token and on the first token, over the first 50 of the 446 products scored above, each text a name, a newline and its description. The even spread is averaged the same way, so the two can be set side by side:

open-model-heads.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 · tokenizers 0.23.3 通过
"""Reads SmolLM2's recorded head averages over the recorded texts, and compares them with an even spread averaged the same way.The point that a few heads specialize and many put their weight on the first token rests on the head counts and the first-token averages."""import numpy as npfrom tokenizers import Tokenizerfrom lib import open_model, transformerfrom lib.products import product_textsfrom lib.store import load_productsheads = open_model.read("heads")previous, first = np.array(heads["previous"]), np.array(heads["first"])   # (layers, heads)print("texts averaged:", heads["texts"])print("heads in all:", previous.size)print("heads with more than half their weight on the previous token:", int((previous > 0.5).sum()))   # a position's weights sum to 1, so above 0.5 is most of itprint("the most weight a head puts on the previous token:", f"{previous.max():.2f}")print("average weight on the first token, layers 1 to 4:", f"{first[:4].mean():.2f}")     # rows count layers from 0: rows 0 to 3print("average weight on the first token, layers 5 to 32:", f"{first[4:].mean():.2f}")    # rows from 4 on are layers 5 to 32print("heads with more than half their weight on the first token:", int((first > 0.5).sum()))# An even spread, averaged the same way: each position i shares its weight equally among the i + 1 tokens it sees.smollm2 = Tokenizer.from_file("data/smollm2-tokenizer/tokenizer.json")end = transformer.load()[1].index(transformer.END)      # [1] is the vocabulary: load() returns weights, vocabulary, manifestrows = load_products("validation").head(500)            # the first 500 validation products, the ones product_texts reads# The same products as product_texts, with their names and descriptions, filtered the same way. Each text is a name, a newline# and a description, as the recording wrote it.texts = [f"{name}\n{description}\n" for (ids, sep), name, description in         zip(product_texts("validation", 500), rows["name"], rows["description"]) if ids[-1] == end and sep < len(ids) - 2]# Only the texts the recording averaged. The +1 is the start token the recording put before every text.lengths = [len(smollm2.encode(text, add_special_tokens=False).ids) + 1 for text in texts[:heads["texts"]]]# A position with k tokens to see gives an even spread 1 / k on each of them. k runs from 2 to the text's length.print("an even spread's weight on the previous token, and on the first:",      f"{np.mean([np.mean(1 / np.arange(2, n + 1)) for n in lengths]):.2f}")
输出python open-model-heads.py
texts averaged: 50
heads in all: 480
heads with more than half their weight on the previous token: 3
the most weight a head puts on the previous token: 0.79
average weight on the first token, layers 1 to 4: 0.13
average weight on the first token, layers 5 to 32: 0.61
heads with more than half their weight on the first token: 286
an even spread's weight on the previous token, and on the first: 0.05

A few heads specialize sharply. 3 heads put more than half their weight on the previous token, and the strongest puts 0.79 there, against an even spread's 0.05. The store's model's most previous-token head, measured the same way, put 0.35 on the previous token. With hundreds of heads, a model can afford to give a few of them one narrow job.

The first token's weight is the larger surprise. In the first four layers, heads put 0.13 of their weight on it on average; from layer 5 on, 0.61, against an even spread's 0.05, and 286 heads put more than half there. That's the attention sink that Xiao and colleagues found in large models (Xiao et al., 2024). They explain it by softmax: a head with nothing useful to attend to still has to put its weight somewhere, and the first token is always there. Chapter 6 found the pattern only weakly in the store's model, whose description tokens put 0.11 of their attention on its two markers, against an even spread's 0.05.

Working rule: Describe a large model's heads by the same measured averages you'd use on a small one, compared with an even spread. Expect a few specialized heads and many that put their weight on the first token, and keep that token when you cut a model's context to save memory.

这部分尚未翻译,因此以英文显示。

Test copying at scale

最近一次修改见于版次 2026.11.11

Chapter 6 showed the store's model copying cover from a product's name into its description, by changing the name and watching the prediction follow. The same test asks whether SmolLM2 does it too. Whether its training text held this product's text can't be checked, and the test doesn't need it: if SmolLM2 predicted cover from memory of the product, taking cover out of the name wouldn't move it.

Check SmolLM2's recorded probabilities of cover and set at one point in the description of double chaise patio furniture cover, after this outdoor patio furniture, with the name as sold, without cover, and ending in set:

open-model-copying.py Python 3.12.15 通过
"""Prints SmolLM2's recorded probabilities of cover and set after the same words of a description, under three versions of the name.The point that each word follows its own name rests on the cover lines for the name as sold and without cover, and the set line for the name ending in set."""from lib import open_modelcopying = open_model.read("copying")     # keyed by the version of the name, each with the probabilities of the tokens after the same wordsfor label, probabilities in copying.items():    for token, p in probabilities.items():        print(f"SmolLM2-360M, {label}, P({token.strip()}):", f"{p:.2f}")     # the recorded tokens carry the space before them; strip removes it
输出python open-model-copying.py
SmolLM2-360M, name as sold, P(cover): 0.75
SmolLM2-360M, name as sold, P(set): 0.00
SmolLM2-360M, name without cover, P(cover): 0.02
SmolLM2-360M, name without cover, P(set): 0.02
SmolLM2-360M, name ending in set, P(cover): 0.01
SmolLM2-360M, name ending in set, P(set): 0.77

Both models copy the next word of an earlier phrase when the phrase repeats. With the name as sold, SmolLM2 gives cover 0.75. Take cover out of the name and it falls to 0.02; end the name in set and set gets 0.77.

The store's model behaved the same way, with 0.97, 0.00 and 0.70 for the name ending in set. That copying is the behaviour of induction heads, heads that find an earlier copy of the current token and attend to what followed it. Olsson and colleagues found them in every model they studied with more than one layer (Olsson et al., 2022).

So copying isn't something scale adds; the store's 4-layer model learned it too. For a store like this one, the habit is useful and risky at once: a model that copies words from the product's name stays on the product, and one that copies from the wrong product's text spreads its mistake.

Working rule: Test a behaviour by changing the input and watching the output, and run the same test on every model you compare. When two models of very different sizes share a behaviour, it's a basic one; look for what differs elsewhere.

这部分尚未翻译,因此以英文显示。

Suggest the next word at scale

最近一次修改见于版次 2026.11.11

The search box is where the store's text matters most. Chapter 8 found the store's model alone giving stool a probability of 0.50 after outdoor bar, and the mix of the store's model and its counts, the model the search box serves, 0.28.

Here are SmolLM2's three likeliest next tokens after the same five prefixes, each given with nothing before it, the way the store's model got them:

open-model-autocomplete.py Python 3.12.15 通过
"""Prints SmolLM2's three likeliest next tokens after five search-box prefixes, and its probability of stool after outdoor bar.The point that its suggestions are the wrong text rests on the top token after each prefix, and on the stool line."""from lib import open_modelfor prefix, recorded in open_model.read("prefixes").items():    shown = ", ".join(f"{token!r} {p:.2f}" for token, p in recorded["top"][:3])   # the three likeliest tokens, as recorded    print(f"{prefix:<12} -> {shown}")# The recorded key keeps the space before stool. Four places: at two, this probability would print as 0.00.print("P(stool | outdoor bar), SmolLM2-360M:", f"{open_model.read('prefixes')['outdoor bar']['P( stool)']:.4f}")
输出python open-model-autocomplete.py
bar          -> 'ry' 0.05, 'ley' 0.05, 'b' 0.04
patio bar    -> '\n' 0.21, 'be' 0.06, ',' 0.06
outdoor bar  -> '\n' 0.28, ',' 0.06, '\n\n' 0.04
kitchen bar  -> '\n' 0.37, '\n\n' 0.03, ',' 0.02
wine bar     -> '\n' 0.20, ',' 0.09, '\n\n' 0.03
P(stool | outdoor bar), SmolLM2-360M: 0.0002

SmolLM2's suggestions read like general English, not like the store's search box. After outdoor bar, patio bar, kitchen bar and wine bar, its likeliest next token is a newline: it takes each prefix as a complete line, likely because short lines like these end in a newline in much of the text it learned from. Its probability of stool after outdoor bar is 0.0002. After bar alone, it suggests pieces of longer words, such as ry and ley, as in barry or barley. None of that is wrong English; it's the wrong text. A search box needs the store's habits, and the store's own models have them. A search box also has to answer within a keystroke, and the store's model runs a small fraction of SmolLM2's arithmetic per token.

Working rule: Don't put a general model in front of shoppers without measuring its suggestions on your own prefixes against what you have. Where a model trained on your text suggests better, keep it, and measure a general model on the jobs where its broader knowledge could matter.

将两个模型并列比较

最近一次修改见于版次 2026.11.11

各项指标都测量完毕后,就可以把两个模型放进同一张表中比较。商店的模型是本书自己的模型,已在第 6 章至第 8 章中测量过;SmolLM2-360M 则通过本章中记录下来的数据来解读。

商店的模型 SmolLM2-360M
参数量 1,334,144 361,821,120
层数与宽度 4 层,宽度为 128 32 层,宽度为 960
每层的查询头数,以及键头和值头数 4 和 4 15 和 5
词表大小 4,142 49,152
文本量(以词元计):商品目录,或训练所用的词元数 2,843,222,即商店的商品目录 4 万亿,主要是通用文本
商店商品名称的逐词困惑度 199.3 54,352.8
商店商品描述的逐词困惑度 37.0 61.2
描述重复名称时的 P(cover) 0.97 0.75
单个注意力头放在前一个词元上的最大权重 0.35 0.79

顺着各列往下读,本章的规律就显现出来了。就各项规模指标而言,SmolLM2 都大得多,而两个模型都学会了相同的基本行为:复制,以及关注前一个词元。凡是涉及商店自身文本的指标,都对小模型有利,因为它学到的只有这些。商店面对的选择并不是小模型与大模型之间的取舍,而是让每个模型分别承担哪项工作。

工作准则: 把你要在其中做选择的每个模型,按对你的工作要紧的指标并列比较,每一列都使用相同的文本和相同的测试。按工作选择模型,而不是用一个模型包办一切。

生产环境中会出什么问题

最近一次修改见于版次 2026.11.11

开放模型会带着其开发者的种种选择,而这些选择不会主动显露:

  • 它的文本不是你的文本。 SmolLM2 对商店名称的评分远差于商店自己的小模型,加上标签也无济于事。在信任任何模型之前,先在你自己的文本上对它进行测量。
  • 它的 tokenizer 不是你的 tokenizer。 两种 tokenizer 给出的逐词元数值无法相互比较;应在相同的事件上按词或按字符评分。
  • 模型的细节必须与运行它的代码相符。 分组的键和值、取值为 100,000 的 RoPE base,以及绑定嵌入,都写在 SmolLM2-360M 的配置里。第 7 章测量过错误的 RoPE base 会带来什么代价:精度略有下降,却不会报错。请使用开发者发布的代码加载模型,并采用固定版本,或者对照该代码逐项检查每个细节。
  • 修订版本会变化。 模型的仓库可能在名称不变的情况下被更新。请固定修订版本,就像 recordings/open_model.py 所做的那样,并将它与每一项结果一起记录下来。
  • 第一个词元很重要。 SmolLM2 后面的大多数注意力头都依赖第一个词元。裁剪上下文时要保留它,并且绝不能让流水线悄悄丢掉它。
  • 规模决定速度。 每生成一个词元都要运行每一层,因此大几百倍的模型,每个词元所需的运算量也大几百倍。在承诺响应时间之前,应在你将用于提供服务的机器上,按你将要处理的文本长度,测量每个词元所用的时间。
  • 许可证和条款随模型而行。 SmolLM2 采用 Apache 2.0;其他开放模型附带其他条款。在以某个模型为基础进行构建之前,请检查该模型及其具体修订版本的许可证。

本章的决定

  • 本书采用的开放模型是 SmolLM2-360M,数据来自两份记录,二者都固定在指定版本:一份是它在商店文本上的运行记录,另一份是它所属系列的各项配置。每个示例的运行都不需要 PyTorch,也不需要下载,结果可以与记录下来的内容核对。
  • 使用不同 tokenizers 的模型按每个词来比较,比较基于相同的文本和相同的事件:对每个名称,计算其中各个词以及结尾,从起始标记开始;对每段描述,计算其中各个词以及结尾,并以名称为条件。
  • 在商店的文本上,商店的小模型胜过 SmolLM2:名称上每个词为 199.3,SmolLM2 为 54,352.8;描述上为 37.0,SmolLM2 为 61.2。搜索框沿用第 8 章选定的组合,即商店模型与其计数的混合。
  • 不同模型的注意力头用相同的度量来比较,并以均匀分布作为参照。SmolLM2 呈现出第 6 章的两种模式,而且更为鲜明:少数注意力头专门关注前一个标记,许多注意力头把权重放在第一个标记上。
  • 从文本前面部分复制是一种基本能力,4 层的模型和 32 层的模型都具备,并且两者用同样的方法检验。
  • 模型的成本在部署之前就可以根据其配置算出:包括它的权重,加上每个处理中的文本各一份键和值缓存。在 SmolLM2 的各种规模中,分组的键和值把 360M 模型的缓存减少到三分之一。

本章,在本版次中

版次 2026.11.16示例于 2026年11月19日 通过本章 7个示例 个示例中有 7 个通过

译自英文版图书,版次 2026.11.16。

引用本节

格式

每种格式均引用英文原书,本译本以其为准。

每种格式都会注明该版次。其链接会打开当前版次,节标题处会说明该节文本最后一次修改的时间。访问日期和检索日期即为您打开此页的当天。

本页内容

第 10 章,共 11 章: Inside an open model

  1. Count the parameters at scale
  2. 比较同一系列的不同规模
  3. 计算运行成本
  4. 用通用模型为商店文本打分
  5. Find the same heads at scale
  6. Test copying at scale
  7. Suggest the next word at scale
  8. 将两个模型并列比较
  9. 生产环境中会出什么问题
  10. 本章的决定

返回顶部全部章节