From predicting to following instructions

閱讀約需 61 分鐘 分鐘,10 節 中共有 9 個範例。

版次 2026.11.16 範例於 2026年11 月 19 通過測試 86 個範例中有 86 個通過 變更內容

由 AI 翻譯。經人工以英文驗證。 我們如何製作書籍

譯自英文書,版次 2026.11.16。 閱讀原文

此部分尚未翻譯,因此以英文呈現。

By the end of this chapter you can explain how a model that predicts the next token becomes one that follows requests, build the two training losses that do it, and measure what that change does and doesn't fix. You'll compare a base model with its instruction-tuned version on the same requests, measure how often each makes up a product detail, check whether its confidence warns you when it does, and find where in a longer context a model stops finding what it was given.

The furniture and home-goods store's product names and descriptions are the text this book models, and every model in the book so far has done one thing: predict the next token of text like its training text. Chapter 10 read SmolLM2-360M, an open model trained that way on general text, and found it scored the store's names far worse, per word, than the store's own small model, and its descriptions worse. People don't want a continuation of their text; they want an answer to their request. Models are taught that in two further steps of training.

Instruction tuning, or supervised fine-tuning, continues training a model on examples of requests paired with good responses, so that predicting the next token of such a pair means answering the request. Wei and colleagues found that tuning on many tasks phrased as instructions made a model better at tasks it had never seen (Wei et al., 2022). Preference tuning then trains a model to prefer the better of two responses to the same request. Ouyang and colleagues had people rank the responses, and trained with reinforcement learning from a reward model learned from those rankings (Ouyang et al., 2022); Direct Preference Optimization, DPO, does preference tuning without a reward model, with a simple loss on the preference pairs themselves (Rafailov et al., 2023).

SmolLM2-360M-Instruct is the base model of chapter 10 after both steps: supervised fine-tuning on public and curated datasets, then DPO on UltraFeedback (model card). UltraFeedback's preference pairs were scored by a large language model, not by people (Cui et al., 2023). This chapter calls SmolLM2-360M the base model and SmolLM2-360M-Instruct the tuned model. The script recordings/instruction_tuning.py ran both once, at pinned revisions, decoding greedily, so each answer is the model's likeliest and the run can be repeated exactly, and saved what they wrote and how likely they found each token it wrote. Every measure of the two models comes from those files.

此部分尚未翻譯,因此以英文呈現。

Build the two training losses by hand

最近一次變動於第 2026.11.16 版

Both steps of tuning are training, with the loop of chapter 8; what changes is the loss. Instruction tuning uses the ordinary next-token loss with one difference: only the response's tokens count. The request is read, as context, but the model isn't trained to predict it, since the point is to answer requests, not to write them.

Lay one product out as a request and a response, its name as the request and its description as the response, the way the store's model already reads it. Then compute that model's loss over every token, and over the response's tokens only:

instruction-loss-by-hand.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 通過
"""Computes the store's model's loss on one product laid out as a request and a response, over every token and overthe response's tokens only. The prose's point rests on the last printed line, the average loss over the response only."""import numpy as npfrom lib import transformerfrom lib.store import load_products_, vocabulary, manifest = transformer.load()index = {token: i for i, token in enumerate(vocabulary)}# the name is unique in the validation split, so its one row is the productproduct = load_products("validation").query("name == 'double chaise patio furniture cover'").iloc[0]# the text is cut at the model's longest length, so a long description may end earlyids = transformer.product_ids(product["name"], product["description"], index, manifest["settings"]["longest"])sep = ids.index(index[transformer.SEP])           # the name plays the request; the description, the response# the last position predicts nothing, since no token follows it, so its row is droppedprobabilities = transformer.softmax(transformer.forward(ids)[:-1])# position i predicts the token at i + 1, so ids[1:] holds each position's target# each token's loss, given the tokens before itloss = -np.log(probabilities[np.arange(len(ids) - 1), ids[1:]])response = np.arange(1, len(ids)) > sep                             # the description's tokens and its endprint("tokens predicted in the text:", len(loss))print("tokens the instruction loss counts, the response's:", int(response.sum()))print("average loss over every token:", f"{loss.mean():.3f}")print("average loss over the response only:", f"{loss[response].mean():.3f}")
輸出python instruction-loss-by-hand.py
tokens predicted in the text: 96
tokens the instruction loss counts, the response's: 90
average loss over every token: 2.267
average loss over the response only: 2.132

Training on the loss over the response alone, across many requests and responses, teaches a model to produce responses. For this product that loss is 2.132, against 2.267 over every token. The instruction loss counts 90 of the 96 tokens the model predicts: the description's tokens and its end.

DPO's loss works on a pair of responses to one request, one preferred and one rejected. DPO compares how much more likely the model being tuned makes each response than the reference model it started from, the model after instruction tuning, and rewards it for raising the preferred one more. With log-probabilities log π of each response under the tuned model and log π_ref under the reference, and a factor β that sets how far the model may move:

margin = β × [(log π(preferred) − log π_ref(preferred)) − (log π(rejected) − log π_ref(rejected))]

loss = −log σ(margin)

with σ the logistic function from chapter 4 (Rafailov et al., 2023, equation 7). Made-up log-probabilities keep the arithmetic in plain view, so this example uses them, with β = 0.1:

dpo-loss-by-hand.py Python 3.12.15 · numpy 2.5.5 通過
"""Computes the DPO margin and loss by hand for two made-up cases, with a fixed beta. The prose's point rests on thetwo loss lines, one for each case."""import numpy as npbeta = 0.1                                       # how far the tuned model may move from the referenceprint("beta:", beta)# Each case: (preferred, rejected) under the tuned model, then (preferred, rejected) under the reference model it# started from. Made-up numbers, to follow the arithmetic.cases = {"tuned model favours the preferred answer more than the reference does": (-12.0, -20.0, -14.0, -18.0),         "tuned model favours it less": (-15.0, -15.0, -14.0, -18.0)}for label, (preferred, rejected, preferred_reference, rejected_reference) in cases.items():    margin = beta * ((preferred - preferred_reference) - (rejected - rejected_reference))    loss = np.log1p(np.exp(-margin))             # -log σ(margin), written so it can't overflow    print(f"{label}, margin:", f"{margin:.2f}")    print(f"{label}, loss:", f"{loss:.3f}")
輸出python dpo-loss-by-hand.py
beta: 0.1
tuned model favours the preferred answer more than the reference does, margin: 0.40
tuned model favours the preferred answer more than the reference does, loss: 0.513
tuned model favours it less, margin: -0.40
tuned model favours it less, loss: 0.913

Lowering the loss pushes the preferred response's probability up and the rejected one's down, measured against where the model started. When the tuned model favours the preferred response more than the reference does, the margin is 0.40 and the loss 0.513; when it favours it less, the margin is -0.40 and the loss rises to 0.913.

Working rule: Mask the request out of the loss when you tune a model on requests and responses, and keep the reference model's log-probabilities beside the tuned model's when you train on preferences. Both are small changes to a loop you already have.

詢問基礎模型與調校後的模型

最近一次變動於第 2026.11.11 版

商店需要為沒有描述的商品撰寫描述,這項工作在第 9 章中交給了它自己的小型模型。基礎模型 SmolLM2-360M 無法被「詢問」,只能讓它「接續文字」,因此給它的是商品清單的開頭:Product:、商品名稱,以及 Description:。調校後的模型 SmolLM2-360M-Instruct 則收到一項請求:Write a short product description for: 與商品名稱。

兩個模型都從第 9 章的商品開始,也就是 front porch welcome sign wall décor。先閱讀各自寫出的內容,再衡量紀錄中描述過的全部 8 項商品,這些商品都沒有可供參考的描述:

base-and-tuned.py Python 3.12.15 通過
"""Prints what the base and the tuned model wrote for one product with no description, then measures every such product'stext for materials its name doesn't give and for repeated four-word phrases. The prose's points rest on the last fourprinted lines, the two models' invented-material counts and repeat shares."""from collections import Counterfrom lib import claims, recorded# the recording's saved texts for every product, read back rather than generated againdescribed = recorded.read("instruction-tuning", "descriptions")product = next(row for row in described if row["name"] == "front porch welcome sign wall décor")print("product:", product["name"])print("base model:", " ".join(product["base"]["text"].split()))print("tuned model:", " ".join(product["tuned"]["text"].split()))def repeats(text):    """The share of four-word phrases that repeat an earlier one in the text."""    words = text.lower().split()    # every window of four consecutive words, so a text of n words has n - 3 of them    phrases = [tuple(words[i:i + 4]) for i in range(len(words) - 3)]    # a phrase counts as repeated each time after its first appearance; max keeps a text of fewer than four words from dividing by zero    return sum(count - 1 for count in Counter(phrases).values()) / max(len(phrases), 1)print("products described, none of which has a description to draw on:", len(described))for role in ["base", "tuned"]:    texts = [row[role]["text"] for row in described]    # materials the text names that the product's name doesn't give: a set difference, not the text's whole list    invented = sum(bool(claims.materials(row[role]["text"]) - claims.materials(row["name"])) for row in described)    print(f"{role} model, descriptions that name a material the name doesn't:", f"{invented} of {len(described)}")    print(f"{role} model, repeated four-word phrases:", f"{sum(map(repeats, texts)) / len(texts):.0%}")
輸出python base-and-tuned.py
product: front porch welcome sign wall décor
base model: Front porch welcome sign wall décor The front porch is a common feature of many homes. It is a place where people can relax and enjoy the outdoors. It is also a place where people can socialize and meet new friends. The front porch is a great place to hang out with your family
tuned model: "Welcome to our front porch welcome sign wall décor, a space that brings warmth and comfort to your home. This wall is a statement piece that showcases your personality and style. With a mix of rustic and modern elements, you can create a unique and inviting atmosphere that reflects your individuality.
products described, none of which has a description to draw on: 8
base model, descriptions that name a material the name doesn't: 2 of 8
base model, repeated four-word phrases: 14%
tuned model, descriptions that name a material the name doesn't: 1 of 8
tuned model, repeated four-word phrases: 0%

基礎模型寫的是前門廊,The front porch is a common feature of many homes,接著談到放鬆與和朋友相聚。它朝著一個合理的方向接續文字,但那並不是商店所需要的方向。調校後的模型寫出了形式上像商品描述的文字,a statement piece that showcases your personality and style,卻把這塊招牌稱為 a space,又說 This wall:對於一項它並不了解的商品,文字相當流暢。在這些商品中,基礎模型的文字有 14% 的四詞片語重複出現,調校後的模型則是 0%。這份紀錄無法說明是調校的哪一個步驟停止了這些循環。

除了名稱之外,兩個模型對這些商品一無所知。在 2 of 8 項描述中,基礎模型寫出了名稱並未提及的材質;調校後的模型則有 1 of 8 項。這些材質都出自模型本身,而不是商店的資料,就像第 9 章中的情況一樣。

實用原則: 需要模型遵循請求時,使用調校後的模型;需要依照前文風格接續文字時,使用基礎模型。不要把流暢當成知識:調校後的模型在同樣缺乏事實的情況下,只是寫出形式更完整的描述。

此部分尚未翻譯,因此以英文呈現。

Format the request as the model was tuned

最近一次變動於第 2026.11.16 版

The tuned model answered in the previous section because its request came in the layout it was tuned on. A tuned model's chat template is the layout it learned: special tokens that mark where a system message, instructions placed before the user's request, the request itself and the model's reply begin and end. The model learned to answer requests only in that layout, so a request in any other form is only text to continue. Libraries that load the model apply the template from the model's tokenizer files (tokenizer configuration).

The tuned model's full prompt, and its reply beside the one it gives to the same request as plain text, show what it actually read:

chat-template.py Python 3.12.15 通過
"""Prints the full prompt the tuned model read, with its chat template, and its reply with and without the template. Theprose's point rests on the last two printed lines: the same words, with and without the template."""from lib import recordedtemplate = recorded.read("instruction-tuning", "template")# the request is the text between the user marker and its end markerprint("request:", template["with"]["prompt"].split("<|im_start|>user\n")[1].split("<|im_end|>")[0])print("what the tuned model reads, with its chat template:")print(template["with"]["prompt"])print("tuned model, with the template:", " ".join(template["with"]["text"].split()))print("tuned model, without it:", " ".join(template["without"]["text"].split()))
輸出python chat-template.py
request: Write a short product description for: front porch welcome sign wall décor
what the tuned model reads, with its chat template:
<|im_start|>system
You are a helpful AI assistant named SmolLM, trained by Hugging Face<|im_end|>
<|im_start|>user
Write a short product description for: front porch welcome sign wall décor<|im_end|>
<|im_start|>assistant

tuned model, with the template: "Welcome to our front porch welcome sign wall décor, a space that brings warmth and comfort to your home. This wall is a statement piece that showcases your personality and style. With a mix of rustic and modern elements, you can create a unique and inviting atmosphere that reflects your individuality.
tuned model, without it: .

Tuning taught the model to follow requests in one layout, not requests in general. With the template, the request sits between <|im_start|>user and <|im_end|>, after a system message the template adds by default, You are a helpful AI assistant named SmolLM, trained by Hugging Face, and before <|im_start|>assistant, where the model's reply begins. The model replies with the description of the previous section. Without the template, given the same words as plain text, the tuned model writes . and stops.

The system message matters too: it's part of every request the model reads, so a different one, such as a description of the store and its tone, can change any reply. That makes it a setting to choose and record, like a decoding setting in chapter 9.

Working rule: Always format requests to a tuned model with its own chat template, applied by the library from the model's tokenizer files, never assembled by hand. Check the template's default system message, since the model reads it on every request, and record the full formatted prompt with every answer you keep.

此部分尚未翻譯,因此以英文呈現。

Measure made-up product details

最近一次變動於第 2026.11.16 版

A model that writes about a product with no facts has to make them up. The fairer test gives it the facts. Both SmolLM2 models are shown a product's name and real description, and asked: What material is this product made of? Answer in a few words, using only the description. The tuned model gets it as a request in its template; the base model gets it as a listing to continue, ending Question:, the question, and Answer:. If the product's data names a material, a good answer names it; if not, a good answer says the description doesn't say.

Checking the answers takes a list of material words, from wood and metal to polyester and leather, in the book's shared module lib.claims. An answer is supported if every material it names is in the product's name or description. An answer is made up if it names a material that the product's name and description don't contain. An answer is declined if it says the product's data doesn't say, and none if it names no material and doesn't decline. The check has limits, and makes judgement calls: it counts engineered wood such as mdf and rubberwood as wood, it counts brushed nickel finish as naming nickel, and it misses a material the list doesn't include. Every answer it classes is recorded for anyone to read. Making up a detail the input doesn't support is one kind of what's called hallucination (Ji et al., 2023), and this test measures exactly that kind.

Class each model's answers for 40 products drawn at random, split by whether the product's data names a material. Where the data names none, will the tuned model make up fewer materials than the base model? Decide before you look.

made-up-materials.py Python 3.12.15 通過
"""Classes both models' answers to the material question, split by whether the product's name or description names amaterial. The prose's point rests on the rows for products whose data names none: made up and declined."""from collections import Counterfrom lib import claims, recordedasked = recorded.read("instruction-tuning", "answers")print("products asked, each with its description given:", len(asked))# a product's data names a material if its description or its name does: the union of the two setsgiven = [bool(claims.materials(row["description"]) | claims.materials(row["name"])) for row in asked]print("products whose name or description names a material:", sum(given))print("products whose name and description name none:", len(given) - sum(given))for role in ["base", "tuned"]:    for label, keep in [("data names a material", True), ("data names none", False)]:        rows = [row for row, named in zip(asked, given) if named == keep]        # each answer is classed against the product's data, by the rules in lib.claims        counts = Counter(claims.classify(row[role]["text"], row["description"], row["name"]) for row in rows)        for kind in ["supported", "made up", "none", "declined"]:            print(f"{role} model, {label}, {kind}:", counts[kind])
輸出python made-up-materials.py
products asked, each with its description given: 40
products whose name or description names a material: 28
products whose name and description name none: 12
base model, data names a material, supported: 24
base model, data names a material, made up: 2
base model, data names a material, none: 2
base model, data names a material, declined: 0
base model, data names none, supported: 0
base model, data names none, made up: 7
base model, data names none, none: 5
base model, data names none, declined: 0
tuned model, data names a material, supported: 23
tuned model, data names a material, made up: 3
tuned model, data names a material, none: 2
tuned model, data names a material, declined: 0
tuned model, data names none, supported: 0
tuned model, data names none, made up: 8
tuned model, data names none, none: 4
tuned model, data names none, declined: 0
Answer Base, data names a material Tuned, data names a material Base, data names none Tuned, data names none
Supported 24 23 0 0
Made up 2 3 7 8
None 2 2 5 4
Declined 0 0 0 0

On the 12 products whose name and description name none, the right answer is that the description doesn't say, and neither model gave it once. Both mostly made a material up instead, and the rest of their answers named none without saying why. Of the products, 28 have data that names a material, and on those both models mostly answer from it.

Instruction tuning didn't fix this, and on these products the base model, given a listing to continue, did no worse. The request said to use only the description; when the description had no answer, the models answered anyway, with a material that products like this one often have. Following the form of an instruction isn't the same as knowing when you don't know.

Working rule: Test a model on questions its input can't answer, not only on ones it can, and count made-up answers separately. Check every factual claim a model makes about a product against the product's data, automatically where you can, as lib.claims does for materials.

檢查信心是否能發出警訊

最近一次變動於第 2026.11.11 版

上一節將調校後模型對材料問題的回答分為有依據與捏造兩類。如果它捏造的回答伴隨著較低的信心,低信心就可以用來標示這些回答。信心與正確率相符的模型稱為校準良好(calibrated)。Guo 等人發現,他們研究的深度神經網路往往並非如此:這些網路的信心高於其正確率所能支持的程度(Guo et al., 2017)。錄製的資料保存了調校後模型為其寫出的每個 token 所給的機率,包括結束 token,因此可以為每個回答計算每個 token 的信心,做法是取幾何平均數:

每個 token 的信心 = exp((log p₁ + … + log pₙ) / n)

這就是平均對數機率的指數,與困惑度(perplexity)所反轉的平均值相同,所以短回答與長回答可以在同一個尺度上衡量。

信心能分辨這兩種回答嗎?比較調校後模型對有依據回答與捏造回答的信心,並計算以每個 token 0.5 為門檻時,兩類各會保留多少個:

answer-confidence.py Python 3.12.15 · numpy 2.5.5 通過
"""Compares the tuned model's confidence in its supported and made-up answers to the material question, and counts what athreshold of 0.5 per token keeps. The prose's point rests on the two average confidences and the two threshold counts."""import numpy as npfrom lib import claims, recordedasked = recorded.read("instruction-tuning", "answers")# only supported and made-up answers are compared; declined and none answers are left outby_kind = {"supported": [], "made up": []}for row in asked:    kind = claims.classify(row["tuned"]["text"], row["description"], row["name"])    if kind in by_kind:        # the geometric mean of the token probabilities: exp of their average log-probability        by_kind[kind].append(np.exp(np.mean(np.log(row["tuned"]["token_probabilities"]))))for kind, confidence in by_kind.items():    print(f"tuned model, {kind} answers:", len(confidence))    print(f"tuned model, {kind} answers, average probability per token:", f"{np.mean(confidence):.2f}")# a fixed cut for the comparison, not one fitted to these answersthreshold = 0.5for kind, confidence in by_kind.items():    print(f"tuned model, {kind} answers above {threshold} per token:", f"{sum(c > threshold for c in confidence)} of {len(confidence)}")
輸出python answer-confidence.py
tuned model, supported answers: 23
tuned model, supported answers, average probability per token: 0.65
tuned model, made up answers: 11
tuned model, made up answers, average probability per token: 0.45
tuned model, supported answers above 0.5 per token: 21 of 23
tuned model, made up answers above 0.5 per token: 3 of 11

信心是一種訊號,而不是過濾器:它會改變機率,卻仍會放過有信心的錯誤。捏造回答的平均信心較低,每個 token 為 0.45,有依據的回答則為 0.65。但以每個 token 0.5 為門檻,會保留 21 of 23 個有依據的回答,同時仍會放過 3 of 11 個捏造的回答。由於回答數量很少,這些結果只能看作趨勢,不能當成可靠的比率。

**實用原則:**不要僅憑模型的信心來決定是否信任某個回答。請在你自己的問題上衡量其信心與正確與否的關係;若要使用門檻,就依這些量測結果來選擇,並計算它放過了多少錯誤。

找出較長的上下文會遺失什麼

最近一次變動於第 2026.11.11 版

材質問題一次只給模型一項商品。商店的助理往往會同時面對多項商品的資料,例如搜尋結果、分類頁面或比較表,並且必須在其他商品之中找出正確商品的詳細資訊。Liu 等人發現,模型對長上下文開頭與結尾的資訊,運用得比中間的資訊更好(Liu et al., 2024)。

將被詢問的商品放在 10 個商品列表之中,位置分為第一個、中間或最後一個。接著詢問調校後的模型 SmolLM2-360M-Instruct,該商品是用什麼材質製成的。針對資料中載明材質的 14 項商品,計算答案中只列出該商品自身材質的次數,計算方式依 lib.claims 的檢查為準:

answer-position.py Python 3.12.15 通過
"""Counts the tuned model's answers that name only the asked product's own materials, with the product first, in the middleor last of a context of listings. The prose's point rests on the three placement lines."""from collections import defaultdictfrom lib import claims, recordedcontext = recorded.read("instruction-tuning", "context")# the other listings in each context; the asked product is one more, as the first printed line showsothers = recorded.read("instruction-tuning", "manifest")["settings"]["context_others"]print("product listings in each context:", others + 1)results = defaultdict(list)for row in context:    # only products whose data names a material: a product with none gives an answer nothing to name    if claims.materials(row["description"]) | claims.materials(row["name"]):        # supported: every material the answer names is in the asked product's own data        results[row["place"]].append(claims.classify(row["tuned"]["text"], row["description"], row["name"]) == "supported")print("products asked about at each place:", len(results["first"]))for place in ["first", "middle", "last"]:    print(f"product placed {place}, answers that name only its own materials:", f"{sum(results[place])} of {len(results[place])}")
輸出python answer-position.py
product listings in each context: 10
products asked about at each place: 14
product placed first, answers that name only its own materials: 12 of 14
product placed middle, answers that name only its own materials: 6 of 14
product placed last, answers that name only its own materials: 12 of 14

位於中間的商品最少被找到,這與 Liu 等人在大得多的模型中發現的模式相同。商品放在最前面時,有 12 of 14 個答案只列出其自身的材質;放在最後面時為 12 of 14;放在中間時為 6 of 14。計數雖然不多,差距卻很大。這裡的上下文很短,只有幾千個字元,遠低於模型的上限:中間位置的細節在上下文遠未填滿之前就已經遺失。

實用原則: 將模型最需要的細節放在上下文的開頭或結尾,而不是中間,並且在任務允許的範圍內,盡量減少提供給模型的無關文字。在信任模型能於大量資料中找出細節之前,應先在每個位置測試從較長上下文中的檢索效果。

此部分尚未翻譯,因此以英文呈現。

Adapt a model to your text

最近一次變動於第 2026.11.16 版

Chapter 10 found the general model far worse than the store's small model on the store's names, per word. Instruction tuning changes how a model answers, not what it knows about one store. The way to teach a general model the store's text is fine-tuning: continue training it on that text, the same training loop as chapter 8, starting from the general model's weights instead of random ones.

Full fine-tuning updates every weight, which for a model of hundreds of millions of parameters takes time and memory. LoRA, low-rank adaptation, freezes the model's weights and trains a small correction beside some of them. For a weight matrix that maps k numbers to d numbers, it trains a matrix B of d by r and a matrix A of r by k, and adds their product to the weights: W + BA (Hu et al., 2022). With a small rank r, there's far less to train and store, and Hu and colleagues found it matched full fine-tuning on the tasks they tested.

A LoRA correction would train far fewer weights; count them on SmolLM2-360M, adapting the query and value weights of every layer, at two ranks, against full fine-tuning. The query weights map 960 numbers to 960; the value weights, because keys and values are grouped, map 960 to 320:

lora-parameters.py Python 3.12.15 通過
"""Counts what LoRA would train on SmolLM2-360M, adapting the query and value weights of every layer at two ranks, andwhat full fine-tuning trains. The prose's point rests on the rank 8 lines and the full fine-tuning count."""from lib import open_modelconfig = open_model.config("SmolLM2-360M")width, layers = config["hidden_size"], config["num_hidden_layers"]# keys and values share heads, so their width is the query width scaled by the ratio of key-value heads to headskey_value_width = config["num_key_value_heads"] * width // config["num_attention_heads"]# every parameter of the model: the full fine-tuning count, and the denominator of each sharetotal = open_model.parameters(config)["total"]for rank in [8, 64]:    # LoRA adds two small matrices beside each adapted weight: width x rank, then rank x output width.    added = layers * (rank * (width + width) + rank * (width + key_value_width))       # on the query and value weights    print(f"rank {rank}, parameters trained:", f"{added:,}")    print(f"rank {rank}, share of the model's parameters:", f"{added / total:.2%}")print("parameters trained by full fine-tuning:", f"{total:,}")
輸出python lora-parameters.py
rank 8, parameters trained: 819,200
rank 8, share of the model's parameters: 0.23%
rank 64, parameters trained: 6,553,600
rank 64, share of the model's parameters: 1.81%
parameters trained by full fine-tuning: 361,821,120

At rank 8, LoRA trains 819,200 parameters, 0.23% of the model, where full fine-tuning trains all 361,821,120. A rank-8 correction is smaller than the store's own small model, and it can be stored and swapped per task while the base weights stay shared.

This book doesn't run the fine-tuning: our judgement is that even LoRA, over the store's text, would take far longer on a laptop's processor than any recording here. It's the next step for a store that wants a general model to write in its own voice: fine-tune on the catalog, then measure the result against the store's own model, per word, as chapter 10 did. Fine-tuning has risks of its own. A model trained on the catalog learns its gaps along with its text, as the store's small model learned to leave a bulb's description empty in chapter 9. And training on one narrow text can weaken what the model knew before; Biderman and colleagues found LoRA forgets less of it than full fine-tuning (Biderman et al., 2024). Both are measured the same way as everything else in this book: on held-out text, before and after.

Working rule: To teach a general model your text, fine-tune it on that text, with LoRA when memory or storage is tight, and measure the result on held-out text of the same kind. Don't expect instruction tuning or a cleverer prompt to supply knowledge the model never learned.

回顧評量標準

最近一次變動於第 2026.11.12 版

本書從第 3 章起,一律以同一種方式為各個模型在名稱資料上評分:以商店分詞器的詞元為單位計算每個詞元的困惑度(perplexity),資料則是商店的驗證名稱。測試名稱則另外保留,讓依據驗證名稱所做的選擇,例如保留哪幾個訓練週期(epoch)與混合比例,不會讓結果顯得過於樂觀。這次最後的比較,是在 4,281 測試名稱上為先前比較過的五個模型評分,混合比例則在驗證資料上選定:

test-names.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 通過
"""Scores the five models the book chose between on the test names, each name's tokens and its end, and prints eachmodel's test perplexity. The prose's point rests on the five test perplexity lines."""import numpy as npfrom lib import neural, transformerfrom lib.names import name_tokensfrom lib.ngrams import KneserNey, ngrams, padded# The test names, untouched until now: every model scores each name's tokens and its end. The mixing shares are the# ones chosen on the validation names, in chapters 5 and 8.test = name_tokens("test")counted = KneserNey(name_tokens("train"), 4)# padded adds the start marker three times and the end marker, so each name's first tokens have a context and its end is scoredfrom_counts = np.array([counted.probability(c, t) for text in test for c, t in ngrams(padded(text, 4), 4)])from_window = neural.token_probabilities(test)_, vocabulary, _ = transformer.load()index = {token: i for i, token in enumerate(vocabulary)}from_transformer = []for name in test:    # the transformer reads the start marker, the name, then <sep>, which ends a name where the counts end with </s>    ids = [index[transformer.START]] + [index[t] for t in name] + [index[transformer.SEP]]    probabilities = transformer.softmax(transformer.forward(ids)[:-1])    # position i predicts token i + 1, so ids[1:] is each token's target, the end of the name included    from_transformer += list(probabilities[np.arange(len(ids) - 1), ids[1:]])from_transformer = np.array(from_transformer)# pooled over every token of every name, not averaged per nameperplexity = lambda p: f"{np.exp(-np.log(p).mean()):.1f}"print("test names:", f"{len(test):,}")print("test perplexity, the 4-gram baseline:", perplexity(from_counts))print("test perplexity, the window model:", perplexity(from_window))# the mix averages the probabilities, not their logs: a share of 0.2 is 20% of the window modelprint("test perplexity, the window model mixed with the counts, share 0.2:", perplexity(0.2 * from_window + 0.8 * from_counts))print("test perplexity, the store's transformer:", perplexity(from_transformer))print("test perplexity, the store's transformer mixed with the counts, share 0.5:", perplexity((from_transformer + from_counts) / 2))
輸出python test-names.py
test names: 4,281
test perplexity, the 4-gram baseline: 22.9
test perplexity, the window model: 35.4
test perplexity, the window model mixed with the counts, share 0.2: 21.6
test perplexity, the store's transformer: 23.7
test perplexity, the store's transformer mixed with the counts, share 0.5: 18.3

依據驗證資料所做的選擇,對結果的美化非常有限:測試欄的數字與驗證欄相當接近。把分數並排來看,一張表就能說明本書的脈絡。

模型 章 驗證名稱 測試名稱
單一詞元的計數,即 1-gram 3 687.2
詞元配對的計數,即 2-gram 3 46.9
四個詞元的計數,即 4-gram 基準 3 23.1 22.9
視窗模型,一個小型神經網路 5 36.4 35.4
視窗模型與計數混合 5 21.9 21.6
商店的 Transformer 8 24.0 23.7
商店的 Transformer 與計數混合 8 18.4 18.3

其中貫穿三個教訓。第一,對於簡短且格式固定的文字,計數法能走得很遠:僅就名稱而言,本書中沒有任何學習式模型勝過 4-gram 基準,而兩種混合模型都勝過它。第二,學習式模型之所以有價值,在於它出錯的地方與計數法不同,這也是每個混合模型都勝過其兩個組成部分的原因。第三,在較長、遠距離上下文有影響的文字上,Transformer 單獨就能領先,正如第 8 章在商品描述上所測得的結果。規模本身並未決定任何事:第 10 章的通用模型大了數百倍,在商店名稱上每個詞的得分為 54,352.8,而商店的 Transformer 為 199.3。

這些數字都無法說明模型是否有用。困惑度衡量的是模型預測與其訓練文字相似之文字的能力,這正是自動完成所需要的。它並不衡量描述是否屬實、回答是否切合請求,也不衡量購物者是否找到他們要找的東西。本章的測試,包括虛構素材、信心程度,以及事實在上下文中的位置,是跨出困惑度的第一步,朝向直接衡量工作本身。

實用原則: 比較模型時,應維持一個固定的評量標準,每次都使用相同的文字、分詞器與衡量方式,如同本書所做的。接著,在選定模型來執行工作之前,先用為該工作設計的測試衡量工作本身:較低的困惑度是測試模型的理由,而不是讓模型上線的理由。

正式環境中會出什麼問題

經過調校的模型是人們不需協助就能使用的那一類,而它的失誤聽起來與成功一樣流暢:

  • 它在應該說「不知道」時仍然作答。 對於資料中沒有指明材質的多數產品,基礎版與調校後的 SmolLM2 模型都捏造了材質,而且兩者都不曾說資料沒有提到。在答案送到購物者面前之前,請對照產品資料檢查每一項事實性陳述。
  • 它的信心值會讓錯誤漏網。 捏造的答案平均而言信心較低,但一個能保留大部分正確答案的門檻,仍然放過了 3 of 11 個捏造的答案。
  • 它的格式是訓練的一部分。 少了聊天範本,調校後的模型只寫出一個句號。請套用模型檔案中的範本,並隨每個答案記錄完整的提示詞。
  • 上下文的中段會被遺漏。 請把重要的細節放在開頭或結尾,並在各個位置測試檢索結果。
  • 它不認識你的文字。 指令調校教的是答案的形式,而不是商店的商品目錄。請以商品目錄進行微調,並加以量測。
  • 流暢會掩蓋缺漏。 調校後的模型所寫的描述讀起來比基礎模型更好,但包含的事實並不更多。評斷生成文字時,請依據對其陳述所做的檢查,而不是它讀起來如何。
  • 它的行為取決於它的修訂版本,以及其範本的預設訊息。 請如錄製內容那樣固定模型的修訂版本,並在任何更新之後重新執行這些量測。

本章的決定

  • 本書在相同的請求上比較基礎模型與微調後的模型,請求只在固定的修訂版本下記錄一次,並以貪婪解碼產生答案,再用任何人都能重跑的檢查來衡量它們的回答,例如針對材質的 lib.claims。
  • 捏造細節不只在輸入能回答的問題上衡量,也在輸入無法回答的問題上衡量。 在商店的商品上,兩個 SmolLM2 模型對於資料中沒有提到材質的多數商品,都捏造了材質,而且沒有一次拒答,因此產生的商品事實一律要對照該商品的資料來檢查。
  • 模型的信心是一種訊號,而不是過濾器。 只有在使用以商店自己的問題量測出的門檻時才採用,而且它放行的錯誤也要計數。
  • 模型需要的細節要放在上下文的開頭或結尾,絕不能埋在其他商品之間。
  • 困惑度仍是本書比較模型的標尺,每次都使用相同的文字、分詞器與事件,而每個模型所選定的任務,則用為該任務設計的測試來衡量。
  • 要讓通用模型學會商店的文字,就是微調,並以逐字方式對照商店自己的模型來衡量;指令微調改變的是模型如何回答,而不是它知道什麼。若要在應用程式內衡量模型,超出本書下一個詞元測試的範圍,那是另一個獨立的主題。

本章,於本版次中

版次 2026.11.16範例於 2026年11 月 19 通過測試本章 9 個範例 個範例中有 9 個通過

譯自英文書,版次 2026.11.16。

引用本節

格式

各種格式皆引用英文原書,本譯本即依循該原書。

每種格式都會標示本版。其連結會開啟目前版次,節標題會說明該節內文最後一次變動的時間。存取日期與擷取日期即為你開啟此頁的當日。

本頁內容

第 11 章,共 11 章: From predicting to following instructions

  1. Build the two training losses by hand
  2. 詢問基礎模型與調校後的模型
  3. Format the request as the model was tuned
  4. Measure made-up product details
  5. 檢查信心是否能發出警訊
  6. 找出較長的上下文會遺失什麼
  7. Adapt a model to your text
  8. 回顧評量標準
  9. 正式環境中會出什麼問題
  10. 本章的決定

返回頂端所有章節