从预测到遵循指令
阅读约需 42分钟 分钟,10节中含9个示例。
学完本章,你能够解释一个预测下一个词元的模型如何变成遵循 requests 的模型,构建完成这一转变的两个训练损失,并衡量这一转变解决了什么、没有解决什么。你将在相同的 requests 上比较基础模型与其指令微调版本,衡量二者各自编造商品细节的频率,检验模型编造时它的置信度是否会向你发出警告,并找出在较长的上下文中,模型从哪个位置起不再能找到所给的内容。
这家家具与家居用品商店的商品名称和描述,是本书建模的文本,而本书到目前为止的每个模型只做一件事:预测与其训练文本相似的文本的下一个词元。第 10 章读取了 SmolLM2-360M,这是一个在通用文本上以这种方式训练的开源模型,并发现它对商店名称的评分,按每个词计算,远差于商店自己的小模型,对描述的评分也更差。人们想要的不是自己文本的续写,而是对自己请求的回答。模型通过另外两个训练步骤学会这一点。
指令微调,也称监督微调,是在由请求与优质回答配成的示例上继续训练模型,使得预测这样一对文本的下一个词元就意味着回答请求。Wei 等人发现,在大量以指令形式表述的任务上进行微调,会使模型在从未见过的任务上表现更好(Wei et al., 2022)。偏好微调随后训练模型,使其在针对同一请求的两个回答中倾向于更好的那一个。Ouyang 等人让人对回答排序,并用强化学习训练,所用的奖励模型是从这些排序中学得的(Ouyang et al., 2022);直接偏好优化(Direct Preference Optimization,DPO)无需奖励模型即可完成偏好微调,只需在偏好对本身上使用一个简单的损失(Rafailov et al., 2023)。
SmolLM2-360M-Instruct 是第 10 章的基础模型经过这两个步骤之后的结果:先在公开和精选的数据集上做监督微调,再在 UltraFeedback 上做 DPO(模型卡)。UltraFeedback 的偏好对由一个大语言模型打分,而不是由人打分(Cui et al., 2023)。本章称 SmolLM2-360M 为基础模型,称 SmolLM2-360M-Instruct 为微调后的模型。脚本 recordings/instruction_tuning.py 在固定的修订版本上运行了两个模型各一次,采用贪心解码,因此每个回答都是模型最可能给出的回答,运行可以精确重复,并保存了它们写出的内容,以及它们对所写的每个词元给出的概率。对这两个模型的所有度量都来自这些文件。
这部分尚未翻译,因此以英文显示。
Build the two training losses by hand
Both steps of tuning are training, with the loop of chapter 8; what changes is the loss. Instruction tuning uses the ordinary next-token loss with one difference: only the response's tokens count. The request is read, as context, but the model isn't trained to predict it, since the point is to answer requests, not to write them.
Lay one product out as a request and a response, its name as the request and its description as the response, the way the store's model already reads it. Then compute that model's loss over every token, and over the response's tokens only:
1"""Computes the store's model's loss on one product laid out as a request and a response, over every token and over2the response's tokens only. The prose's point rests on the last printed line, the average loss over the response only.3"""4import numpy as np56from lib import transformer7from lib.store import load_products89_, vocabulary, manifest = transformer.load()10index = {token: i for i, token in enumerate(vocabulary)}11# the name is unique in the validation split, so its one row is the product12product = load_products("validation").query("name == 'double chaise patio furniture cover'").iloc[0]13# the text is cut at the model's longest length, so a long description may end early14ids = transformer.product_ids(product["name"], product["description"], index, manifest["settings"]["longest"])15sep = ids.index(index[transformer.SEP]) # the name plays the request; the description, the response1617# the last position predicts nothing, since no token follows it, so its row is dropped18probabilities = transformer.softmax(transformer.forward(ids)[:-1])19# position i predicts the token at i + 1, so ids[1:] holds each position's target20# each token's loss, given the tokens before it21loss = -np.log(probabilities[np.arange(len(ids) - 1), ids[1:]])22response = np.arange(1, len(ids)) > sep # the description's tokens and its end23print("tokens predicted in the text:", len(loss))24print("tokens the instruction loss counts, the response's:", int(response.sum()))25print("average loss over every token:", f"{loss.mean():.3f}")26print("average loss over the response only:", f"{loss[response].mean():.3f}")
python instruction-loss-by-hand.pytokens predicted in the text: 96
tokens the instruction loss counts, the response's: 90
average loss over every token: 2.267
average loss over the response only: 2.132
Training on the loss over the response alone, across many requests and responses, teaches a model to produce responses. For this product that loss is 2.132, against 2.267 over every token. The instruction loss counts 90 of the 96 tokens the model predicts: the description's tokens and its end.
DPO's loss works on a pair of responses to one request, one preferred and one rejected. DPO compares how much more likely the model being tuned makes each response than the reference model it started from, the model after instruction tuning, and rewards it for raising the preferred one more. With log-probabilities log π of each response under the tuned model and log π_ref under the reference, and a factor β that sets how far the model may move:
margin = β × [(log π(preferred) − log π_ref(preferred)) − (log π(rejected) − log π_ref(rejected))]
loss = −log σ(margin)
with σ the logistic function from chapter 4 (Rafailov et al., 2023, equation 7). Made-up log-probabilities keep the arithmetic in plain view, so this example uses them, with β = 0.1:
1"""Computes the DPO margin and loss by hand for two made-up cases, with a fixed beta. The prose's point rests on the2two loss lines, one for each case.3"""4import numpy as np56beta = 0.1 # how far the tuned model may move from the reference7print("beta:", beta)8# Each case: (preferred, rejected) under the tuned model, then (preferred, rejected) under the reference model it9# started from. Made-up numbers, to follow the arithmetic.10cases = {"tuned model favours the preferred answer more than the reference does": (-12.0, -20.0, -14.0, -18.0),11 "tuned model favours it less": (-15.0, -15.0, -14.0, -18.0)}12for label, (preferred, rejected, preferred_reference, rejected_reference) in cases.items():13 margin = beta * ((preferred - preferred_reference) - (rejected - rejected_reference))14 loss = np.log1p(np.exp(-margin)) # -log σ(margin), written so it can't overflow15 print(f"{label}, margin:", f"{margin:.2f}")16 print(f"{label}, loss:", f"{loss:.3f}")
python dpo-loss-by-hand.pybeta: 0.1
tuned model favours the preferred answer more than the reference does, margin: 0.40
tuned model favours the preferred answer more than the reference does, loss: 0.513
tuned model favours it less, margin: -0.40
tuned model favours it less, loss: 0.913
Lowering the loss pushes the preferred response's probability up and the rejected one's down, measured against where the model started. When the tuned model favours the preferred response more than the reference does, the margin is 0.40 and the loss 0.513; when it favours it less, the margin is -0.40 and the loss rises to 0.913.
Working rule: Mask the request out of the loss when you tune a model on requests and responses, and keep the reference model's log-probabilities beside the tuned model's when you train on preferences. Both are small changes to a loop you already have.
向基础模型和调优模型提问
商店需要为没有描述的商品生成描述,第 9 章曾把这项工作交给它自己的小模型。基础模型 SmolLM2-360M 不能被“提问”,只能被“续写”,所以给它的是商品清单的开头:Product:、商品名称,以及 Description:。调优模型 SmolLM2-360M-Instruct 得到的是一个请求:Write a short product description for:加上商品名称。
两个模型都从第 9 章的商品 front porch welcome sign wall décor 开始。先看各自写了什么,再统计记录中描述过的全部 8 件商品,每件商品都没有可参考的描述:
1"""Prints what the base and the tuned model wrote for one product with no description, then measures every such product's2text for materials its name doesn't give and for repeated four-word phrases. The prose's points rest on the last four3printed lines, the two models' invented-material counts and repeat shares.4"""5from collections import Counter67from lib import claims, recorded89# the recording's saved texts for every product, read back rather than generated again10described = recorded.read("instruction-tuning", "descriptions")11product = next(row for row in described if row["name"] == "front porch welcome sign wall décor")12print("product:", product["name"])13print("base model:", " ".join(product["base"]["text"].split()))14print("tuned model:", " ".join(product["tuned"]["text"].split()))151617def repeats(text):18 """The share of four-word phrases that repeat an earlier one in the text."""19 words = text.lower().split()20 # every window of four consecutive words, so a text of n words has n - 3 of them21 phrases = [tuple(words[i:i + 4]) for i in range(len(words) - 3)]22 # a phrase counts as repeated each time after its first appearance; max keeps a text of fewer than four words from dividing by zero23 return sum(count - 1 for count in Counter(phrases).values()) / max(len(phrases), 1)242526print("products described, none of which has a description to draw on:", len(described))27for role in ["base", "tuned"]:28 texts = [row[role]["text"] for row in described]29 # materials the text names that the product's name doesn't give: a set difference, not the text's whole list30 invented = sum(bool(claims.materials(row[role]["text"]) - claims.materials(row["name"])) for row in described)31 print(f"{role} model, descriptions that name a material the name doesn't:", f"{invented} of {len(described)}")32 print(f"{role} model, repeated four-word phrases:", f"{sum(map(repeats, texts)) / len(texts):.0%}")
python base-and-tuned.pyproduct: front porch welcome sign wall décor
base model: Front porch welcome sign wall décor The front porch is a common feature of many homes. It is a place where people can relax and enjoy the outdoors. It is also a place where people can socialize and meet new friends. The front porch is a great place to hang out with your family
tuned model: "Welcome to our front porch welcome sign wall décor, a space that brings warmth and comfort to your home. This wall is a statement piece that showcases your personality and style. With a mix of rustic and modern elements, you can create a unique and inviting atmosphere that reflects your individuality.
products described, none of which has a description to draw on: 8
base model, descriptions that name a material the name doesn't: 2 of 8
base model, repeated four-word phrases: 14%
tuned model, descriptions that name a material the name doesn't: 1 of 8
tuned model, repeated four-word phrases: 0%
基础模型写的是前廊,The front porch is a common feature of many homes,随后写到放松和会见朋友。它朝一个合理的方向续写文本,但这并不是商店需要的方向。调优模型写出了具有商品描述形态的文字,a statement piece that showcases your personality and style,却把这块牌子称为 a space 和 This wall:对一件自己并不理解的商品,写得很流畅。就所有商品而言,基础模型的文本中有 14% 的四词短语重复出现,调优模型则为 0%。记录无法说明是调优的哪一步消除了这些循环。
除了商品名称,两个模型对商品一无所知。基础模型在 2 of 8 条描述中写出了名称里没有的材质,调优模型则在 1 of 8 条中如此,这些材质都来自模型本身,而不是商店的数据,与第 9 章的情形相同。
**工作准则:**需要让模型照请求行事时,使用调优模型;需要按前文的风格续写文本时,使用基础模型。不要把流畅当作知识:在同样缺少事实的情况下,调优模型只是写出了形态更好的描述。
按模型调优时的格式组织请求(可保留;或改为“按模型调优时所用的格式来组织请求”)
在上一节中,调优后的模型之所以能够作答,是因为它收到的请求采用了它在调优时学到的版式。调优后模型的聊天模板(chat template)就是它学到的版式:由若干特殊标记构成,用来标明系统消息(system message,即放在用户请求之前的指令)、用户请求本身以及模型回复各自从哪里开始、到哪里结束。模型只学会了在这种版式下回答请求,因此任何其他形式的请求对它来说都只是需要续写的文本。加载模型的库会根据模型的分词器文件(分词器配置)应用该模板。
下面给出调优后模型的完整提示,以及它对同一请求(以纯文本形式给出)的回复,以便对照它实际读到的内容:
1"""Prints the full prompt the tuned model read, with its chat template, and its reply with and without the template. The2prose's point rests on the last two printed lines: the same words, with and without the template.3"""4from lib import recorded56template = recorded.read("instruction-tuning", "template")7# the request is the text between the user marker and its end marker8print("request:", template["with"]["prompt"].split("<|im_start|>user\n")[1].split("<|im_end|>")[0])9print("what the tuned model reads, with its chat template:")10print(template["with"]["prompt"])11print("tuned model, with the template:", " ".join(template["with"]["text"].split()))12print("tuned model, without it:", " ".join(template["without"]["text"].split()))
python chat-template.pyrequest: Write a short product description for: front porch welcome sign wall décor
what the tuned model reads, with its chat template:
<|im_start|>system
You are a helpful AI assistant named SmolLM, trained by Hugging Face<|im_end|>
<|im_start|>user
Write a short product description for: front porch welcome sign wall décor<|im_end|>
<|im_start|>assistant
tuned model, with the template: "Welcome to our front porch welcome sign wall décor, a space that brings warmth and comfort to your home. This wall is a statement piece that showcases your personality and style. With a mix of rustic and modern elements, you can create a unique and inviting atmosphere that reflects your individuality.
tuned model, without it: .
调优教会模型的是按一种版式遵循请求,而不是遵循一般意义上的请求。使用模板时,请求位于 <|im_start|>user 与 <|im_end|> 之间,前面是模板默认添加的系统消息 You are a helpful AI assistant named SmolLM, trained by Hugging Face,后面是 <|im_start|>assistant,模型的回复从这里开始。模型给出的回复就是上一节中的那段描述。不使用模板、把同样的文字作为纯文本给出时,调优后的模型写出 . 便停止了。
系统消息同样重要:它是模型读到的每一次请求的一部分,因此换一条不同的系统消息(例如对商店及其语气的描述)就可能改变任何一次回复。这使它成为一项需要选定并记录的设置,如同第 9 章中的解码设置。
工作准则: 向调优后的模型发送请求时,始终使用它自带的聊天模板,由库根据模型的分词器文件应用,切勿手工拼装。检查模板的默认系统消息,因为模型在每次请求时都会读到它;并且为每个保留下来的回答,记录完整的格式化提示。
衡量编造的商品细节
模型在没有任何事实的情况下撰写商品内容,只能自行编造。更公平的测试是把事实提供给它。两个 SmolLM2 模型都会看到商品的名称和真实描述,并被问到:What material is this product made of? Answer in a few words, using only the description. 调优后的模型以模板中的请求形式收到这个问题;基础模型收到的是一份待续写的清单,结尾依次是 Question:、问题和 Answer:。如果商品数据中写明了材质,好的回答应当说出该材质;如果没有写明,好的回答应当说明描述中没有提到。
检查回答需要一份材质词表,从 wood、metal 到 polyester、leather,放在本书的共享模块 lib.claims 中。如果回答中提到的每一种材质都出现在商品的名称或描述中,该回答属于有依据。如果回答提到了商品名称和描述中都不包含的材质,该回答属于编造。如果回答说明商品数据中没有写明,该回答属于拒答;如果回答既没有提到任何材质,也没有拒答,则属于无材质。这一检查有其局限,并且需要做出判断:它把 mdf 和 rubberwood 这类工程木材算作木材,把 brushed nickel finish 算作提到了镍,对词表之外的材质则会漏检。它归类的每一条回答都有记录,任何人都可以查阅。编造输入所不支持的细节,是所谓幻觉(hallucination)的一种(Ji et al., 2023),本测试衡量的正是这一种。
对随机抽取的 40 件商品,为每个模型的回答归类,并按商品数据是否写明材质分组。在数据没有写明材质的商品上,调优后的模型编造的材质会比基础模型少吗?在查看结果之前先做出判断。
1"""Classes both models' answers to the material question, split by whether the product's name or description names a2material. The prose's point rests on the rows for products whose data names none: made up and declined.3"""4from collections import Counter56from lib import claims, recorded78asked = recorded.read("instruction-tuning", "answers")9print("products asked, each with its description given:", len(asked))10# a product's data names a material if its description or its name does: the union of the two sets11given = [bool(claims.materials(row["description"]) | claims.materials(row["name"])) for row in asked]12print("products whose name or description names a material:", sum(given))13print("products whose name and description name none:", len(given) - sum(given))14for role in ["base", "tuned"]:15 for label, keep in [("data names a material", True), ("data names none", False)]:16 rows = [row for row, named in zip(asked, given) if named == keep]17 # each answer is classed against the product's data, by the rules in lib.claims18 counts = Counter(claims.classify(row[role]["text"], row["description"], row["name"]) for row in rows)19 for kind in ["supported", "made up", "none", "declined"]:20 print(f"{role} model, {label}, {kind}:", counts[kind])
python made-up-materials.pyproducts asked, each with its description given: 40
products whose name or description names a material: 28
products whose name and description name none: 12
base model, data names a material, supported: 24
base model, data names a material, made up: 2
base model, data names a material, none: 2
base model, data names a material, declined: 0
base model, data names none, supported: 0
base model, data names none, made up: 7
base model, data names none, none: 5
base model, data names none, declined: 0
tuned model, data names a material, supported: 23
tuned model, data names a material, made up: 3
tuned model, data names a material, none: 2
tuned model, data names a material, declined: 0
tuned model, data names none, supported: 0
tuned model, data names none, made up: 8
tuned model, data names none, none: 4
tuned model, data names none, declined: 0
| 回答 | 基础模型,数据写明材质 | 调优模型,数据写明材质 | 基础模型,数据未写明 | 调优模型,数据未写明 |
|---|---|---|---|---|
| 有依据 | 24 | 23 | 0 | 0 |
| 编造 | 2 | 3 | 7 | 8 |
| 无材质 | 2 | 2 | 5 | 4 |
| 拒答 | 0 | 0 | 0 | 0 |
在名称和描述都没有写明材质的 12 件商品上,正确的回答是描述中没有提到,而两个模型一次也没有这样回答。两者多数时候转而编造了一种材质,其余回答则没有提到任何材质,也没有说明原因。在这些商品中,有 28 件的数据写明了材质,对这些商品,两个模型大多能依据数据作答。
指令调优没有解决这个问题,而在这些商品上,拿到待续写清单的基础模型表现并不更差。请求中要求只使用描述;当描述中没有答案时,模型仍然作答,给出的是这类商品常见的某种材质。遵循指令的形式,并不等于知道自己何时不知道。
工作准则: 测试模型时,不仅要用输入能够回答的问题,也要用输入无法回答的问题,并把编造的回答单独计数。模型对商品提出的每一项事实性断言,都要与商品数据核对,能自动核对的就自动核对,就像 lib.claims 对材质所做的那样。
检查置信度能否发出警示
上一节把微调后模型对材料问题的回答分为有依据的和编造的。如果编造的回答伴随着较低的置信度,那么低置信度就可以用来标记它们。一个模型的置信度与其答对的频率相符,就称该模型是校准的(calibrated)。Guo 等人发现,他们研究的深度神经网络往往并不校准:这些网络的置信度高于其准确率所能支持的程度(Guo et al., 2017)。记录保存了微调后模型为其写出的每个词元(包括结束词元)给出的概率,因此可以为每个回答计算每词元置信度,计算方式为几何平均数:
每词元置信度 = exp((log p₁ + … + log pₙ) / n)
即平均对数概率的指数。困惑度所取倒数的正是这一平均值,所以短回答和长回答可以放在同一尺度上衡量。
置信度能否区分这两类回答?比较微调后模型对有依据回答和编造回答的置信度,并统计以每词元 0.5 为阈值时,两类回答各会保留多少个:
1"""Compares the tuned model's confidence in its supported and made-up answers to the material question, and counts what a2threshold of 0.5 per token keeps. The prose's point rests on the two average confidences and the two threshold counts.3"""4import numpy as np56from lib import claims, recorded78asked = recorded.read("instruction-tuning", "answers")9# only supported and made-up answers are compared; declined and none answers are left out10by_kind = {"supported": [], "made up": []}11for row in asked:12 kind = claims.classify(row["tuned"]["text"], row["description"], row["name"])13 if kind in by_kind:14 # the geometric mean of the token probabilities: exp of their average log-probability15 by_kind[kind].append(np.exp(np.mean(np.log(row["tuned"]["token_probabilities"]))))16for kind, confidence in by_kind.items():17 print(f"tuned model, {kind} answers:", len(confidence))18 print(f"tuned model, {kind} answers, average probability per token:", f"{np.mean(confidence):.2f}")19# a fixed cut for the comparison, not one fitted to these answers20threshold = 0.521for kind, confidence in by_kind.items():22 print(f"tuned model, {kind} answers above {threshold} per token:", f"{sum(c > threshold for c in confidence)} of {len(confidence)}")
python answer-confidence.pytuned model, supported answers: 23
tuned model, supported answers, average probability per token: 0.65
tuned model, made up answers: 11
tuned model, made up answers, average probability per token: 0.45
tuned model, supported answers above 0.5 per token: 21 of 23
tuned model, made up answers above 0.5 per token: 3 of 11
置信度是一种信号,而不是过滤器:它会改变概率的对比,却会放过高置信度的错误。编造的回答平均置信度较低,每词元为 0.45,而有依据的回答为 0.65。但以每词元 0.5 为阈值,会保留 21 of 23 个有依据的回答,同时仍会放过 3 of 11 个编造的回答。由于回答数量很少,这些结果只能说明趋势,不能当作可靠的比率。
**工作准则:**不要只凭模型的置信度来决定是否信任一个回答。要在你自己的问题上衡量其置信度与答对之间的关系;如果使用阈值,就根据这些测量结果来选择,并统计它放过了多少错误。
找出更长的上下文丢失了什么
材料问题每次只给模型一个产品。商店的助手往往要同时面对多个产品的数据,例如搜索结果、分类页面或对比表,并且必须在其他产品之中找到正确产品的详细信息。Liu 等人发现,模型对长上下文开头和结尾的信息利用得比对中间的信息更好(Liu et al., 2024)。
把被询问的产品放在 10 个产品列表之中:第一个、中间或最后一个。然后向调优后的模型 SmolLM2-360M-Instruct 询问该产品由什么材料制成。按 lib.claims 的检查方式,统计只列出该产品自身材料的回答数量,范围是数据中写明了材料的 14 个产品:
1"""Counts the tuned model's answers that name only the asked product's own materials, with the product first, in the middle2or last of a context of listings. The prose's point rests on the three placement lines.3"""4from collections import defaultdict56from lib import claims, recorded78context = recorded.read("instruction-tuning", "context")9# the other listings in each context; the asked product is one more, as the first printed line shows10others = recorded.read("instruction-tuning", "manifest")["settings"]["context_others"]11print("product listings in each context:", others + 1)12results = defaultdict(list)13for row in context:14 # only products whose data names a material: a product with none gives an answer nothing to name15 if claims.materials(row["description"]) | claims.materials(row["name"]):16 # supported: every material the answer names is in the asked product's own data17 results[row["place"]].append(claims.classify(row["tuned"]["text"], row["description"], row["name"]) == "supported")18print("products asked about at each place:", len(results["first"]))19for place in ["first", "middle", "last"]:20 print(f"product placed {place}, answers that name only its own materials:", f"{sum(results[place])} of {len(results[place])}")
python answer-position.pyproduct listings in each context: 10
products asked about at each place: 14
product placed first, answers that name only its own materials: 12 of 14
product placed middle, answers that name only its own materials: 6 of 14
product placed last, answers that name only its own materials: 12 of 14
位于中间的产品最不容易被找到,这与 Liu 等人在大得多的模型中发现的规律一致。产品放在第一个时,有 12 of 14 个回答只列出其自身的材料;放在最后一个时为 12 of 14;放在中间时为 6 of 14。样本数量不大,但差距很明显。这里的上下文很短,只有几千个字符,远未达到模型的上限:上下文远未填满,中间的细节就已经丢失了。
工作准则: 把模型最需要的细节放在上下文的开头或结尾,而不是中间,并在任务允许的范围内尽量少给它无关文本。在相信模型能从众多内容中找到细节之前,先测试它在每个位置上从较长上下文中检索信息的能力。
让模型适应你的文本
第 10 章发现,在商店的商品名称上,按每个词计,通用模型远不如商店自己的小模型。指令微调改变的是模型如何作答,而不是它对某一家商店了解多少。要让通用模型学会商店的文本,办法是微调(fine-tuning):在这些文本上继续训练模型,训练循环与第 8 章相同,只是起点是通用模型的权重,而不是随机权重。
完全微调会更新每一个权重,对于拥有数亿参数的模型,这既耗时又占内存。LoRA(低秩适配,low-rank adaptation)则冻结模型的权重,只在其中一部分权重旁训练一个小的修正项。对于把 k 个数映射为 d 个数的权重矩阵,它训练一个 d×r 的矩阵 B 和一个 r×k 的矩阵 A,并把二者的乘积加到权重上:W + BA(Hu et al., 2022)。秩 r 取得较小时,需要训练和存储的内容少得多,Hu 等人发现,在他们测试的任务上,LoRA 与完全微调的效果相当。
LoRA 修正项要训练的权重少得多;下面在 SmolLM2-360M 上数一数,适配每一层的查询权重和值权重,取两种秩,并与完全微调对比。查询权重把 960 个数映射为 960 个数;由于键和值是分组的,值权重把 960 个数映射为 320 个数:
1"""Counts what LoRA would train on SmolLM2-360M, adapting the query and value weights of every layer at two ranks, and2what full fine-tuning trains. The prose's point rests on the rank 8 lines and the full fine-tuning count.3"""4from lib import open_model56config = open_model.config("SmolLM2-360M")7width, layers = config["hidden_size"], config["num_hidden_layers"]8# keys and values share heads, so their width is the query width scaled by the ratio of key-value heads to heads9key_value_width = config["num_key_value_heads"] * width // config["num_attention_heads"]10# every parameter of the model: the full fine-tuning count, and the denominator of each share11total = open_model.parameters(config)["total"]12for rank in [8, 64]:13 # LoRA adds two small matrices beside each adapted weight: width x rank, then rank x output width.14 added = layers * (rank * (width + width) + rank * (width + key_value_width)) # on the query and value weights15 print(f"rank {rank}, parameters trained:", f"{added:,}")16 print(f"rank {rank}, share of the model's parameters:", f"{added / total:.2%}")17print("parameters trained by full fine-tuning:", f"{total:,}")
python lora-parameters.pyrank 8, parameters trained: 819,200
rank 8, share of the model's parameters: 0.23%
rank 64, parameters trained: 6,553,600
rank 64, share of the model's parameters: 1.81%
parameters trained by full fine-tuning: 361,821,120
在秩为 8 时,LoRA 训练 819,200 个参数,占模型的 0.23%,而完全微调要训练全部 361,821,120 个参数。秩为 8 的修正项比商店自己的小模型还小,而且可以按任务分别存储和替换,基础权重则保持共享。
本书不运行这一微调:我们的判断是,即使是 LoRA,在商店的文本上训练,在笔记本电脑的处理器上也要比本书中任何一次记录都久得多。对于希望通用模型以自己的口吻写作的商店,这是下一步:在商品目录上微调,然后像第 10 章那样,按每个词把结果与商店自己的模型对比。微调本身也有风险。在商品目录上训练的模型,会连同文本一起学到目录的缺漏,就像第 9 章中商店的小模型学会了把灯泡的描述留空。而且在某一段狭窄的文本上训练,可能削弱模型原有的知识;Biderman 等人发现,LoRA 遗忘的内容比完全微调少(Biderman et al., 2024)。这两点的衡量方式与本书中其他一切相同:在留出文本上,比较训练前后。
**工作准则:**要让通用模型学会你的文本,就在该文本上对它进行微调,内存或存储紧张时使用 LoRA,并在同类的留出文本上衡量结果。不要指望指令微调或更巧妙的提示词能提供模型从未学到的知识。
回看衡量标准
本书从第 3 章起,一直用同一种方式给名称上的每个模型打分:以商店分词器的词元为单位计算困惑度,使用的是商店的验证名称。测试名称被单独留出,这样,依据验证名称所做的选择,例如保留哪些轮次、取多大的混合比例,就不会让结果显得更好。最后的这次比较,在 4,281 测试名称上给前面比较过的五个模型打分,其中的混合比例由验证集选定:
1"""Scores the five models the book chose between on the test names, each name's tokens and its end, and prints each2model's test perplexity. The prose's point rests on the five test perplexity lines.3"""4import numpy as np56from lib import neural, transformer7from lib.names import name_tokens8from lib.ngrams import KneserNey, ngrams, padded910# The test names, untouched until now: every model scores each name's tokens and its end. The mixing shares are the11# ones chosen on the validation names, in chapters 5 and 8.12test = name_tokens("test")13counted = KneserNey(name_tokens("train"), 4)14# padded adds the start marker three times and the end marker, so each name's first tokens have a context and its end is scored15from_counts = np.array([counted.probability(c, t) for text in test for c, t in ngrams(padded(text, 4), 4)])16from_window = neural.token_probabilities(test)17_, vocabulary, _ = transformer.load()18index = {token: i for i, token in enumerate(vocabulary)}19from_transformer = []20for name in test:21 # the transformer reads the start marker, the name, then <sep>, which ends a name where the counts end with </s>22 ids = [index[transformer.START]] + [index[t] for t in name] + [index[transformer.SEP]]23 probabilities = transformer.softmax(transformer.forward(ids)[:-1])24 # position i predicts token i + 1, so ids[1:] is each token's target, the end of the name included25 from_transformer += list(probabilities[np.arange(len(ids) - 1), ids[1:]])26from_transformer = np.array(from_transformer)2728# pooled over every token of every name, not averaged per name29perplexity = lambda p: f"{np.exp(-np.log(p).mean()):.1f}"30print("test names:", f"{len(test):,}")31print("test perplexity, the 4-gram baseline:", perplexity(from_counts))32print("test perplexity, the window model:", perplexity(from_window))33# the mix averages the probabilities, not their logs: a share of 0.2 is 20% of the window model34print("test perplexity, the window model mixed with the counts, share 0.2:", perplexity(0.2 * from_window + 0.8 * from_counts))35print("test perplexity, the store's transformer:", perplexity(from_transformer))36print("test perplexity, the store's transformer mixed with the counts, share 0.5:", perplexity((from_transformer + from_counts) / 2))
python test-names.pytest names: 4,281
test perplexity, the 4-gram baseline: 22.9
test perplexity, the window model: 35.4
test perplexity, the window model mixed with the counts, share 0.2: 21.6
test perplexity, the store's transformer: 23.7
test perplexity, the store's transformer mixed with the counts, share 0.5: 18.3
依据验证集所做的选择,对结果的美化微乎其微:测试列的数值与验证列很接近。把这些分数并排放在一起,一张表就讲完了本书的故事。
其中贯穿着三点结论。在简短而程式化的文本上,计数的作用相当大:仅就名称而言,本书中没有任何一个学习得到的模型胜过 4 元基线,而两种混合模型都胜过了它。学习得到的模型之所以有价值,在于它出错的地方与计数不同,这也是每个混合模型都胜过其两个组成部分的原因。而在较长的文本上,远处的上下文很重要,Transformer 凭自身就取得了领先,第 8 章在描述文本上对此做了测量。规模本身并不能决定什么:第 10 章中的通用模型比它大数百倍,在商店名称上每个词的困惑度为 54,352.8,而商店的 Transformer 为 199.3。
这些数字都不能说明一个模型是否有用。困惑度衡量的是模型预测与其训练文本相似的文本的能力,这正是自动补全所需要的。它不衡量一段描述是否真实,不衡量回答是否切合请求,也不衡量顾客是否找到了他们想要的东西。本章的测试,即虚构材料、置信度,以及事实在上下文中的位置,是超越困惑度的第一步,目标是衡量任务本身。
**工作准则:**比较模型时使用一个固定的衡量标准,每次都用相同的文本、分词器和度量,就像本书所做的那样。然后,在选定模型来承担任务之前,用为该任务专门设计的测试来衡量任务本身:更低的困惑度是测试一个模型的理由,而不是部署它的理由。
生产环境中会出什么问题
经过调优的模型是人们无需帮助就能使用的那一类,它的失误听起来与它的成功一样流畅:
- 它在应当说不知道时仍然作答。 对于数据中没有指明材质的大多数商品,基础版和调优版 SmolLM2 模型都编造了一种材质,而且两者都从未说过数据中没有写明。在任何事实性陈述送到购物者面前之前,都应对照商品数据逐条核对。
- 它的置信度会放过错误。 编造的答案平均而言置信度较低,但一个能保留大多数正确答案的阈值,仍然放过了 3 of 11 个编造的答案。
- 它的格式是训练的一部分。 没有聊天模板时,调优后的模型只写出了一个句号。应使用模型文件中的模板,并将完整的提示词与每个答案一同记录下来。
- 上下文的中间部分会丢失。 把重要的细节放在开头或结尾,并在每个位置上测试检索效果。
- 它并不了解你的文本。 指令调优教给模型的是答案的形式,而不是某家商店的商品目录。应在商品目录上进行微调,并加以测量。
- 流畅会掩盖缺漏。 调优后的模型所写的描述读起来比基础模型的更好,但包含的事实并不更多。判断生成的文本,应依据对其陈述的核查,而不是它读起来如何。
- 它的行为取决于其修订版本和模板的默认消息。 如录制内容所示,应固定模型的修订版本,并在任何更新之后重新运行这些测量。
本章的结论
- 本书在相同的请求上比较基础模型与微调后的模型,这些请求在固定的版本下记录一次,并采用贪心解码;本书用任何人都能重新运行的检查来衡量它们的回答,例如针对材质的
lib.claims。 - 编造细节的情况,既在输入能够回答的问题上衡量,也在输入无法回答的问题上衡量。 在商店的商品上,两个 SmolLM2 模型对于数据中没有写明材质的大多数商品,都编造出了一种材质,而且一次也没有拒绝回答,因此生成的商品事实一律要对照该商品的数据进行检查。
- 模型的置信度是一种信号,而不是过滤器。 只有配合在商店自己的问题上测得的阈值才能使用它,并且要统计它放过的错误。
- 模型所需的细节应放在上下文的开头或结尾, 绝不能埋在其他商品之间。
- 困惑度仍是本书用来比较模型的标尺, 每次都使用相同的文本、分词器和事件;而为某项任务所选的每个模型,则用专为该任务设计的测试来衡量。
- 要让通用模型学会商店的文本,就需要微调, 并按每个词与商店自己的模型作比较来衡量;指令微调改变的是模型如何回答,而不是它知道什么。在应用内部衡量模型,超出了本书中下一个词元测试的范围,那是另一个独立的主题。
本章,在本版次中
版次 2026.11.16示例于 2026年11月19日 通过本章 9个示例 个示例中有 9 个通过
译自英文版图书,版次 2026.11.16。