用 LoRA 微调分类头
阅读约需 36分钟 分钟,6节中含5个示例。
学完本章,你将能够用 LoRA 在带标签的消息上训练一个分类头,统计训练所改变的参数数量,按明确规定的规则在开发集上选定移交阈值,并把该阈值与它所产生的结果记录在一起。
到目前为止,路由器运行的是替身模型,它以文本形式写出答案,代替托管模型;它使用的是一个占位阈值 0.5,这个阈值无人选定,每条输出也都标明它只是占位值。答案必须是一个团队(路由结果),阈值也必须经过选定,因此本章用一个分类头取代这种答案:它在一次前向传播中同时返回团队和置信度,并且在报告任何测试结果之前,先在开发集上选定阈值。训练需要路由器各团队对应的带标签消息,这些消息来自公开数据集 BANKING77。第 3 章选定了 SmolLM2-135M-Instruct 作为基础模型,本章只微调这一个模型,不微调其他模型:正如第 3 章所说,360M 模型仍然只作为对照。路由器的评测使用第 2 章确定的样本。
本章不将适配器与 360M 模型比较,也不做任何蒸馏或量化。第 6、7、9 章分别对适配器进行蒸馏、量化和版本管理。
为各团队打分的分类头
替身模型把答案写成文本,解码时由一份语法把文本限定在 77 个团队名称之内(第 4 章)。分类头的做法与替身不同:它保留语言模型的各层,另外加上一个有 77 个输出的输出层,每个输出对应一个团队。对每条消息,模型产生 77 个分数。softmax 把这些分数转换为 77 个总和为一的概率。路由器返回概率最高的团队,置信度就是该概率。
答案是 77 个团队中的一个索引,所以它必定是其中之一。第 4 章通过约束文本,为带提示词的模型取得了同样的保证。这个分类头从不写出文本,因此没有什么需要约束。
一条消息只需一次前向传播:网络从输入到分数运行一遍,没有逐个写出词元的循环。团队和置信度都来自同一次传播。这对转交环节很重要。把消息转给人工处理的标志,与它所保护的团队由同一组概率算出,因此两者不会在两次调用之间产生偏差。若另用一次打分调用,就需要再检查两者是否一致。
这里的置信度含义比替身的更窄。在第 1 章中,置信度是替身所记录答案具有的概率,它并不是对各团队的 softmax。在本章中,置信度是所选团队的 softmax 概率。这两个数都叫置信度,也都要与阈值比较,但它们不是同一个量,所以为其中一个选定的阈值不能用于另一个。
当两个团队得分相同时,router_lib 的 classify 函数取标签索引较小的一个。这源于 torch 的 argmax,其文档写道:"If there are multiple maximal values then the indices of the first maximal value are returned."(torch 2.15.0+cpu 文档)。固定的平局规则使输出可以复现。
工作准则: 从同一次前向传播中读取团队和置信度,并把概率最高的输出作为团队。两者放在同一个函数中,使转交决定与它所保护的团队来自同一个答案。不要把分类头的置信度与替身的置信度并列,仿佛它们度量的是同一件事。
适配器只训练模型的一小部分
完全微调会改变基础模型中的每一个权重。LoRA 走的是另一条路。提出 LoRA 的论文 Hu et al. (2021) 指出,它冻结预训练权重,并把可训练的低秩矩阵注入 Transformer 的各层,训练得到的 LoRA 矩阵就是适配器。基础模型保持原样,而适配器正是本书要存储、加载和做版本管理的部分。
秩 r 是每个新增矩阵的内部维度,取值保持很小。peft 将新增的更新量按 lora_alpha 除以 r 进行缩放(peft 0.21.3,src/peft/tuners/lora/layer.py,第 281 行:self.scaling[adapter_name] = lora_alpha / r),因此 lora_alpha 决定这一更新量所占的分量。本书使用的秩为 8,lora_alpha 设为 16。新增矩阵放在每个注意力块中名为 q_proj 和 v_proj 的投影层里。论文摘要并未指明这些层,所以这一选择出自本书,并非论文的规定。我们推荐把它作为起点,下文记录的适配器就是在这一任务上对该推荐的检验。其他目标层属于需要实测的选择,而不是关于该方法的既定事实。
分类头同样参与训练。对本书的设置,peft 会这样处理:按序列分类器构建的模型,会把它的 classifier 层和 score 层列入 modules_to_save(peft 0.21.3,src/peft/peft_model.py,第 1830 行:classifier_module_names = ["classifier", "score"]),所以训练时 77 个输出的那一层与适配器一起更新。只数适配器的矩阵,得到的数字会小于代码实际训练的数字,差额就是分类头。下一节的运行从加载后的模型中统计这一划分:适配器中是 460800,分类头中是 44352,二者合计为 505152。
适配器的清单文件记录了基础模型的修订版本,读者可以在评分之前,将该版本与 models.toml 核对。
工作准则: 在同一份配置中设定秩、缩放系数和目标层,并在训练开始前,根据加载后的模型统计可训练参数的数量。打印总数时要把分类头计入。漏掉分类头的统计只反映了训练所改变内容的一部分,也与优化器实际更新的参数对不上。
三十步远远不够
在使用录制结果之前,本书先展示同一方案在短时间运行下的表现。下面的运行在 dev 之外的训练行上进行,共 30 步,批大小为 8,种子为 0,随后对 test_200 打分,即第 2 章抽取的固定样本的前 200 条。这是一次失败的运行,本书也如实将其作为失败展示。短时间运行只用于检查流水线能否从头到尾跑通,在这里它的作用仅限于此。
1"""Train for 30 steps on the recorded recipe, then score test_200 at the placeholder threshold of 0.5.2Read the three step losses against the uniform-guess loss, then the hand-off line at that threshold. The recording runs 1,125 steps."""34import math56import torch78torch.set_num_threads(2)910import router_lib # noqa: E4021112router_lib.setup_runtime()13REPO = "HuggingFaceTB/SmolLM2-135M-Instruct"14STEPS, BATCH, SEED = 30, 8, 01516train, order = router_lib.training_rows(SEED) # the seed sets the shuffled order of the training rows, the one the recording used17tok, model = router_lib.new_classifier(REPO) # SmolLM2 with a 77-way head, LoRA rank 8, lora_alpha 16, on q_proj and v_proj18lora, head = router_lib.parameter_split(model) # LoRA tensors have "lora_" in their names; every other trainable tensor is the head19print(f"trainable parameters: {lora + head}")20print(f"LoRA parameters: {lora}")21print(f"head parameters: {head}")2223losses = router_lib.train_steps(model, tok, train, order, STEPS, BATCH) # AdamW at learning rate 1e-3, router_lib's default24print(f"training steps: {STEPS}")25print(f"batch size: {BATCH}")26print(f"seed: {SEED}")27for step in (10, 20, 30):28 print(f"loss at step {step}: {losses[step - 1]:.2f}") # train_steps returns one loss per step, from step 1 at index 0, so step 10 is index 929print(f"loss of a uniform guess over 77 teams: {math.log(77):.2f}")3031result = router_lib.score_test_sample(model, tok, "test_200", 0.5) # 0.5 is chapter 1's placeholder; this run chooses no threshold32n = result["n"]33print(f"messages on test_200: {n}")34print(f"accuracy on test_200: {result['correct']}/{n} ({result['accuracy']:.1%})")35print(f"handed off at placeholder 0.50: {result['handed_off']}/{n} ({result['handoff_rate']:.1%})")36if result["accuracy_not_handed_off"] is None: # None when no message is kept, so there is no accuracy to print37 print("accuracy on messages not handed off: not defined (0 messages not handed off)")38else:39 m = result["not_handed_off"]40 print(f"accuracy on messages not handed off: {result['correct_not_handed_off']}/{m} ({result['accuracy_not_handed_off']:.1%})")
python lora-thirty-steps.pytrainable parameters: 505152
LoRA parameters: 460800
head parameters: 44352
training steps: 30
batch size: 8
seed: 0
loss at step 10: 4.13
loss at step 20: 6.12
loss at step 30: 4.38
loss of a uniform guess over 77 teams: 4.34
messages on test_200: 200
accuracy on test_200: 5/200 (2.5%)
handed off at placeholder 0.50: 200/200 (100.0%)
accuracy on messages not handed off: not defined (0 messages not handed off)
三十步不足以用来路由。这次运行训练出的分类头在 test_200 上的准确率为 5/200 (2.5%),并且在占位阈值下把每条消息都交给了人工:200/200 (100.0%)。把每条消息都转交人工的路由器,不会把任何消息分给某个团队,因此它预测的标签永远不会到达调用方。由于没有消息被保留,未转交消息上的准确率没有定义,输出中也是这样显示的。
在打印出的三个步数上,损失并没有平稳下降:第 10 步为 4.13,第 20 步为 6.12,第 30 步为 4.38。第 20 步高于第 10 步,第 30 步低于第 20 步,但仍高于第 10 步。30 步中的三个点构不成学习曲线。第 30 步的损失接近均匀猜测的 4.34,所以此时分类头仍接近随机水平。均匀猜测给每个团队相同的概率,即 77 分之一,因此其损失是 77 的自然对数。“录制的适配器”一节所用的录制结果以同一方案运行了多得多的步数,本书报告的准确率即来自该结果。
在这里对 test_200 打分只是报告失败,并不据此选择任何东西:方案在运行之前已经固定,阈值留待之后在 dev 上选择。要判断训练是否有效,应查看损失和 dev 上的各项度量,而不是测试样本,这是第 2 章为每个样本定下的规则。针对测试样本调整方案,会使人没有干净的数字可以报告,样本也就不再是测试。
工作准则: 运行短时间训练,只是为了检查流水线能否从头到尾跑通,不要用它的分类头来路由。评判微调后的分类头,应在足够长的运行达到第 2 章所定的各项度量之后,在 dev 划分上进行。不要把寥寥几个点的损失曲线当作分类头正在学习的证据。
只在训练行上训练,绝不在测试行上训练
训练文件包含带标注的消息,第 2 章已按团队、以固定的种子从中划出开发集。训练使用的是未留作开发集的那些训练行。训练期间从不读取测试文件。三十步并不够中的训练顺序,是由去掉开发集之后剩下的行构建的,并以种子 0 打乱,因此每次运行的顺序都相同。
开发集承担着各项决策。它包含来自训练文件 train.csv 的 1003 条消息。没有任何训练步骤读取这些消息,它们只用于选择阈值,以及检查最后一节中的校准情况。
训练配方写入适配器的清单,因此读者可以看到该适配器是用什么训练出来的:基础模型的固定版本、数据提交、批大小、种子和步数。要重复训练,需要这些值;要检查某个结果是否来自干净的划分,则需要数据提交。
当结果看起来很好时,这条规则最为重要。如果训练运行读取了测试样本,它就见过随后用来给它评分的消息,因此它的测试准确率无法说明任何关于新消息的情况。把开发集和测试集分开,才使下一节中的测试准确率成为公允的报告。
工作准则: 只在开发集之外的训练行上训练。在清单中记录种子、批大小、步数和数据提交。当开发集太小,无法确定阈值时,应从训练文件中抽取更多消息,绝不从测试文件中抽取,正如第 2 章所述。
这部分尚未翻译,因此以英文显示。
Use the recorded adapter
The working router uses banking77-lora-v1: the adapter from the full recorded run, version 1, written by recordings/lora_full.py. That script trained the adapter once, on the recipe in the section on training rows, and wrote its files and a manifest. The manifest holds the base model's revision, the data commit, the threshold, the recipe's hash and the hashes of the output files. Offline, the adapter loads and scores test_200 at the threshold from its manifest, and nothing is trained.
Take the adapter as it was recorded. A different recipe becomes a new run, named v2; the book does not edit the adapter by hand. Keeping the recording fixed is what lets a reader check the numbers in this chapter against the files they downloaded.
1"""Load the recorded adapter banking77-lora-v1 and score test_200 at the threshold in its manifest; no training runs here.2Read the accuracy line with the two hand-off lines printed under it."""34import os56os.environ["HF_HUB_OFFLINE"] = "1"7os.environ["TRANSFORMERS_OFFLINE"] = "1"89import json # noqa: E4021011import torch # noqa: E4021213torch.set_num_threads(2)1415import router_lib # noqa: E4021617router_lib.setup_runtime()18RECORDED = router_lib.HERE / "data" / "recorded" / "lora-full"19manifest = json.loads((RECORDED / "manifest.json").read_text())20threshold = manifest["handoff_threshold"] # the threshold the recording chose on dev; this script chooses none2122tokenizer, model = router_lib.load_classifier(RECORDED / "adapter") # the base model comes from the adapter's config, at its revision in models.toml23test = router_lib.read_rows("test")24sample = router_lib.read_sample("test_200")25golds = [test[i][1] for i in sample]26preds, confs = router_lib.classify(model, tokenizer, [test[i][0] for i in sample]) # a tie goes to the lower label index2728result = router_lib.metrics(preds, golds, confs, threshold) # a message is handed off when its confidence is below the threshold29n = result["n"]30print(f"adapter: {manifest['adapter']}")31print(f"handoff threshold from manifest: {threshold:.2f}")32print(f"messages on test_200: {n}")33print(f"accuracy on test_200: {result['correct']}/{n} ({result['accuracy']:.1%})")34print(f"handed off: {result['handed_off']}/{n} ({result['handoff_rate']:.1%})")35if result["accuracy_not_handed_off"] is None: # None when no message is kept, so there is no accuracy to print36 print("accuracy on messages not handed off: not defined (0 messages not handed off)")37else:38 m = result["not_handed_off"]39 print(40 f"accuracy on messages not handed off: {result['correct_not_handed_off']}/{m} "41 f"({result['accuracy_not_handed_off']:.1%})"42 )
python lora-recorded-adapter.pyadapter: banking77-lora-v1
handoff threshold from manifest: 0.63
messages on test_200: 200
accuracy on test_200: 172/200 (86.0%)
handed off: 33/200 (16.5%)
accuracy on messages not handed off: 156/167 (93.4%)
The recorded adapter is right on 172/200 (86.0%) of the messages, against 9/200 (4.5%) for the prompted 135M model that chapter 4 measured on the same 200. The 30-step run, at 5/200 (2.5%), falls below the prompted model. A handed-off message counts as wrong when its team is wrong. The output beside this paragraph gives the hand-off count and the accuracy on messages not handed off, each with its count and denominator, as chapter 2 requires.
The threshold is not chosen on this sample. It came from the manifest, which the section on choosing the threshold explains. It is the threshold the recorded adapter's hand-off counts on test_200 use: 0.63. Reading it from the manifest keeps the choice on dev, where the rule made it.
The recorded adapter is the router's model, and the stand-in is the baseline. The two were measured with different thresholds, so their hand-off counts are not the same kind of result. Compare them on the same messages, with each threshold named beside its count. The comparison below does that for test_200, reading the stand-in's recorded answers for the same messages.
1"""Set the stand-in's recorded answers beside the recorded adapter's live predictions on the same 200 test messages.2The last two lines count where the models disagree."""34import os56os.environ["HF_HUB_OFFLINE"] = "1"7os.environ["TRANSFORMERS_OFFLINE"] = "1"89import json # noqa: E40210import math # noqa: E4021112import torch # noqa: E4021314torch.set_num_threads(2)1516import router_lib # noqa: E4021718router_lib.setup_runtime()19STANDIN_THRESHOLD = 0.5 # chapter 1's placeholder; the stand-in's own threshold is never chosen on dev20ADAPTER = router_lib.HERE / "data" / "recorded" / "lora-full"2122with open(router_lib.HERE / "data" / "recorded" / "standin-baseline" / "answers.jsonl") as f:23 answers = {row["index"]: row for row in map(json.loads, f)} # keyed by the test-row index that read_sample returns24threshold = json.loads((ADAPTER / "manifest.json").read_text())["handoff_threshold"]2526test = router_lib.read_rows("test")27sample = router_lib.read_sample("test_200")28golds = [test[i][1] for i in sample]29standin_preds = [answers[i]["label"] for i in sample]30standin_confs = [math.exp(answers[i]["logprob"]) for i in sample] # the recording stores log-probabilities; exp turns one back into the probability its answer had3132tokenizer, model = router_lib.load_classifier(ADAPTER / "adapter")33adapter_preds, adapter_confs = router_lib.classify(model, tokenizer, [test[i][0] for i in sample])3435standin = router_lib.metrics(standin_preds, golds, standin_confs, STANDIN_THRESHOLD)36adapter = router_lib.metrics(adapter_preds, golds, adapter_confs, threshold)37n = standin["n"]38s_right = [p == g for p, g in zip(standin_preds, golds)]39a_right = [p == g for p, g in zip(adapter_preds, golds)]4041print(f"messages on test_200: {n}")42print(f"stand-in correct: {standin['correct']}/{n} ({standin['accuracy']:.1%})")43print(f"adapter banking77-lora-v1 correct: {adapter['correct']}/{n} ({adapter['accuracy']:.1%})")44print(f"stand-in placeholder threshold: {STANDIN_THRESHOLD:.2f}")45print(f"stand-in handed off: {standin['handed_off']}/{n} ({standin['handoff_rate']:.1%})")46print(f"adapter threshold from manifest: {threshold:.2f}")47print(f"adapter handed off: {adapter['handed_off']}/{n} ({adapter['handoff_rate']:.1%})")48print(f"right for the adapter, wrong for the stand-in: {sum(a and not s for a, s in zip(a_right, s_right))}/{n}")49print(f"right for the stand-in, wrong for the adapter: {sum(s and not a for a, s in zip(a_right, s_right))}/{n}")
python compare-with-standin.pymessages on test_200: 200
stand-in correct: 109/200 (54.5%)
adapter banking77-lora-v1 correct: 172/200 (86.0%)
stand-in placeholder threshold: 0.50
stand-in handed off: 9/200 (4.5%)
adapter threshold from manifest: 0.63
adapter handed off: 33/200 (16.5%)
right for the adapter, wrong for the stand-in: 69/200
right for the stand-in, wrong for the adapter: 6/200
Matching the two models message by message shows which messages each one misses. The stand-in's answers come from its recording, and the adapter runs live on the same 200 messages. The stand-in's confidence is the probability its recorded answer had, from the recording's log-probability. The stand-in's threshold is chapter 1's placeholder, 0.50, and the output labels it as one. Its hand-off count, 9/200 (4.5%), says what that placeholder does to the stand-in, and nothing about the stand-in's own threshold, which was never chosen on dev. The adapter hands off 33/200 (16.5%) at 0.63.
The adapter is right on 172/200 (86.0%) of these messages, the stand-in on 109/200 (54.5%). Where the two disagree, the adapter alone is right on 69/200, and the stand-in alone on 6/200. Two accuracies can be equal while the two models miss different messages, and the disagreement counts show where that happens.
The comparison is one sample of 200 messages. Chapter 1 said that its small sample can show that a router fails, but cannot tell two routers apart with confidence. Chapter 1's sample was smaller. The discordant counts here, 69/200 against 6/200, are stronger evidence than that sample could give. For a firmer claim, run the same comparison on the full test_1001 sample, which the stand-in's recording also covers.
Working rule: Compare two models only on the same messages, and name the threshold each one used beside its count. When one threshold is a placeholder, say so beside its count, and do not read its hand-off count as the model's own result.
在开发集上选择阈值
阈值是把消息转交给人工处理的分界线。第 2 章设定了它的度量方式,第 1 章则把它留作占位符。本书的规则只在开发集(dev)上选择阈值:在步长为 0.01 的网格上,取使未转人工的消息的准确率不低于目标值 0.90 的最低置信度阈值。该规则连同其取值一并写入清单(manifest),因此读者既能看到规则,也能看到它得出的数值。测试样本不参与其中,这是第 2 章的要求。
1"""Find the hand-off threshold on dev by the manifest's rule, then check it against the manifest's value.2Read the chosen threshold beside the three threshold lines. test_200 is not read here."""34import json56import torch78torch.set_num_threads(2)910import router_lib # noqa: E4021112router_lib.setup_runtime()13RECORDED = router_lib.HERE / "data" / "recorded" / "lora-full"14manifest_threshold = json.loads((RECORDED / "manifest.json").read_text())["handoff_threshold"]15TARGET = 0.90 # the book's target accuracy on messages not handed off: a choice the book makes, not one the data sets1617tokenizer, model = router_lib.load_classifier(RECORDED / "adapter")18rows = router_lib.read_rows("train")19dev = router_lib.read_sample("dev")20golds = [rows[i][1] for i in dev]21preds, confs = router_lib.classify_by_length(model, tokenizer, [rows[i][0] for i in dev]) # batches by length to pad less; results come back in dev order22n = len(dev)23chosen = router_lib.choose_threshold(preds, golds, confs, TARGET) # the rule lives in router_lib: the lowest 0.01 grid value that meets TARGET; None if no value meets it, and the prints below would then fail2425print(f"dev messages: {n}")26print(f"target accuracy on messages not handed off: {TARGET:.2f}")27print(f"chosen threshold: {chosen:.2f}")28print(f"threshold in manifest: {manifest_threshold:.2f}")29print(f"chosen threshold matches manifest: {'yes' if chosen == manifest_threshold else 'no'}") # both come from the same 0.01 grid, so equal means the same threshold30for t in sorted({0.50, chosen, 0.90}): # 0.50 is chapter 1's placeholder; a set drops the chosen value if it is one of the other two31 r = router_lib.metrics(preds, golds, confs, t)32 print(f"threshold {t:.2f} handed off: {r['handed_off']}/{n} ({r['handoff_rate']:.1%})")33 print(f"threshold {t:.2f} accuracy on messages not handed off: {r['correct_not_handed_off']}/{r['not_handed_off']} ({r['accuracy_not_handed_off']:.1%})")
python choose-threshold-on-dev.pydev messages: 1003
target accuracy on messages not handed off: 0.90
chosen threshold: 0.63
threshold in manifest: 0.63
chosen threshold matches manifest: yes
threshold 0.50 handed off: 80/1003 (8.0%)
threshold 0.50 accuracy on messages not handed off: 811/923 (87.9%)
threshold 0.63 handed off: 148/1003 (14.8%)
threshold 0.63 accuracy on messages not handed off: 770/855 (90.1%)
threshold 0.90 handed off: 352/1003 (35.1%)
threshold 0.90 accuracy on messages not handed off: 620/651 (95.2%)
用已记录的适配器对开发集打分并应用该规则,得到的阈值为 0.63。与清单中的值核对后输出 yes;若不一致,就说明清单与规则互相矛盾。
三个阈值可以说明其中的取舍,下表把它们并列,并给出各自的数量和比率。提高阈值转人工的消息不会变少,而规则选取的是满足目标的最低值。每一行都是同一个适配器处理同一批开发集消息,因此只有阈值不同。
| 阈值 | 开发集上转人工的消息 | 开发集上未转人工的消息的准确率 |
|---|---|---|
| 0.50 | 80/1003 (8.0%) | 811/923 (87.9%) |
| 0.63,已选定 | 148/1003 (14.8%) | 770/855 (90.1%) |
| 0.90 | 352/1003 (35.1%) | 620/651 (95.2%) |
阈值为 0.50 时,路由器留下的消息更多,但留下的消息准确率较低,未达到目标。阈值为 0.90 时,留下的消息在三者中最准确,但转给人工的开发集消息也最多。选定的值是三个阈值中满足目标的最低者,因此它转人工的消息比 0.50 多,比 0.90 少。该表显示了转人工要付出的代价,以及换来的准确率。目标设在哪里是业务上的选择,规则无法替人作出。本书对留下的消息采用 0.90 的准确率;自行运行路由器的银行,则会根据这一取舍确定自己的目标。
适配器在生产环境中保持不变,但它所见的消息会变。假设某个产品问题使得一周内出现大量关于同一团队的消息。阈值是在开发集的消息构成上选定的,所以即使适配器完全不变,转人工的比率也可能自行变化。转人工增多而适配器没有任何改动,说明应当首先检查消息的构成。若凭目测不断调整阈值,直到转人工的比率看起来正确,就放弃了选择该规则时所依据的性质:路由器留下的开发集消息,其准确率至少达到目标值。
置信度并不是团队判断正确的概率。Guo 等人(2017)指出他们的网络“校准不佳”(poorly calibrated),即网络的置信度与其实际答对的频率不符。本书在开发集上检查自己的适配器。开发集消息按置信度分为五个区间,每个区间给出消息数、准确率和平均置信度,这两个量正是 Guo 等人衡量校准时所比较的。按置信度对开发集分组,可以看出 0.9 是否意味着十次中对九次。
1"""Group the dev messages into five bands by the adapter's confidence, and print each band's count, accuracy and mean confidence.2Read the top band's mean confidence against its accuracy. Only dev is read."""34import os56os.environ["HF_HUB_OFFLINE"] = "1"7os.environ["TRANSFORMERS_OFFLINE"] = "1"89import torch # noqa: E4021011torch.set_num_threads(2)1213import router_lib # noqa: E4021415router_lib.setup_runtime()16RECORDED = router_lib.HERE / "data" / "recorded" / "lora-full"17BANDS = [(0.0, 0.2), (0.2, 0.4), (0.4, 0.6), (0.6, 0.8), (0.8, 1.0)]1819tokenizer, model = router_lib.load_classifier(RECORDED / "adapter")20rows = router_lib.read_rows("train")21dev = router_lib.read_sample("dev")22golds = [rows[i][1] for i in dev]23preds, confs = router_lib.classify_by_length(model, tokenizer, [rows[i][0] for i in dev]) # results come back in dev order, so index i matches golds[i]2425print(f"dev messages: {len(dev)}")26counted = 027for lo, hi in BANDS:28 last = hi == 1.029 members = [i for i, c in enumerate(confs) if lo <= c < hi or (last and c == 1.0)] # lower edges are in, upper edges out; the top band also takes 1.0, so each message is counted once30 counted += len(members)31 name = f"band {lo:.1f}-{hi:.1f}"32 print(f"{name} messages: {len(members)}")33 if members:34 correct = sum(preds[i] == golds[i] for i in members)35 print(f"{name} correct: {correct}/{len(members)} ({correct / len(members):.1%})")36 print(f"{name} mean confidence: {sum(confs[i] for i in members) / len(members):.1%}")37 else:38 print(f"{name} correct: not defined (0 messages)")39 print(f"{name} mean confidence: not defined (0 messages)")40print(f"messages in bands: {counted}")
python calibration-bands-on-dev.pydev messages: 1003
band 0.0-0.2 messages: 1
band 0.0-0.2 correct: 1/1 (100.0%)
band 0.0-0.2 mean confidence: 19.9%
band 0.2-0.4 messages: 41
band 0.2-0.4 correct: 10/41 (24.4%)
band 0.2-0.4 mean confidence: 32.9%
band 0.4-0.6 messages: 92
band 0.4-0.6 correct: 49/92 (53.3%)
band 0.4-0.6 mean confidence: 51.0%
band 0.6-0.8 messages: 123
band 0.6-0.8 correct: 84/123 (68.3%)
band 0.6-0.8 mean confidence: 70.6%
band 0.8-1.0 messages: 746
band 0.8-1.0 correct: 693/746 (92.9%)
band 0.8-1.0 mean confidence: 96.4%
messages in bands: 1003
开发集消息大多落在最高区间,即 0.8 到 1.0,该区间有 746 条。最低区间为 0.0 到 0.2,有 1 条,数量太少,其准确率无法说明低置信度的情形。
在开发集上,置信度对消息的排序是有用的,但在最高区间,置信度高于准确率:最高区间的平均置信度为 96.4%,而准确率为 693/746 (92.9%)。五个区间中有四个的准确率落在其区间范围内,最低区间是唯一的例外,它只有一条消息。从第二个区间往上,准确率逐级上升。在中间两个区间,准确率与平均置信度相近;而在 0.2 到 0.4 的区间,平均置信度明显高于准确率。
这些区间所能显示的仅限于此:0.2 到 0.4 的区间只有 41 条消息。这里的每个数字都来自同一份开发集样本,而这份样本也用于选择阈值,所以这些数字都不能说明适配器在新消息上的校准程度。
规则所选定的路由器,是在开发集上得到的经验分界线,并连同其规则和数据一起记录。它是一条分界线,而不是消息的模型,因此其数字描述的是开发集的消息构成。读者把它的转人工比率和留下的消息的准确率视为路由器的结果,并在消息构成变化时重新检查开发集上的规则。
**工作准则:**按所述规则在开发集上选择阈值,并把阈值及规则写入适配器的清单。报告转人工的比率时,要同时给出留下的消息的准确率,二者各附数量,并至少展示选定值两侧各一个设置。在相信置信度可以作为分界线之前,先在开发集上检查校准;当消息的构成发生变化时,从训练文件中重新抽取开发集样本,再次选择阈值。
本章,在本版次中
版次 2026.11.9示例于 2026年11月20日 通过本章 5个示例 个示例中有 5 个通过
译自英文版图书,版次 2026.11.9。