以 LoRA 微調分類頭
閱讀約需 40 分鐘 分鐘,6 節 中共有 5 個範例。
讀完本章後,你可以用 LoRA 在已標註的訊息上訓練分類頭、計算訓練所改變的參數數量、依明訂的規則在開發集上選定轉交門檻,並把該門檻與它所產生的結果記錄在一起。
到目前為止,路由器執行的是替身模型,它以文字寫出答案,用來取代託管模型,並採用一個無人選定、且每次輸出都標示為暫代值的暫代門檻 0.5。答案必須是一個團隊,門檻也必須經過選定,因此本章以分類頭取代該答案:分類頭在一次前向傳遞中同時回傳團隊與信心值,並在回報任何測試結果之前,先在開發集上選定門檻。訓練需要對應路由器各團隊的已標註訊息,這些訊息來自公開資料集 BANKING77。第 3 章選定 SmolLM2-135M-Instruct 作為基礎模型,本章只微調這個模型,不微調其他模型:如第 3 章所述,360M 仍僅作為比較之用。路由器的評估使用第 2 章所固定的樣本。
本章不將此轉接器與 360M 比較,也不進行蒸餾或量化。第 6、7 與 9 章會對轉接器進行蒸餾、量化與版本管理。
為各團隊評分的分類頭
替身模型以文字寫出答案,並在解碼時由文法把該文字限制在 77 個團隊名稱之內(第 4 章)。分類頭的運作方式不同:它保留語言模型的各層,另外加上一個具有 77 個輸出的輸出層,每個團隊對應一個輸出。對於每則訊息,模型產生 77 個分數。Softmax 將這些分數轉換為 77 個總和為一的機率。路由器回傳機率最高的團隊,而信心值就是該機率。
答案是指向 77 個團隊的索引,因此必然是其中之一。第 4 章透過限制文字,對提示式模型達成相同的保證。這個分類頭從不寫出文字,所以沒有什麼需要限制。
一則訊息只需要一次前向傳遞:網路從輸入到分數執行一次,沒有逐一寫出詞元的迴圈。團隊與信心值都來自同一次傳遞。這對轉交給人工的流程很重要。把訊息送交人工處理的旗標,與它所保護的團隊是由同一組機率算出的,因此兩者不會在兩次呼叫之間產生分歧。若另用一次評分呼叫,就需要額外檢查兩者是否一致。
這裡的信心值,意義比替身模型的更窄。在第 1 章中,信心值是替身模型所記錄之答案具有的機率,它並不是對各團隊做 softmax 的結果。在本章中,信心值是所選團隊的 softmax 機率。這兩個數字都稱為信心值,也都與門檻值比較,但它們並非同一個量,因此為其中一個選定的門檻值,不能拿來當作另一個的門檻值。
當兩個團隊同分時,router_lib 的 classify 函式取較小的標籤索引。這源自 torch 的 argmax,其文件寫道:「If there are multiple maximal values then the indices of the first maximal value are returned.」(若有多個最大值,則回傳第一個最大值的索引。)(torch 2.15.0+cpu 文件)。對同分情形採用固定規則,可使輸出具有可重現性。
實用原則: 從同一次前向傳遞讀取團隊與信心值,並以機率最高的輸出作為團隊。將兩者放在同一個函式中,使轉交給人工的決定與它所保護的團隊來自同一個答案。不要把分類頭的信心值與替身模型的信心值並列比較,彷彿兩者衡量的是同一件事。
此部分尚未翻譯,因此以英文呈現。
Adapters train a small part of the model
Full fine-tuning changes every weight in the base model. LoRA takes a different route. The paper that introduced LoRA, Hu et al. (2021), says it freezes the pre-trained weights and injects trainable low-rank matrices into the layers of the Transformer, and the trained LoRA matrices are the adapter. The base model stays as it was, and the adapter is the part this book stores, loads and versions.
The rank, r, is the inner size of each added matrix, and it is kept small. Peft scales the added update by lora_alpha divided by r (peft 0.21.3, src/peft/tuners/lora/layer.py, line 281: self.scaling[adapter_name] = lora_alpha / r), so lora_alpha sets how strongly the update counts. This book uses a rank of 8, with lora_alpha set to 16. The added matrices go in the projection layers named q_proj and v_proj in each attention block. The abstract does not name those layers, so this choice is the book's, not something the paper prescribes. We recommend it as a starting point, and the recorded adapter below is the test of that recommendation on this task. Other targets are a choice to measure, not a fact about the method.
The head is trained as well. Peft does that for this book's setup: a model built as a sequence classifier lists its classifier and score layers in modules_to_save (peft 0.21.3, src/peft/peft_model.py, line 1830: classifier_module_names = ["classifier", "score"]), so training moves the 77-output layer and the adapter together. Count only the adapter's matrices and you get a smaller number than the one the code trains, and the difference is the head. The next section's run counts the split from the loaded model: 460800 in the adapter and 44352 in the head, which together make 505152.
The adapter's manifest records the base model's revision, so a reader can check that revision against models.toml before scoring.
Working rule: Set the rank, the scaling and the target layers in one configuration, and count the trainable parameters from the loaded model before training starts. Print the total with the head in it. A count that leaves out the head reports part of what training changes, and it will not match what the optimizer updates.
三十步並不足夠
在本書使用錄製結果之前,先展示同一份配方的短程訓練會產生什麼結果。下方的執行共進行 30 步,批次大小為 8,種子為 0,使用 dev 以外的訓練資料列,然後對 test_200 評分,也就是第 2 章抽出的固定樣本中的前 200 筆資料。這是一次失敗的執行,本書也將它當作失敗來呈現。短程執行檢查的是管線能否從頭到尾運作,這是它在此處唯一的用途。
1"""Train for 30 steps on the recorded recipe, then score test_200 at the placeholder threshold of 0.5.2Read the three step losses against the uniform-guess loss, then the hand-off line at that threshold. The recording runs 1,125 steps."""34import math56import torch78torch.set_num_threads(2)910import router_lib # noqa: E4021112router_lib.setup_runtime()13REPO = "HuggingFaceTB/SmolLM2-135M-Instruct"14STEPS, BATCH, SEED = 30, 8, 01516train, order = router_lib.training_rows(SEED) # the seed sets the shuffled order of the training rows, the one the recording used17tok, model = router_lib.new_classifier(REPO) # SmolLM2 with a 77-way head, LoRA rank 8, lora_alpha 16, on q_proj and v_proj18lora, head = router_lib.parameter_split(model) # LoRA tensors have "lora_" in their names; every other trainable tensor is the head19print(f"trainable parameters: {lora + head}")20print(f"LoRA parameters: {lora}")21print(f"head parameters: {head}")2223losses = router_lib.train_steps(model, tok, train, order, STEPS, BATCH) # AdamW at learning rate 1e-3, router_lib's default24print(f"training steps: {STEPS}")25print(f"batch size: {BATCH}")26print(f"seed: {SEED}")27for step in (10, 20, 30):28 print(f"loss at step {step}: {losses[step - 1]:.2f}") # train_steps returns one loss per step, from step 1 at index 0, so step 10 is index 929print(f"loss of a uniform guess over 77 teams: {math.log(77):.2f}")3031result = router_lib.score_test_sample(model, tok, "test_200", 0.5) # 0.5 is chapter 1's placeholder; this run chooses no threshold32n = result["n"]33print(f"messages on test_200: {n}")34print(f"accuracy on test_200: {result['correct']}/{n} ({result['accuracy']:.1%})")35print(f"handed off at placeholder 0.50: {result['handed_off']}/{n} ({result['handoff_rate']:.1%})")36if result["accuracy_not_handed_off"] is None: # None when no message is kept, so there is no accuracy to print37 print("accuracy on messages not handed off: not defined (0 messages not handed off)")38else:39 m = result["not_handed_off"]40 print(f"accuracy on messages not handed off: {result['correct_not_handed_off']}/{m} ({result['accuracy_not_handed_off']:.1%})")
python lora-thirty-steps.pytrainable parameters: 505152
LoRA parameters: 460800
head parameters: 44352
training steps: 30
batch size: 8
seed: 0
loss at step 10: 4.13
loss at step 20: 6.12
loss at step 30: 4.38
loss of a uniform guess over 77 teams: 4.34
messages on test_200: 200
accuracy on test_200: 5/200 (2.5%)
handed off at placeholder 0.50: 200/200 (100.0%)
accuracy on messages not handed off: not defined (0 messages not handed off)
三十步不足以用來分派訊息。這次執行訓練出的分類頭在 test_200 上答對 5/200 (2.5%),而在暫定門檻下,它把每一則訊息都交給人工處理:200/200 (100.0%)。把每一則訊息都轉交出去的分派器,不會把任何訊息指派給任何團隊,因此它預測的標籤永遠不會送到呼叫端。由於沒有任何訊息被保留,未轉交訊息的準確率便無從定義,輸出中也是這樣說明的。
在列印出的三個步驟中,損失並未穩定下降:第 10 步為 4.13,第 20 步為 6.12,第 30 步為 4.38。第 20 步高於第 10 步,第 30 步低於第 20 步,卻仍高於第 10 步。30 步所得的三個點並不構成學習曲線。第 30 步的損失接近均勻猜測的 4.34,因此此時分類頭仍接近隨機水準。均勻猜測會給每個團隊相同的機率,即 77 分之一,所以它的損失是 77 的自然對數。〈錄製的轉接器執行〉一節所用的錄製結果,以同一份配方執行了多得多的步數,本書所報告的準確率即出自該結果。
在此對 test_200 評分只是回報這次失敗,並未據此選擇任何事項:配方在執行前就已固定,門檻則稍後在 dev 上選定。要判斷訓練是否有效,應閱讀損失與 dev 上的各項指標,而不是測試樣本,這是第 2 章為每一份樣本訂下的規則。若針對測試樣本調整配方,就沒有乾淨的數字可供報告,該樣本也不再是測試。
實用原則: 以短程訓練檢查管線能否從頭到尾運作,不要用它的分類頭來分派訊息。評估微調後的分類頭時,應在 dev 切分上,以第 2 章訂定的指標進行,並且要在訓練長到足以達到這些指標之後。不要把只有幾個點的損失曲線當作分類頭正在學習的證據。
只用訓練資料列訓練,絕不使用測試資料列
訓練檔案包含已標註的訊息,而第 2 章已使用固定的種子,依團隊從中劃分出開發集。訓練時使用的是未保留給開發集的訓練資料列。訓練期間絕不讀取測試檔案。三十個步驟並不足夠中的訓練順序,是由移除開發集之後剩餘的資料列建立,並以種子 0 進行洗牌,因此每次執行的結果都相同。
開發集負責支撐各項決策。它包含來自訓練檔案 train.csv 的 1003 則訊息。沒有任何訓練步驟會讀取這些訊息,它們只用於選擇門檻值,以及檢查最後一節中的校準。
訓練配方會寫入轉接器的資訊清單,讓讀者能看出轉接器是以什麼訓練而成:基礎模型的固定修訂版本、資料提交、批次大小、種子與步數。重複訓練需要這些值,而要確認某項結果來自乾淨的劃分,則需要資料提交。
這項規則在結果看起來很好時最為重要。若訓練執行讀取了測試樣本,它就已經看過後續用來評分的訊息,因此其測試準確率無法反映對新訊息的任何表現。將開發集與測試集分開,才能使下一節中的測試準確率成為公平的報告。
實用原則: 只使用開發集以外的訓練資料列進行訓練。在資訊清單中記錄種子、批次大小、步數與資料提交。當開發集太小而無法決定門檻值時,應如第 2 章所述,從訓練檔案中抽取更多訊息,絕不從測試檔案中抽取。
此部分尚未翻譯,因此以英文呈現。
Use the recorded adapter
The working router uses banking77-lora-v1: the adapter from the full recorded run, version 1, written by recordings/lora_full.py. That script trained the adapter once, on the recipe in the section on training rows, and wrote its files and a manifest. The manifest holds the base model's revision, the data commit, the threshold, the recipe's hash and the hashes of the output files. Offline, the adapter loads and scores test_200 at the threshold from its manifest, and nothing is trained.
Take the adapter as it was recorded. A different recipe becomes a new run, named v2; the book does not edit the adapter by hand. Keeping the recording fixed is what lets a reader check the numbers in this chapter against the files they downloaded.
1"""Load the recorded adapter banking77-lora-v1 and score test_200 at the threshold in its manifest; no training runs here.2Read the accuracy line with the two hand-off lines printed under it."""34import os56os.environ["HF_HUB_OFFLINE"] = "1"7os.environ["TRANSFORMERS_OFFLINE"] = "1"89import json # noqa: E4021011import torch # noqa: E4021213torch.set_num_threads(2)1415import router_lib # noqa: E4021617router_lib.setup_runtime()18RECORDED = router_lib.HERE / "data" / "recorded" / "lora-full"19manifest = json.loads((RECORDED / "manifest.json").read_text())20threshold = manifest["handoff_threshold"] # the threshold the recording chose on dev; this script chooses none2122tokenizer, model = router_lib.load_classifier(RECORDED / "adapter") # the base model comes from the adapter's config, at its revision in models.toml23test = router_lib.read_rows("test")24sample = router_lib.read_sample("test_200")25golds = [test[i][1] for i in sample]26preds, confs = router_lib.classify(model, tokenizer, [test[i][0] for i in sample]) # a tie goes to the lower label index2728result = router_lib.metrics(preds, golds, confs, threshold) # a message is handed off when its confidence is below the threshold29n = result["n"]30print(f"adapter: {manifest['adapter']}")31print(f"handoff threshold from manifest: {threshold:.2f}")32print(f"messages on test_200: {n}")33print(f"accuracy on test_200: {result['correct']}/{n} ({result['accuracy']:.1%})")34print(f"handed off: {result['handed_off']}/{n} ({result['handoff_rate']:.1%})")35if result["accuracy_not_handed_off"] is None: # None when no message is kept, so there is no accuracy to print36 print("accuracy on messages not handed off: not defined (0 messages not handed off)")37else:38 m = result["not_handed_off"]39 print(40 f"accuracy on messages not handed off: {result['correct_not_handed_off']}/{m} "41 f"({result['accuracy_not_handed_off']:.1%})"42 )
python lora-recorded-adapter.pyadapter: banking77-lora-v1
handoff threshold from manifest: 0.63
messages on test_200: 200
accuracy on test_200: 172/200 (86.0%)
handed off: 33/200 (16.5%)
accuracy on messages not handed off: 156/167 (93.4%)
The recorded adapter is right on 172/200 (86.0%) of the messages, against 9/200 (4.5%) for the prompted 135M model that chapter 4 measured on the same 200. The 30-step run, at 5/200 (2.5%), falls below the prompted model. A handed-off message counts as wrong when its team is wrong. The output beside this paragraph gives the hand-off count and the accuracy on messages not handed off, each with its count and denominator, as chapter 2 requires.
The threshold is not chosen on this sample. It came from the manifest, which the section on choosing the threshold explains. It is the threshold the recorded adapter's hand-off counts on test_200 use: 0.63. Reading it from the manifest keeps the choice on dev, where the rule made it.
The recorded adapter is the router's model, and the stand-in is the baseline. The two were measured with different thresholds, so their hand-off counts are not the same kind of result. Compare them on the same messages, with each threshold named beside its count. The comparison below does that for test_200, reading the stand-in's recorded answers for the same messages.
1"""Set the stand-in's recorded answers beside the recorded adapter's live predictions on the same 200 test messages.2The last two lines count where the models disagree."""34import os56os.environ["HF_HUB_OFFLINE"] = "1"7os.environ["TRANSFORMERS_OFFLINE"] = "1"89import json # noqa: E40210import math # noqa: E4021112import torch # noqa: E4021314torch.set_num_threads(2)1516import router_lib # noqa: E4021718router_lib.setup_runtime()19STANDIN_THRESHOLD = 0.5 # chapter 1's placeholder; the stand-in's own threshold is never chosen on dev20ADAPTER = router_lib.HERE / "data" / "recorded" / "lora-full"2122with open(router_lib.HERE / "data" / "recorded" / "standin-baseline" / "answers.jsonl") as f:23 answers = {row["index"]: row for row in map(json.loads, f)} # keyed by the test-row index that read_sample returns24threshold = json.loads((ADAPTER / "manifest.json").read_text())["handoff_threshold"]2526test = router_lib.read_rows("test")27sample = router_lib.read_sample("test_200")28golds = [test[i][1] for i in sample]29standin_preds = [answers[i]["label"] for i in sample]30standin_confs = [math.exp(answers[i]["logprob"]) for i in sample] # the recording stores log-probabilities; exp turns one back into the probability its answer had3132tokenizer, model = router_lib.load_classifier(ADAPTER / "adapter")33adapter_preds, adapter_confs = router_lib.classify(model, tokenizer, [test[i][0] for i in sample])3435standin = router_lib.metrics(standin_preds, golds, standin_confs, STANDIN_THRESHOLD)36adapter = router_lib.metrics(adapter_preds, golds, adapter_confs, threshold)37n = standin["n"]38s_right = [p == g for p, g in zip(standin_preds, golds)]39a_right = [p == g for p, g in zip(adapter_preds, golds)]4041print(f"messages on test_200: {n}")42print(f"stand-in correct: {standin['correct']}/{n} ({standin['accuracy']:.1%})")43print(f"adapter banking77-lora-v1 correct: {adapter['correct']}/{n} ({adapter['accuracy']:.1%})")44print(f"stand-in placeholder threshold: {STANDIN_THRESHOLD:.2f}")45print(f"stand-in handed off: {standin['handed_off']}/{n} ({standin['handoff_rate']:.1%})")46print(f"adapter threshold from manifest: {threshold:.2f}")47print(f"adapter handed off: {adapter['handed_off']}/{n} ({adapter['handoff_rate']:.1%})")48print(f"right for the adapter, wrong for the stand-in: {sum(a and not s for a, s in zip(a_right, s_right))}/{n}")49print(f"right for the stand-in, wrong for the adapter: {sum(s and not a for a, s in zip(a_right, s_right))}/{n}")
python compare-with-standin.pymessages on test_200: 200
stand-in correct: 109/200 (54.5%)
adapter banking77-lora-v1 correct: 172/200 (86.0%)
stand-in placeholder threshold: 0.50
stand-in handed off: 9/200 (4.5%)
adapter threshold from manifest: 0.63
adapter handed off: 33/200 (16.5%)
right for the adapter, wrong for the stand-in: 69/200
right for the stand-in, wrong for the adapter: 6/200
Matching the two models message by message shows which messages each one misses. The stand-in's answers come from its recording, and the adapter runs live on the same 200 messages. The stand-in's confidence is the probability its recorded answer had, from the recording's log-probability. The stand-in's threshold is chapter 1's placeholder, 0.50, and the output labels it as one. Its hand-off count, 9/200 (4.5%), says what that placeholder does to the stand-in, and nothing about the stand-in's own threshold, which was never chosen on dev. The adapter hands off 33/200 (16.5%) at 0.63.
The adapter is right on 172/200 (86.0%) of these messages, the stand-in on 109/200 (54.5%). Where the two disagree, the adapter alone is right on 69/200, and the stand-in alone on 6/200. Two accuracies can be equal while the two models miss different messages, and the disagreement counts show where that happens.
The comparison is one sample of 200 messages. Chapter 1 said that its small sample can show that a router fails, but cannot tell two routers apart with confidence. Chapter 1's sample was smaller. The discordant counts here, 69/200 against 6/200, are stronger evidence than that sample could give. For a firmer claim, run the same comparison on the full test_1001 sample, which the stand-in's recording also covers.
Working rule: Compare two models only on the same messages, and name the threshold each one used beside its count. When one threshold is a placeholder, say so beside its count, and do not read its hand-off count as the model's own result.
在 dev 上選擇門檻
門檻是將訊息交給人工處理的分界。第 2 章已訂定其衡量方式,第 1 章則將它留作佔位值。本書的規則只在 dev 切分上選擇門檻:在間距為 0.01 的網格上,取使「未轉交訊息的準確率」至少達到目標 0.90 的最低信心門檻。這項規則連同其值一併寫入 manifest,讀者因此既能看到規則,也能看到它得出的數字。測試樣本不參與其中,這是第 2 章的要求。
1"""Find the hand-off threshold on dev by the manifest's rule, then check it against the manifest's value.2Read the chosen threshold beside the three threshold lines. test_200 is not read here."""34import json56import torch78torch.set_num_threads(2)910import router_lib # noqa: E4021112router_lib.setup_runtime()13RECORDED = router_lib.HERE / "data" / "recorded" / "lora-full"14manifest_threshold = json.loads((RECORDED / "manifest.json").read_text())["handoff_threshold"]15TARGET = 0.90 # the book's target accuracy on messages not handed off: a choice the book makes, not one the data sets1617tokenizer, model = router_lib.load_classifier(RECORDED / "adapter")18rows = router_lib.read_rows("train")19dev = router_lib.read_sample("dev")20golds = [rows[i][1] for i in dev]21preds, confs = router_lib.classify_by_length(model, tokenizer, [rows[i][0] for i in dev]) # batches by length to pad less; results come back in dev order22n = len(dev)23chosen = router_lib.choose_threshold(preds, golds, confs, TARGET) # the rule lives in router_lib: the lowest 0.01 grid value that meets TARGET; None if no value meets it, and the prints below would then fail2425print(f"dev messages: {n}")26print(f"target accuracy on messages not handed off: {TARGET:.2f}")27print(f"chosen threshold: {chosen:.2f}")28print(f"threshold in manifest: {manifest_threshold:.2f}")29print(f"chosen threshold matches manifest: {'yes' if chosen == manifest_threshold else 'no'}") # both come from the same 0.01 grid, so equal means the same threshold30for t in sorted({0.50, chosen, 0.90}): # 0.50 is chapter 1's placeholder; a set drops the chosen value if it is one of the other two31 r = router_lib.metrics(preds, golds, confs, t)32 print(f"threshold {t:.2f} handed off: {r['handed_off']}/{n} ({r['handoff_rate']:.1%})")33 print(f"threshold {t:.2f} accuracy on messages not handed off: {r['correct_not_handed_off']}/{r['not_handed_off']} ({r['accuracy_not_handed_off']:.1%})")
python choose-threshold-on-dev.pydev messages: 1003
target accuracy on messages not handed off: 0.90
chosen threshold: 0.63
threshold in manifest: 0.63
chosen threshold matches manifest: yes
threshold 0.50 handed off: 80/1003 (8.0%)
threshold 0.50 accuracy on messages not handed off: 811/923 (87.9%)
threshold 0.63 handed off: 148/1003 (14.8%)
threshold 0.63 accuracy on messages not handed off: 770/855 (90.1%)
threshold 0.90 handed off: 352/1003 (35.1%)
threshold 0.90 accuracy on messages not handed off: 620/651 (95.2%)
以記錄下來的 adapter 對 dev 評分並套用該規則,得到的門檻為 0.63。與 manifest 中的值比對後,輸出為 yes;若兩者不符,就表示 manifest 與規則互相矛盾。
三個門檻可以呈現其中的取捨,下表將三者並列,並附上各自的數量與比率。提高門檻不會使轉交的訊息變少,而規則會選出符合目標的最低值。每一列都是同一個 adapter 處理同一批 dev 訊息,因此只有門檻不同。
| 門檻 | dev 上轉交的訊息 | dev 上未轉交訊息的準確率 |
|---|---|---|
| 0.50 | 80/1003 (8.0%) | 811/923 (87.9%) |
| 0.63(所選) | 148/1003 (14.8%) | 770/855 (90.1%) |
| 0.90 | 352/1003 (35.1%) | 620/651 (95.2%) |
在 0.50 時,路由器保留較多訊息,但保留下來的訊息準確率較低,未能達成目標。在 0.90 時,保留下來的訊息在三者中最為準確,但交給人工處理的 dev 訊息也最多。所選的值是三個門檻中符合目標的最低者,因此轉交的訊息比 0.50 多,比 0.90 少。此表顯示了轉交數量上的代價,以及換得的準確率。目標要設在哪裡,是規則無法代為決定的商業選擇。本書採用的是保留訊息 0.90 的準確率;銀行若要運行自己的路由器,則會依這項取捨訂出自己的目標。
adapter 在正式環境中保持不變,但它所見到的訊息並非如此。假設某項產品問題使許多關於同一團隊的訊息在一週內湧入。門檻是依 dev 的訊息組成選出的,所以即使 adapter 完全沒有改變,轉交率也可能自行變動。轉交數上升而 adapter 並未更動,就是應該先檢查訊息組成的徵兆。憑目測來回調整門檻,直到轉交率看起來正確(Looks right)為止,會放棄當初選擇這項規則所要保住的特性:路由器保留的 dev 訊息,其準確率至少達到目標。
信心值並不是團隊判斷正確的機率。Guo 等人(2017)指出他們的網路「校準不佳」(poorly calibrated),意思是其信心與實際答對的頻率不相符。本書在 dev 上檢查自己的 adapter。dev 訊息依信心分為五個區間,每個區間各有訊息數、準確率與平均信心,後兩者正是 Guo 等人衡量校準時所比較的兩個量。依信心對 dev 分組,即可看出 0.9 是否真的代表十次中有九次答對。
1"""Group the dev messages into five bands by the adapter's confidence, and print each band's count, accuracy and mean confidence.2Read the top band's mean confidence against its accuracy. Only dev is read."""34import os56os.environ["HF_HUB_OFFLINE"] = "1"7os.environ["TRANSFORMERS_OFFLINE"] = "1"89import torch # noqa: E4021011torch.set_num_threads(2)1213import router_lib # noqa: E4021415router_lib.setup_runtime()16RECORDED = router_lib.HERE / "data" / "recorded" / "lora-full"17BANDS = [(0.0, 0.2), (0.2, 0.4), (0.4, 0.6), (0.6, 0.8), (0.8, 1.0)]1819tokenizer, model = router_lib.load_classifier(RECORDED / "adapter")20rows = router_lib.read_rows("train")21dev = router_lib.read_sample("dev")22golds = [rows[i][1] for i in dev]23preds, confs = router_lib.classify_by_length(model, tokenizer, [rows[i][0] for i in dev]) # results come back in dev order, so index i matches golds[i]2425print(f"dev messages: {len(dev)}")26counted = 027for lo, hi in BANDS:28 last = hi == 1.029 members = [i for i, c in enumerate(confs) if lo <= c < hi or (last and c == 1.0)] # lower edges are in, upper edges out; the top band also takes 1.0, so each message is counted once30 counted += len(members)31 name = f"band {lo:.1f}-{hi:.1f}"32 print(f"{name} messages: {len(members)}")33 if members:34 correct = sum(preds[i] == golds[i] for i in members)35 print(f"{name} correct: {correct}/{len(members)} ({correct / len(members):.1%})")36 print(f"{name} mean confidence: {sum(confs[i] for i in members) / len(members):.1%}")37 else:38 print(f"{name} correct: not defined (0 messages)")39 print(f"{name} mean confidence: not defined (0 messages)")40print(f"messages in bands: {counted}")
python calibration-bands-on-dev.pydev messages: 1003
band 0.0-0.2 messages: 1
band 0.0-0.2 correct: 1/1 (100.0%)
band 0.0-0.2 mean confidence: 19.9%
band 0.2-0.4 messages: 41
band 0.2-0.4 correct: 10/41 (24.4%)
band 0.2-0.4 mean confidence: 32.9%
band 0.4-0.6 messages: 92
band 0.4-0.6 correct: 49/92 (53.3%)
band 0.4-0.6 mean confidence: 51.0%
band 0.6-0.8 messages: 123
band 0.6-0.8 correct: 84/123 (68.3%)
band 0.6-0.8 mean confidence: 70.6%
band 0.8-1.0 messages: 746
band 0.8-1.0 correct: 693/746 (92.9%)
band 0.8-1.0 mean confidence: 96.4%
messages in bands: 1003
多數 dev 訊息落在最高的區間,即 0.8 到 1.0,其中有 746 則。最低的區間為 0.0 到 0.2,只有 1 則,數量太少,其準確率無法說明低信心的情況。
在 dev 上,信心對訊息的排序是有用的,但在最高區間,信心高於準確率:最高區間的平均信心為 96.4%,準確率則為 693/746 (92.9%)。五個區間中有四個的準確率落在其區間範圍內,最低區間是唯一的例外,而它只有一則訊息。從第二個區間往上,準確率逐步上升。在中間兩個區間,準確率與平均信心相當接近;而在 0.2 到 0.4 的區間,平均信心則遠高於準確率。
這些區間所能顯示的僅止於此:0.2 到 0.4 的區間只有 41 則訊息。此處每個數字都來自同一份 dev 樣本,而這份樣本同時也用來選擇門檻,所以沒有任何一個數字能顯示 adapter 在新訊息上的校準程度。
規則所選出的路由器,是在 dev 上取得的經驗分界,並連同其規則與資料一併記錄。它是一個分界,而不是訊息的模型,因此其數字描述的是 dev 的訊息組成。讀者將其轉交率與保留訊息的準確率視為路由器的成果,並在訊息組成改變時重新檢查 dev 規則。
實用原則: 依所述規則在 dev 切分上選擇門檻,並將其值與規則寫入 adapter 的 manifest。報告轉交率時,要與保留訊息的準確率並列,且各附數量,並至少呈現所選值兩側各一個設定。在信任信心值作為分界之前,先在 dev 上檢查校準;當訊息組成改變時,從訓練檔中重新抽取新的 dev 樣本,再次選擇門檻。
本章,於本版次中
版次 2026.11.9範例於 2026年11 月 20 通過測試本章 5 個範例 個範例中有 5 個通過
譯自英文書,版次 2026.11.9。