以人工評分衡量評審者的錯誤,再校正通過率
閱讀約需 54 分鐘 分鐘,7 節 中共有 6 個範例。
讀完本章,你將能夠衡量評審者的判定與人工評分有何差異,將這些差異拆分為漏判(將正確回覆判為不通過)與誤判通過,並針對這些錯誤校正觀察到的通過率,同時在旁寫明該校正所依據的假設。
第 5 章的結尾提出了校準任務的第二種措辭,並確立了一項規則:評審者的措辭須經衡量,而非預先假定。答案評審者會對照書面參考答案閱讀每則回覆,並標記為 PASS 或 FAIL,而助理基準執行的通過率即來自這位評審者。這個比率既是關於助理的數字,也是關於評審者的數字,因為評審者可能讓錯誤的回覆通過,也可能判定正確的回覆不通過。第 5 章撰寫了評分準則並檢視了各項判定。本章要問的是:這些判定有多常出錯、錯在哪個方向,以及團隊在掌握這些資訊之後,該如何處理這個比率。
人工評分及其性質
評審只能對照以其他方式產生的標籤來檢驗。本書使用 HelpSteer2 的人工評分,這是 NVIDIA 公開的資料集。每一列包含一則提示、對該提示的一則回應,以及由人給出的五項屬性分數,範圍為 0 到 4。這些回應來自其他模型。本章只使用其中一項屬性:正確性。取樣程式碼會把分數轉成通過或不通過:3 分或 4 分視為 PASS,0 到 2 分視為 FAIL。這個切分點是本書的選擇,並且在執行任何評審之前就已決定。
HelpSteer2 並不是本書自己的資料。它的回應來自其他模型,提示屬於一般性質,也不是為 sablefile 而寫。因此,這些標籤衡量的是評審在另一類問題上的表現,而這類問題與答案評審所回答的問題不同。這個差距正是遷移假設的主題,本章後續內容也會不斷回到這一點。
校準評審依據一份只針對這一項屬性撰寫的評分準則閱讀。準則詢問回覆中的每項事實與結果是否正確,並要求評審不考慮長度、語氣、風格與格式,如第 5 章所述。本書自己的測試集原本也可以用來校準,但本書沒有針對 sablefile 回覆的人工評分。因此校準改用公開資料列,代價是必須接受遷移假設。資料集並未提供正確性的通過線。這條線由本書自行設定,本章中的每個比率都要對照這條線來解讀。換一條線會使所有比率改變,所以讀者若要把本章與其他研究比較,應先確認該研究把線設在哪裡。
校準用的資料列是從資料集的驗證切分中,以固定種子抽出的樣本。文字中含有可列印 ASCII 以外字元的資料列,在抽樣前已先剔除。抽樣從每個切分各取約 45 列 FAIL 與 45 列 PASS,且每則提示只保留一列。因此基準率是由抽樣決定的,而不是在資料集中發現的。樣本分為開發集與保留集。同一資料集中還有針對同一提示的成對回應,以及人對兩者的偏好。偏好強度為零的成對資料則予以排除。
切分方式決定了這些數字可以如何使用。開發集是本書設定一切事項的地方,包括校正所用的比率。保留集只在本章讀取一次,且在其他一切都固定之後才讀取,這裡沒有任何選擇是依據它做出的。保留集的數字最初是在記錄檢查時計算的,早於本章的撰寫。這裡照當時的結果報告,並未為了挑選任何東西而重新計算。
實用原則: 校準時,請對照為評審所回答的問題而製作的標籤;若做不到,請在文中說明兩者相距多遠。在查看任何結果之前,先寫下把分數轉成評審通過或不通過的規則。用來調整的切分,要與只報告一次的切分分開。
一致率與 kappa
一致率是評審與人工給出相同判定的資料列所占的比例。當某一種判定占絕大多數時,一致率可能造成誤導。一個全部判為 PASS 的評審,與以 PASS 為主的樣本會有很高的一致率,而單靠一致率,無法分辨這樣的評審與謹慎的評審。解決的方法是:只根據兩者各自的 PASS 比率,求出兩組判定偶然會達到的一致率,再把這部分機遇扣除。kappa 就是這樣做的。對於兩位評分者做通過或不通過的判定,kappa 等於觀察到的一致率減去機遇一致率,再除以一減機遇一致率。McHugh 所寫的 kappa 統計量回顧給出了這個公式。當兩位評分者在每一列上都一致時,kappa 為 1。
1"""How often the reference-free correctness judge agrees with the human ratings on the HelpSteer2 sample, and Cohen's kappa2for the same counts, on dev, on held-out and on both together.34Agreement alone reads higher than the judge's skill when most rows are one verdict, so the kappa line is the one to read5against chance: the chance agreement printed beside it is what the judge would reach by guessing at its own PASS rate."""6import lib # first: fixes the thread counts and seeds before anything else loads78from lib.calibration import agreement, chance_agreement, cohens_kappa, confusion, helpsteer2_judgments9from lib.stats import pct1011for label, split in (("all", None), ("dev", "dev"), ("held-out", "held-out")):12 counts = confusion(helpsteer2_judgments(split))13 print(f"agreement, {label}: {pct(counts.tp + counts.tn, counts.n)}")14 # the chance figure is an expected rate, not a count, so it has no denominator beside it15 print(f"chance agreement, {label}: {100 * chance_agreement(counts):.1f}%")16 # kappa is printed with two decimals (plan section 6.5), not as a percentage17 print(f"kappa, {label}: {cohens_kappa(counts):.2f}")
python calibration-agreement.pyagreement, all: 70.6% (127 of 180)
chance agreement, all: 50.3%
kappa, all: 0.41
agreement, dev: 71.1% (64 of 90)
chance agreement, dev: 50.2%
kappa, dev: 0.42
agreement, held-out: 70.0% (63 of 90)
chance agreement, held-out: 50.3%
kappa, held-out: 0.40
全部資料列的一致率為 70.6% (127 of 180)。僅依兩者的 PASS 比率計算,機遇一致率為 50.3%。kappa 為 0.41,取到小數點後兩位且不加百分比符號,因為它是自成尺度的統計量,不是比率。一致率與機遇一致率都是百分比,因此可以直接比較。kappa 是已經扣除機遇的數字,所以應以它來判讀評審超出機遇之外的一致程度。
kappa 的範圍從 −1 到 1。在機遇一致率的水準時為 0;當兩位評分者一致的次數比機遇所預期的更少時,kappa 為負值,McHugh 的回顧也指出了這一點。
請依序閱讀這三個數字:先看一致率,再看旁邊的機遇數字,最後看 kappa。若一致率上升,而機遇一致率也隨之上升,那麼增加的部分可能來自基準率,而不是來自評審。
兩個切分的讀法相同。在 dev 上,一致率為 71.1% (64 of 90),dev 上的 kappa 為 0.42。在 held-out 上,一致率為 70.0% (63 of 90),kappa 為 0.40。
基準率正是 kappa 重要的原因。當某一種判定占據大多數資料列時,機遇一致率很高,評審即使分辨兩種判定的能力很差,也可能得到很高的一致率。這裡抽出的樣本中,PASS 列與 FAIL 列的數量大致相當,因此機遇一致率接近一半。dev 上人工的 PASS 率為 51.1% (46 of 90)。樣本偏斜的團隊應預期機遇一致率會接近其一致率,所以即使一致率看起來很高,kappa 仍會接近零。這類團隊應先讀 kappa。
一致率衡量的是與人工標註的接近程度,而不是回覆是否正確。高一致率只說明評審的判定與人的判定相符,並不表示助理的回覆正確。kappa 能告訴你一致率距離機遇有多遠,卻無法告訴你評審往哪個方向出錯。這需要接下來的兩個比率。
第 5 章介紹過第二種校準措辭,即評分準則 B。校準用的評分準則,以下稱評分準則 A,指出沒有做出任何錯誤陳述的回覆「並非不正確」。評分準則 B 改寫了這一點:它在 A 說「並非不正確」之處寫成「正確」,並刪去關於誤導與信心的要點。評分準則 B 只在 dev 上記錄過。下一個範例把兩種措辭放在相同的 dev 資料列上並排比較。
1"""How often the second rubric's verdicts agree with the human ratings on the HelpSteer2 sample, beside the first rubric's,2on dev and on held-out, with kappa, TPR and TNR and the PASS rates of judge and humans.34The second rubric was recorded on dev only, so its held-out line says so instead of printing a rate. The paired difference5is on dev alone: each dev row's agreement under rubric B minus its agreement under rubric A, bootstrapped over rows. A6reply with no readable verdict is printed by row ID and left out; it is never counted as FAIL."""7import lib # first: fixes the thread counts and seeds before anything else loads89from dataclasses import replace1011from lib.calibration import cohens_kappa, confusion, helpsteer2_judgments, readable_verdicts12from lib.prompts import HS2B_SHA25613from lib.replay import ReplayModel14from lib.stats import bootstrap_mean, interval_points, pct1516RUBRIC_B = "judge-helpsteer2-rubric-b"171819def report(name, split, judgments):20 c = confusion(judgments)21 print(f"agreement, {name}, {split}: {pct(c.tp + c.tn, c.n)}")22 print(f"kappa, {name}, {split}: {cohens_kappa(c):.2f}")23 print(f"TPR, {name}, {split}: {pct(c.tp, c.tp + c.fn)}")24 print(f"TNR, {name}, {split}: {pct(c.tn, c.tn + c.fp)}")25 print(f"judge PASS rate, {name}, {split}: {pct(c.tp + c.fp, c.n)}")26 print(f"human PASS rate, {name}, {split}: {pct(c.tp + c.fn, c.n)}")272829def same(j):30 return int(j.judge_pass == j.human_pass)313233base = {j.id: j for j in helpsteer2_judgments()} # rubric A's verdicts and the human PASS for every row, by id34recorded = {item for item, _ in ReplayModel(RUBRIC_B, HS2B_SHA256).keys()}35verdicts_b, unreadable = readable_verdicts(RUBRIC_B, sorted(recorded & base.keys()), HS2B_SHA256)36rows_b = {i: replace(base[i], judge_pass=v == "PASS") for i, v in verdicts_b.items()}3738print(f"unreadable, rubric B: {len(unreadable)} of {len(recorded)} replies")39for item_id, _ in unreadable:40 print(f"unreadable, rubric B, row {item_id}: excluded, not counted as FAIL")41for split in ("dev", "held-out"):42 report("rubric A", split, [j for j in base.values() if j.split == split])43 b = [j for j in rows_b.values() if j.split == split]44 if b:45 report("rubric B", split, b)46 else:47 print(f"rubric B, {split}: no readable replies recorded")4849dev_ids = [i for i in verdicts_b if base[i].split == "dev"]50diffs = [same(rows_b[i]) - same(base[i]) for i in dev_ids] # the same row under both rubrics51lo, hi = bootstrap_mean(diffs) # level 0.95 for one comparison; seed SEED, 2,000 resamples52n = len(dev_ids)53print(f"paired difference, dev: {100 * sum(diffs) / n:+.1f} points, 95% interval {interval_points(lo, hi)}, n = {n}")54reading = "rubric B better" if lo > 0 else "rubric B worse" if hi < 0 else f"no difference detected at n = {n}"55caveat = ", on dev only with no held-out run" if lo > 0 or hi < 0 else "" # a dev-only difference carries its caveat56print(f"paired difference, dev, reading: {reading}{caveat}")
python rubric-b-agreement.pyunreadable, rubric B: 0 of 90 replies
agreement, rubric A, dev: 71.1% (64 of 90)
kappa, rubric A, dev: 0.42
TPR, rubric A, dev: 80.4% (37 of 46)
TNR, rubric A, dev: 61.4% (27 of 44)
judge PASS rate, rubric A, dev: 60.0% (54 of 90)
human PASS rate, rubric A, dev: 51.1% (46 of 90)
agreement, rubric B, dev: 74.4% (67 of 90)
kappa, rubric B, dev: 0.49
TPR, rubric B, dev: 84.8% (39 of 46)
TNR, rubric B, dev: 63.6% (28 of 44)
judge PASS rate, rubric B, dev: 61.1% (55 of 90)
human PASS rate, rubric B, dev: 51.1% (46 of 90)
agreement, rubric A, held-out: 70.0% (63 of 90)
kappa, rubric A, held-out: 0.40
TPR, rubric A, held-out: 82.6% (38 of 46)
TNR, rubric A, held-out: 56.8% (25 of 44)
judge PASS rate, rubric A, held-out: 63.3% (57 of 90)
human PASS rate, rubric A, held-out: 51.1% (46 of 90)
rubric B, held-out: no readable replies recorded
paired difference, dev: +3.3 points, 95% interval [-1.1, +7.8] points, n = 90
paired difference, dev, reading: no difference detected at n = 90
dev 上沒有顯示出差異。在評分準則 B 下,一致率為 74.4% (67 of 90)。在相同資料列上,評分準則 A 下的一致率為 71.1% (64 of 90)。配對差異為 +3.3 points, 95% interval [-1.1, +7.8] points, n = 90。對每一列 dev 資料,程式碼在某種措辭與人工一致時記為 1,不一致時記為 0。配對差異是評分準則 B 的得分減去評分準則 A 的得分,再對 dev 資料列取平均,其區間是對這些資料列做自助法(bootstrap)得到的。範例對該區間的判讀為 no difference detected at n = 90。
這個區間包含零,因此 dev 上的估計並未顯示評分準則 B 比較好。評分準則 B 沒有 held-out 的執行結果,所以它的 dev 估計沒有第二次檢驗。它無法讀取的回覆數為 0 of 90 replies,因此沒有任何回覆被排除。
實用原則: 報告一致率時要並列 kappa,並對照旁邊印出的機遇數字來解讀 kappa。每個數字都要註明評分準則與切分,避免把兩種措辭當成同一種來解讀。只有在配對區間的下界高於零,而且是在 held-out 上,或是在 dev 上並於主張旁附註該限制時,才可說某一種措辭比較好。當區間跨越零時,應說明在現有的 n 之下沒有偵測到差異,不要把較高的點估計當成主張。
此部分尚未翻譯,因此以英文呈現。
True positive and true negative rates
Agreement and kappa say how often the judge agrees with the humans. They do not say which way it errs. The two rates that do are the true positive rate and the true negative rate. The true positive rate, TPR, is the share of human PASS rows that the judge passes. The true negative rate, TNR, is the share of human FAIL rows that the judge fails. A judge that passes too much has a high TPR and a low TNR, because its errors are false passes. A judge that fails too much is the reverse. The correction in the next section needs both rates, since each one fixes a different error.
1"""The judge's true positive rate and true negative rate on the HelpSteer2 sample, with the counts beside each, on dev, on2held-out and on both together.34PASS is the positive class: TPR is the share of human PASS rows the judge passes, and TNR the share of human FAIL rows it5fails. The two rates are the judge's error profile; a lenient judge has a high TPR and a low TNR."""6import lib # first: fixes the thread counts and seeds before anything else loads78from lib.calibration import confusion, helpsteer2_judgments9from lib.stats import pct1011# PASS is the positive class: the rows a human rated PASS are the positives, and TPR is taken over them.12print("positive class: PASS")13for label, split in (("all", None), ("dev", "dev"), ("held-out", "held-out")):14 counts = confusion(helpsteer2_judgments(split))15 print(f"TPR, {label}: {pct(counts.tp, counts.tp + counts.fn)}")16 print(f"TNR, {label}: {pct(counts.tn, counts.tn + counts.fp)}")
python tpr-tnr.pypositive class: PASS
TPR, all: 81.5% (75 of 92)
TNR, all: 59.1% (52 of 88)
TPR, dev: 80.4% (37 of 46)
TNR, dev: 61.4% (27 of 44)
TPR, held-out: 82.6% (38 of 46)
TNR, held-out: 56.8% (25 of 44)
Here PASS is the positive class, so TPR is taken over the human PASS rows and TNR over the human FAIL rows. Each rate prints its counts beside it, so a reader can see how many rows it rests on.
On all rows, TPR is 81.5% (75 of 92). TNR is 59.1% (52 of 88). The judge passes most of the replies that humans passed, and it also passes a fair share of the replies humans failed. The weaker of the two rates is TNR, which makes this judge lenient.
The direction matters for anyone who reads a pass rate, and it depends on where the rate sits. The pass-rate correction in the next section leaves a rate unchanged at the crossover point, p* = (1 − TNR) / (2 − TPR − TNR). Below p* the correction lowers the rate, and above p* it raises the rate. For a lenient judge, p* sits high, so a reported rate below it reads too high and a rate above it reads too low. On dev, the crossover point is 66.4% (TPR 80.4%, TNR 61.4%). Knowing where a rate sits against that point tells a team which way its reported number leans, before any correction is applied. Agreement alone would hide this. Two judges can agree with the humans to the same degree and still differ in which error they make more often, and the rates they report would differ with it.
On dev, TPR is 80.4% (37 of 46), and TNR is 61.4% (27 of 44). On held-out, TPR is 82.6% (38 of 46), and TNR is 56.8% (25 of 44). The held-out split lands close to dev on both rates, with TNR a few points lower.
Each rate is taken over one group of human labels, and each split has few enough rows that another sample could move the rates. This chapter prints no interval for TPR or TNR. Read the rates as the judge's error on this sample, not as its error in general.
Working rule: Report TPR and TNR together, with their counts, and say which way the judge errs. Take the rates a correction uses from dev alone, and quote both beside any corrected rate. Report held-out once, after the choices are made, and keep it out of every choice.
此部分尚未翻譯,因此以英文呈現。
Correcting a pass rate
The observed pass rate counts two kinds of verdict: true passes the judge caught, and false passes it let through. The correction tries to undo both at once. Given the judge's TPR and TNR, it solves for the true share of passes that would produce the observed rate. The book writes it as (observed + TNR - 1) / (TPR + TNR - 1). Rogan and Gladen (1978) address the same problem for screening tests, and their abstract says that adjustments of this kind are given. The book has read only that abstract, so the formula here is the book's own statement of the correction, as its code computes it, and not a quotation from the paper.
The correction has a limit. It divides by TPR + TNR - 1. A judge whose two rates sit near what chance would give has a denominator near zero, and then small errors in either rate become large swings in the corrected rate. The book's code refuses to correct when that denominator is below 0.1, and it raises an error instead of printing a rate. The run below stays well clear of that floor. In practice a team does four things in order: measure TPR and TNR on a calibration set kept apart from the set it will correct, check that the denominator clears the floor, apply the correction to the observed rate over the readable replies, and write the transfer assumption beside the result.
1"""The baseline's answer-judge pass rate over the test items the judge gave a readable verdict, corrected for the judge's errors2with TPR and TNR from dev only. The corrected rate is printed beside the observed rate and the two rates the correction uses.34One judge reply has no PASS or FAIL on its first line. It is left out of the rate and printed by name, never counted as a5FAIL. The TPR and TNR come from HelpSteer2, where the judge rates other models' answers without a reference. This judge reads6the answers against a written reference. The correction assumes the two error profiles transfer; nothing here tests that.7The last lines apply the same correction to the calibration set itself: the rates are measured on those same rows, so that8check shows the arithmetic, not how well the correction generalises."""9import lib # first: fixes the thread counts and seeds before anything else loads1011from lib.calibration import (12 confusion,13 corrected_pass_rate,14 crossover_pass_rate,15 helpsteer2_judgments,16 readable_verdicts,17 tnr,18 tpr,19)20from lib.judging import passed21from lib.prompts import REF_SHA25622from lib.stats import pct23from lib.testset import load_items2425items = load_items() # all 120 items of the test set, dev and held-out together26found, unreadable = readable_verdicts("judge-answers-baseline", [item.id for item in items], REF_SHA256)27details = "".join(f' ({item_id}, "{first}")' for item_id, first in unreadable)28print(f"unreadable verdicts: {len(unreadable)}{details}") # left out of the rate below, not counted as FAIL29passes = int(passed(found).sum()) # found is keyed in sorted id order30observed = passes / len(found)31print(f"observed pass rate, readable items: {pct(passes, len(found))}")3233dev = confusion(helpsteer2_judgments("dev")) # the only rows the correction's TPR and TNR may come from34print(f"TPR, dev: {pct(dev.tp, dev.tp + dev.fn)}")35print(f"TNR, dev: {pct(dev.tn, dev.tn + dev.fp)}")36# Below this observed rate the correction lowers the rate; above it, the correction raises it.37crossover = crossover_pass_rate(tpr(dev), tnr(dev))38print(f"crossover point, dev: {100 * crossover:.1f}% (TPR {100 * tpr(dev):.1f}%, TNR {100 * tnr(dev):.1f}%)")3940corrected = corrected_pass_rate(observed, tpr(dev), tnr(dev)) # refused if TPR + TNR - 1 is below 0.141print(f"corrected pass rate: {100 * corrected:.1f}%")4243# The sanity check: the same correction on the calibration set, with rates measured on those same rows. It checks the44# arithmetic, not transfer to the answers: it should return the human PASS rate, and it does by construction.45calibration = confusion(helpsteer2_judgments())46judge_rate = (calibration.tp + calibration.fp) / calibration.n47print(f"calibration set, judge PASS rate: {pct(calibration.tp + calibration.fp, calibration.n)}")48print(f"calibration set, corrected judge PASS rate: {100 * corrected_pass_rate(judge_rate, tpr(calibration), tnr(calibration)):.1f}%")49print(f"calibration set, human PASS rate: {pct(calibration.tp + calibration.fn, calibration.n)}")50calibration_crossover = crossover_pass_rate(tpr(calibration), tnr(calibration)) # the same test, on the same rows51print(f"crossover point, calibration set: {100 * calibration_crossover:.1f}% "52 f"(TPR {100 * tpr(calibration):.1f}%, TNR {100 * tnr(calibration):.1f}%)")
python corrected-pass-rate.pyunreadable verdicts: 1 (q-0043, "VERDICT: PARTIAL")
observed pass rate, readable items: 76.5% (91 of 119)
TPR, dev: 80.4% (37 of 46)
TNR, dev: 61.4% (27 of 44)
crossover point, dev: 66.4% (TPR 80.4%, TNR 61.4%)
corrected pass rate: 90.5%
calibration set, judge PASS rate: 61.7% (111 of 180)
calibration set, corrected judge PASS rate: 51.1%
calibration set, human PASS rate: 51.1% (92 of 180)
crossover point, calibration set: 68.9% (TPR 81.5%, TNR 59.1%)
The input is the baseline's answer-judge verdicts for the test items, all 120 of them, dev and held-out together. One reply had no readable verdict, because its first line was not PASS or FAIL. The example prints it by name and leaves it out of the rate, since a reply the judge could not read is not a FAIL. Chapter 5 explains the two ways the code reads an unreadable verdict. The line for it reads 1 (q-0043, "VERDICT: PARTIAL"). Chapter 1 defines the pass rate over every item in the set. This rate is over the readable items, so it departs from that definition by the one reply left out.
The observed pass rate over the readable items is 76.5% (91 of 119). The correction uses TPR and TNR from dev, and those are the only rows it may use. Its TPR on dev is 80.4% (37 of 46). Its TNR on dev is 61.4% (27 of 44).
The corrected rate is 90.5%. The corrected rate estimates the share of sablefile's replies that a person would pass, under the book's correctness cut. It is not the judge's pass rate, which is the observed rate above, and it is not the assistant's accuracy. The estimate holds only if the judge's errors on sablefile's answers match its errors on HelpSteer2 rows. That is the transfer assumption, and the TPR and TNR above come from a reference-free judge rating other models' responses, while the answer judge checks sablefile's replies against a reference.
The corrected rate is higher than the observed rate, and the crossover point shows why. The observed rate sits above the crossover point, where the judge reads low. At that rate the judge's misses on true passes outweigh its false passes, so the correction raises it.
The result is also close to the code's ceiling. The code clips the corrected rate to the range from 0 to 1. The unclipped value passes 1 exactly when the observed rate is above TPR, because then the numerator is larger than the denominator. Here the observed rate sits under TPR by less than four points. If two of the dev PASS rows the judge passed had been failed instead, TPR would fall below the observed rate, and the corrected rate would print 100.0%.
Working rule: Correct a pass rate only with TPR and TNR measured on a calibration set. Quote both beside the corrected rate, refuse when the denominator is small, and say in the same paragraph what the correction assumes. The next section gives the interval and the one-row moves that go beside any corrected rate.
一列資料對校正後通過率的影響
1"""How much the corrected pass rate of corrected-pass-rate depends on its inputs. A 95% bootstrap interval resamples both the2dev calibration rows and the readable test items, with the book's fixed seed. It prints the share of resamples that reach3the 100% cap, and the corrected rate when one dev row moves TPR or TNR up or down. Only dev rows set the correction's rates."""4import lib # first: fixes the thread counts and seeds before anything else loads56from dataclasses import replace78from lib.calibration import confusion, correction_interval, corrected_pass_rate, helpsteer2_judgments, readable_verdicts, tnr, tpr9from lib.judging import passed10from lib.prompts import REF_SHA25611from lib.stats import RESAMPLES, interval_pct, pct12from lib.testset import load_items1314items = load_items() # all 120 items of the test set, as in corrected-pass-rate15found, _ = readable_verdicts("judge-answers-baseline", [item.id for item in items], REF_SHA256)16observed = int(passed(found).sum()) / len(found)1718dev_rows = helpsteer2_judgments("dev")19dev = confusion(dev_rows)20lo, hi, capped, refused = correction_interval(dev_rows, passed(found)) # resamples the dev rows and the readable items21print(f"corrected pass rate, 95% interval: {interval_pct(lo, hi)}")22print(f"resamples at the 100% cap: {pct(capped, RESAMPLES)}")23print(f"resamples refused, TPR + TNR - 1 below 0.1: {pct(refused, RESAMPLES)}")2425# One dev row moved: a human PASS row judged the other way changes TPR; a human FAIL row changes TNR.26moved = {27 "TPR one row higher": replace(dev, tp=dev.tp + 1, fn=dev.fn - 1),28 "TPR one row lower": replace(dev, tp=dev.tp - 1, fn=dev.fn + 1),29 "TNR one row higher": replace(dev, tn=dev.tn + 1, fp=dev.fp - 1),30 "TNR one row lower": replace(dev, tn=dev.tn - 1, fp=dev.fp + 1),31}32for label, counts in moved.items():33 print(f"corrected, {label}: {100 * corrected_pass_rate(observed, tpr(counts), tnr(counts)):.1f}%")
python correction-sensitivity.pycorrected pass rate, 95% interval: [62.1%, 100.0%]
resamples at the 100% cap: 29.5% (590 of 2000)
resamples refused, TPR + TNR - 1 below 0.1: 0.1% (2 of 2000)
corrected, TPR one row higher: 86.0%
corrected, TPR one row lower: 95.5%
corrected, TNR one row higher: 91.0%
corrected, TNR one row lower: 90.0%
這個範例以書中固定的隨機種子,對開發資料列與可讀項目一併重抽樣 2,000 次,並觀察校正後通過率的分布範圍。95% 區間為 [62.1%, 100.0%]。這個區間很寬,因為 TPR 與 TNR 各自只依據數十列開發資料,而且區間的上端正是上限本身。部分重抽樣會達到上限,其比例為 29.5% (590 of 2000)。因分母低於 0.1 而被拒絕的重抽樣為 0.1% (2 of 2000),區間則是根據其餘的重抽樣計算。
單列的變動如下。若有一列人工判為通過的開發資料,被評審判為 PASS 而非 FAIL,TPR 會高出一列,校正後的通過率會是 86.0%。若這樣的一列被判為 FAIL,TPR 會低一列,校正後的通過率會是 95.5%。至於 TNR,若有一列人工判為不通過、而評審判為通過的開發資料,在被判為 FAIL 時,比率會變為 91.0%;在被判為 PASS 時,則會變為 90.0%。TPR 是需要留意的比率:TPR 一列的變動會使答案移動四到五個百分點,而 TNR 一列只會移動約半個百分點。
這個區間涵蓋開發資料列與可讀項目的抽樣變異,但不涵蓋轉移假設。若評審在 sablefile 的回覆上犯錯的方式不同,通過率的變動可能超過重抽樣所顯示的幅度。
實用原則: 報告校正後通過率時,請附上本書用於各項比率的自助法(bootstrap)區間、位於上限的重抽樣比例,以及單列的變動,讓讀者能看出單一資料列的影響有多大。
此部分尚未翻譯,因此以英文呈現。
The transfer assumption
The TPR and TNR above come from rows where the judge read other models' responses with no reference, under the correctness rubric, and the PASS line there is HelpSteer2 correctness of 3 or more. The answer judge reads sablefile's replies against a written reference, and a PASS there means the reply agrees with the reference, with a correct decline counting as a pass. The two judges answer different questions about what right means. The transfer assumption is that the judge's miss rate on human passes, and its pass-through rate on human fails, are the same on sablefile's replies as on the HelpSteer2 rows. Nothing in this chapter tests that. The correction holds only if the differences in input, replies, questions and PASS criterion leave the error profile unchanged. Chapter 5 sets out the difference between a judge with a reference and one without.
The last lines of the same example apply the correction to the calibration rows, with rates taken from those rows. On those rows, the judge's PASS rate is 61.7% (111 of 180). Its crossover point is 68.9% (TPR 81.5%, TNR 59.1%), and the judge's rate sits below it, so the correction lowers the rate. The corrected judge PASS rate is 51.1%. The human PASS rate is 51.1% (92 of 180).
The corrected judge rate returns the human rate. That follows from the algebra, because the rates were measured on those same rows. The check confirms that the arithmetic is right. It says nothing about whether the correction applies to sablefile, and a reader should not take it as evidence for the transfer.
The better fix is labels on the book's own replies. The book has none, because no person has rated sablefile's replies. A reader with such labels should calibrate on them, keeping a split apart from the set being corrected, as this chapter keeps dev apart from held-out. That would remove most of the transfer assumption, where the public rows only pay for it.
Working rule: Write the transfer assumption beside every corrected rate, in the same paragraph, and name the difference between the calibration judge and the judge the rate describes. Check the arithmetic on the rows the rates came from, and say that such a check does not test transfer. Until human labels on the answers the rate describes exist and have been checked against the judge, call the corrected rate an estimate that holds if the assumption does.
成對校準
成對評審讀入兩則回覆,從中挑出較好的一則,而不是對每一則分別判定通過或不通過。人類一方的依據是 HelpSteer2 中,針對同一提示的兩則回應之間的偏好標註。成對評審有一項已知的弱點:哪一則回覆放在 A 位置、哪一則放在 B 位置,可能左右它的選擇。第 5 章說明本書為何要求評審以兩種順序各問一次每組配對,第 7 章則分析位置造成的順序改變。本節要問的是:把兩種順序都計入之後,與人類偏好的一致率會是什麼樣子。本節印出的順序改變數,就是第 5 章印出的那個數字,在此只用來說明兩種比率之間的差距。
每組配對都是一個 HelpSteer2 提示,附兩則回應及人類在兩者之間的偏好。平手的配對已被剔除,因此每組配對都有人類的選擇。評審給出 TIE 算作未命中,因為人類選了其中一則,評審卻說兩則回覆相當。只取單一順序,得到的比率取決於評審偏好哪一個字母;因此只回報單一順序的團隊,回報的是一個會隨回覆位置而移動的數字。兩種順序都計入,可以消除這份自由度,代價是數字較低。我們建議對成對評審回報較嚴格的那個比率,並把單一順序的比率僅當作診斷兩者差距的工具來讀。
執行範例之前,請先預測:只有在兩種順序都選中人類所選回應時才算數的配對,其得分會高於還是低於單獨取任一種順序。然後再用輸出檢驗你的猜測。
1"""How often the pairwise judge's verdict agrees with the human preference on each HelpSteer2 pair, dev and held-out apart.2Three rates: the A,B order alone, the B,A order alone (both mapped back to the original responses), and both orders together,3where a pair counts only when both orders pick the human's response. A flipped pair (the same letter won both times) can4never count in the last rate, so the flipped count printed beside it is the number of misses that come from position.56Each single-order rate uses the pairs whose verdict in that order is readable; a TIE is judged and is not a pick, so it is7a miss. The interval is the 95% percentile bootstrap over pairs of the both-orders rate."""8import lib # first: fixes the thread counts and seeds before anything else loads910from lib.judging import both_orders, human_pairs11from lib.stats import bootstrap_mean, interval_pct, pct1213AB = "judge-hs2-pairs-ab"14BA = "judge-hs2-pairs-ba"1516pairs = human_pairs() # all 150 pairs, sorted by id, each with its split and the human's pick17orders = both_orders(AB, BA, list(pairs)) # both recordings read for every pair, sorted by id1819for split in ("dev", "held-out"):20 rows = [o for o in orders if pairs[o.pair_id].split == split]21 human = {o.pair_id: pairs[o.pair_id].preferred for o in rows}22 ab = [o.ab == human[o.pair_id] for o in rows if o.ab is not None]23 ba = [o.ba == human[o.pair_id] for o in rows if o.ba is not None]24 both = [o.ab == human[o.pair_id] and o.ba == human[o.pair_id] for o in rows if o.ab is not None and o.ba is not None]25 flipped = sum(1 for o in rows if o.kind == "flipped")26 print(f"agreement, A,B order, {split}: {pct(sum(ab), len(ab))}")27 print(f"agreement, B,A order, {split}: {pct(sum(ba), len(ba))}")28 print(f"agreement, both orders, {split}: {pct(sum(both), len(both))}; flipped pairs: {flipped}")29 lo, hi = bootstrap_mean([float(hit) for hit in both]) # 95% interval over pairs, the level for one comparison30 print(f"95% interval, both orders, {split}: {interval_pct(lo, hi)}")
python pairwise-agreement.pyagreement, A,B order, dev: 77.3% (58 of 75)
agreement, B,A order, dev: 73.3% (55 of 75)
agreement, both orders, dev: 65.3% (49 of 75); flipped pairs: 15
95% interval, both orders, dev: [53.3%, 76.0%]
agreement, A,B order, held-out: 74.7% (56 of 75)
agreement, B,A order, held-out: 76.0% (57 of 75)
agreement, both orders, held-out: 62.7% (47 of 75); flipped pairs: 19
95% interval, both orders, held-out: [52.0%, 73.3%]
兩種順序都計入時,一致率低於任何單一順序,而差距來自順序改變的配對。在 dev 上,單看 A,B 順序得到 77.3% (58 of 75),單看 B,A 順序得到 73.3% (55 of 75),兩種順序都計入則得到較嚴格的比率 65.3% (49 of 75); flipped pairs: 15。
分號之後的數字是順序改變的配對數,範例輸出將它標為 flipped。依第 5 章的定義,順序改變的配對是指判定結果跟隨位置而非回覆的配對。順序改變的配對永遠不可能計入雙順序比率,因此每一組在該比率中都是未命中。
held-out 切分呈現相同的模式。A,B 順序為 74.7% (56 of 75),B,A 順序為 76.0% (57 of 75),兩種順序都計入則為 62.7% (47 of 75); flipped pairs: 19。
雙順序比率以配對為單位的區間,在 dev 上是 [53.3%, 76.0%],在 held-out 上是 [52.0%, 73.3%]。這些區間很寬,因為每個切分的配對數都不多。站得住腳的解讀是整體模式:一個會隨位置改變選擇的評審,無法以任何單一順序來解讀為與人類一致。
讀完本章,讀者能夠衡量評審超出機率水準的一致程度、它在這份樣本上的未命中與誤判通過,以及一個經過校正的通過率,並在旁邊寫明其遷移假設。這些衡量建立在公開資料列上,而不是 sablefile 的回覆上;而且即使雙順序計入,成對評審仍會因位置而損失配對。第 7 章要在本書自己的錄製資料上測試評審所能顯示的極限,從位置開始,讀者也將在那裡學到何時該停止信任一項判定。
實用原則: 當評審比較兩則回覆時,兩種順序都要執行,並回報雙向計入的一致率,同時附上順序改變的配對數。切勿把單一順序的比率當作評審與人類的一致率。要回報以配對為單位的區間,並說明順序改變的配對算作未命中,因為那正是位置所造成的代價。
本章,於本版次中
版次 2026.11.3範例於 2026年11 月 21 通過測試本章 6 個範例 個範例中有 6 個通過
譯自英文書,版次 2026.11.3。