判斷一項改動是否更好,以及需要多少題目
閱讀約需 48 分鐘 分鐘,6 節 中共有 5 個範例。
讀完本章,你能利用成對比較上的區間,判斷助理兩個版本之間的差異是否不只是雜訊。你也能算出,在測試足以顯示某個大小的差異之前,需要多少題目。
第 1 章對助理量測了一次,並為其通過率給出區間。接著,它在二十道題目上比較了兩個提示詞。兩者差異的區間有 30.0 points 寬,該章以一項承諾作結:之後會有一章算出,差異需要多少題目才能顯示出來。本章履行這項承諾。更動提示詞的團隊需要三個答案。通過率的變動,是否出於測試中的隨機波動以外的原因?這項比較是否有能力顯示出差異?而若要顯示出團隊在意的那個差異,需要多少題目?
這裡的每個數字都來自同一個助理模型 Qwen3-4B-Instruct-2507,以及同一個評審,也就是以溫度零執行的同一個模型。這些區間涵蓋的是抽取題目所產生的波動。第一節量測的是對回覆取樣所產生的波動,不附區間。這些區間並未涵蓋評審的偏差,這部分由第 7 章檢驗。閱讀每項結果時,請記得這一點。
單一題目內的變異
同一個問題問兩次,模型可能給出不同的回答。本章的比較對每個題目與提示詞只使用一則已記錄的回覆,因此每個判定都只建立在一則回覆上。溫度較高時,模型會進行取樣,同一題目的判定可能從一次取樣到下一次取樣而改變。這對比較而言很重要。兩個提示詞之間的差異若小於取樣之間的擺盪幅度,就根本稱不上差異;而只從單次執行讀出的比率,會把這種擺盪藏起來。
變異集收錄了助理對每個開發題目的五次取樣回覆,取樣溫度為 0.7。評審以溫度零閱讀每一則回覆。若一個題目的五個判定並非全部相同,該題目即為翻轉。計算翻轉的題數,就能在做出任何修改之前,量出判定在取樣之間的變動程度。
1"""The flip rate over the dev variance set: the share of items whose judge verdict differs across their five samples. The2printed line 'flip rate' rests on the per-item PASS counts, which are printed with it: how many items passed 0, 1, 2, 3, 43or 5 of 5 samples, and the mean pass rate. The temperature-0 baseline's pass rate over the same items is printed for4comparison. An item with an unreadable verdict is printed by id and left out, never counted as FAIL."""5import lib # first: fixes the thread counts and seeds before anything else loads67from lib.calibration import readable_verdicts8from lib.judging import passed9from lib.prompts import REF_SHA25610from lib.stats import pct11from lib.testset import load_items12from lib.variance import VARIANCE_SAMPLES, item_pass_counts1314VARIANCE = "judge-answers-variance"15BASELINE = "judge-answers-baseline"16SAMPLES = len(VARIANCE_SAMPLES)1718items = [item.id for item in load_items(split="dev")] # eighty dev items, sorted by id19counts = item_pass_counts(VARIANCE, items)20excluded = [row for row in counts if not row.complete]21kept = [row for row in counts if row.complete]22for row in excluded:23 samples = ", ".join(str(sample) for sample in row.unreadable)24 print(f"unreadable, {row.item_id}: samples {samples} (left out, not counted as FAIL)")25print(f"left out of the flip rate (unreadable verdict): {len(excluded)}")2627n = len(kept)28print(f"variance items compared (n): {n}")29print(f"flip rate: {pct(sum(1 for row in kept if row.flaky), n)}")30spread = [sum(1 for row in kept if row.passes == passes) for passes in range(SAMPLES + 1)]31assert sum(spread) == n # every kept item falls in exactly one PASS count32for passes, count in enumerate(spread):33 print(f"items with {passes} of {SAMPLES} PASS: {count}")34print(f"mean per-item pass rate across samples: {pct(sum(row.passes for row in kept), n * SAMPLES)}")3536found, baseline_unreadable = readable_verdicts(BASELINE, [row.item_id for row in kept], REF_SHA256)37assert not baseline_unreadable # q-0043 is already left out above, so no kept item is unreadable under the baseline38baseline = passed(found)39print(f"baseline pass rate, same items (temperature 0): {pct(int(baseline.sum()), len(baseline))}")
python flip-rate.pyunreadable, q-0043: samples 1, 4 (left out, not counted as FAIL)
left out of the flip rate (unreadable verdict): 1
variance items compared (n): 79
flip rate: 2.5% (2 of 79)
items with 0 of 5 PASS: 16
items with 1 of 5 PASS: 1
items with 2 of 5 PASS: 0
items with 3 of 5 PASS: 0
items with 4 of 5 PASS: 1
items with 5 of 5 PASS: 61
mean per-item pass rate across samples: 78.5% (310 of 395)
baseline pass rate, same items (temperature 0): 78.5% (62 of 79)
程式逐題統計其五次取樣中有幾次獲評審判為通過。判定無法讀取的題目會被排除,不計為失敗,輸出中會指出被排除的題目。
多數題目不會變動。61 個題目五次取樣全部通過,16 個題目五次全部失敗。在翻轉的題目中,1 個只有一次取樣通過,1 個有四次取樣通過。
翻轉率為 2.5% (2 of 79)。
所有取樣的平均通過率為 78.5% (310 of 395)。基準線(每個題目在溫度零下取一則回覆)的通過率為 78.5% (62 of 79)。兩者相等,是因為兩個翻轉的題目朝相反方向拉動。這項結果適用於 P1,也就是變異集所用的提示詞,其他提示詞可能不那麼穩定。
這是一個範圍很窄的結果。含有更多臨界問題的題目集會翻轉得更多,而翻轉率是以題數計算,本身沒有信賴區間。翻轉題數這麼少,對於下一次修改多常會翻轉,幾乎說明不了什麼。
翻轉率只衡量在題目固定的情況下,來自取樣這個助理回覆的離散程度。兩個提示詞之間的比較,還會隨各提示詞答對的題目而變動。最後一節把這個取樣離散程度當作下界,而且只用於取樣其回覆的比較:差異需要多少題目。
實用原則: 在信任單次執行的比率之前,先以實際上線的溫度執行變異集,並將翻轉率與平均通過率並列閱讀。翻轉率說明取樣使單一提示詞的判定移動多遠。只有當該比較同樣對回覆取樣時,它才構成該比較的界限。
此部分尚未翻譯,因此以英文呈現。
The bootstrap over items
Chapter 1 showed an interval on a pass rate and used it to read a small set. This section explains the method, because the rest of the chapter uses it three ways: on a rate, on a paired difference, and on a difference that has to clear a stricter level.
The method is the percentile bootstrap. The items in a set are one sample of the questions a team could have written, so a different set would give a somewhat different rate. To estimate how much, draw items from the set with replacement, as many as the set holds, so that some items come up twice and some not at all. Compute the statistic for that draw, repeat the draw many times, and keep the middle 95% of the results. The bootstrap adds no data. It reuses the items the set already has, and it shows how far the statistic would move if the set had been another draw of the same kind.
The unit of resampling is the item, and that is the rule to keep. For a rate, each item contributes its verdict. For a paired difference, each item contributes one number: the verdict of prompt B minus the verdict of the baseline, which is plus one, zero or minus one. For the variance set, the five samples of an item are kept together, because they answer one question. Drawing them apart would make them look like five independent questions, and the interval would come out too narrow. Miller (2024) makes the same point for questions that come in related groups. That paper is a preprint and a practitioner's note, not a reviewed method, so this book uses it for the reason and not as proof.
The book fixes the settings so that a number can be reproduced. The method draws 2,000 resamples with a seed of zero, and it creates a fresh random generator inside each call. The same data therefore gives the same interval, whatever ran before it. The seed is a choice, not a property of the data. A different seed moves the endpoints a little, which is why the settings are written down.
The method also depends on the set being a fair sample of the questions, which Chapter 1 argued and no interval can check. A small set gives a rough interval. The endpoints of a percentile interval on a set of this size are a first estimate, and they should be read as approximate.
Working rule: Resample items, never runs within an item, and keep an item's samples together. Fix the resample count and the seed in the code, so the same data gives the same interval. When the set holds few items, call the interval rough and say so beside it.
此部分尚未翻譯,因此以英文呈現。
Paired comparisons
Two prompts answer the same questions, so their results are paired. A paired comparison subtracts, for each item, the verdict of the baseline from the verdict of prompt B, and averages the differences over the set. Its interval comes from the bootstrap over items.
Pairing matters because most items agree. An item that both prompts pass, or both fail, contributes nothing to the difference, whatever the question's difficulty. The comparison's spread comes from the items on which the two prompts split. An unpaired comparison takes two separate pass rates and sets them side by side. It carries the variation between questions on every item, including the ones that agree, and that variation can hide a real difference. The paired version cancels the part the two prompts share. The argument for paired differences in Miller (2024) says the same: the variance of a paired difference falls as long as the two models' scores on each item are positively correlated.
1"""Compares prompt-b (P2) with the baseline prompt (P1) on the eighty dev items: each pass rate, the paired difference with2its 95% interval, and how the items split between the two prompts. 'Better' is used only when the whole interval is above3zero (plan section 6.4); the interval here includes zero, so the reading is no difference detected."""4import lib # first: fixes the thread counts and seeds before anything else loads56from lib.calibration import readable_verdicts7from lib.judging import passed8from lib.prompts import REF_SHA2569from lib.stats import bootstrap_mean, interval_points, pct10from lib.testset import load_items1112BASELINE = "judge-answers-baseline"13PROMPT_B = "judge-answers-prompt-b"1415items = [item.id for item in load_items(split="dev")] # eighty dev items, sorted by id16p1_found, p1_unreadable = readable_verdicts(BASELINE, items, REF_SHA256)17p2_found, p2_unreadable = readable_verdicts(PROMPT_B, items, REF_SHA256)18for label, unreadable in (("P1 (baseline)", p1_unreadable), ("P2 (prompt-b)", p2_unreadable)):19 details = "".join(f' ({item_id}, "{first}")' for item_id, first in unreadable)20 print(f"unreadable verdicts, {label}: {len(unreadable)}{details}")21# An item without a readable verdict under either prompt is dropped from both sides: a paired difference needs the same items.22excluded = {item_id for item_id, _ in p1_unreadable + p2_unreadable}23kept = [item_id for item_id in items if item_id not in excluded]24first = passed({item_id: p1_found[item_id] for item_id in kept}) # 1.0 or 0.0 per item, in id order25second = passed({item_id: p2_found[item_id] for item_id in kept})2627n = len(kept)28lo, hi = bootstrap_mean(second - first) # 95%, one comparison: the default level29print(f"P1 (baseline) pass rate: {pct(int(first.sum()), n)}")30print(f"P2 (prompt-b) pass rate: {pct(int(second.sum()), n)}")31print(f"difference, P2 minus P1: {100 * float((second - first).mean()):+.1f} points")32print(f"95% interval of the difference: {interval_points(lo, hi)}")33print(f"paired items (n): {n}")34print(f"items only P1 passes: {int(((first == 1) & (second == 0)).sum())}")35print(f"items only P2 passes: {int(((first == 0) & (second == 1)).sum())}")36print(f"items both pass: {int(((first == 1) & (second == 1)).sum())}")37print(f"items neither passes: {int(((first == 0) & (second == 0)).sum())}")38if lo > 0:39 print(f"reading: P2 is better at n = {n}")40elif hi < 0:41 print(f"reading: P2 is worse at n = {n}")42else:43 print(f"reading: no difference detected at n = {n}")
python paired-prompt-b.pyunreadable verdicts, P1 (baseline): 1 (q-0043, "VERDICT: PARTIAL")
unreadable verdicts, P2 (prompt-b): 0
P1 (baseline) pass rate: 78.5% (62 of 79)
P2 (prompt-b) pass rate: 81.0% (64 of 79)
difference, P2 minus P1: +2.5 points
95% interval of the difference: [-3.8, +10.1] points
paired items (n): 79
items only P1 passes: 3
items only P2 passes: 5
items both pass: 59
items neither passes: 12
reading: no difference detected at n = 79
Here prompt B runs against the baseline on the dev items. An item with a verdict that can't be read under either prompt is left out of both sides, because a paired difference needs the same items on both sides. The printed lines give each pass rate, the difference, its interval and the number of paired items.
The baseline passes 78.5% (62 of 79). Prompt B passes 81.0% (64 of 79). The difference, prompt B minus the baseline, is +2.5 points, with a 95% interval of [-3.8, +10.1] points.
The split shows where the difference lives. 59 items pass under both prompts, and 12 fail under both. Only the items that split move the difference: 5 pass only under prompt B, and 3 pass only under the baseline.
The interval includes zero, so the reading is no difference detected at n = 79. The interval runs from a drop to a rise, and this set can't say which end is nearer. No difference detected is a finding about a set at its size. It is not a claim that the prompts are equal, and the text shouldn't say they are.
Working rule: Compare variants on the same items, paired, and report the difference with its interval and n. Call one variant better only when the whole interval sits above zero, and worse only when the whole interval sits below it. When the interval includes zero, report no difference detected at that n, and say how many items the difference would need, as How many questions a difference needs works out.
此部分尚未翻譯,因此以英文呈現。
Held-out, once
Dev items are for building. A team may tune a prompt on them, read the failures and change the next version. The dev results can't then prove that a change works, because the change was shaped by them. The held-out set is the check that isn't tuned. A comparison on held-out is run once, against rules fixed before the run: the variants, the level and the reading. Run it again after a change of mind, or choose the level after seeing the number, and the interval no longer means what it says.
This section runs three comparisons on the held-out set, each variant against the baseline. Prompt B is the one from Paired comparisons. The variant k5 retrieves five sections instead of three. The variant unfiltered retrieves from every version of the docs at once, with three sections, instead of only the asked version.
1"""Compares prompt-b, k5 and unfiltered with the baseline on the held-out items, each variant once, at the level the2Bonferroni rule gives for three comparisons (98.33%). Each variant gets its paired difference, interval, n and a reading by3plan section 6.4. Nothing was chosen with these numbers: the variants, the level and the reading rule were fixed before4this run, and the run happens once."""5import lib # first: fixes the thread counts and seeds before anything else loads67from lib.calibration import readable_verdicts8from lib.judging import passed9from lib.prompts import REF_SHA25610from lib.stats import RESAMPLES, bootstrap_mean, interval_points11from lib.testset import load_items1213BASELINE = "baseline"14VARIANTS = ("prompt-b", "k5", "unfiltered")15M = len(VARIANTS) # one comparison per variant, each against the baseline16LEVEL = 1 - 0.05 / M # Bonferroni: 1 - 0.05/m, which is 98.33% for m = 3 (plan section 6.3)1718items = [item.id for item in load_items(split="held-out")] # sorted by id19print(f"comparisons (m): {M}")20print(f"share of the 5% for each comparison, 0.05 / m: {0.05 / M:.4f}")21print(f"level of each interval, 1 - 0.05 / m: {100 * LEVEL:.2f}%")22print(f"resamples beyond each end of an interval: {RESAMPLES * (1 - LEVEL) / 2:.1f}")23base_found, base_unreadable = readable_verdicts(f"judge-answers-{BASELINE}", items, REF_SHA256)24details = "".join(f' ({item_id}, "{first}")' for item_id, first in base_unreadable)25print(f"unreadable verdicts, baseline: {len(base_unreadable)}{details}")26for name in VARIANTS:27 found, unreadable = readable_verdicts(f"judge-answers-{name}", items, REF_SHA256)28 details = "".join(f' ({item_id}, "{first}")' for item_id, first in unreadable)29 print(f"unreadable verdicts, {name}: {len(unreadable)}{details}")30 # An item unreadable under the baseline or this variant is dropped from both sides of this pair only.31 excluded = {item_id for item_id, _ in base_unreadable + unreadable}32 kept = [item_id for item_id in items if item_id not in excluded]33 first = passed({item_id: base_found[item_id] for item_id in kept})34 second = passed({item_id: found[item_id] for item_id in kept})35 n = len(kept)36 lo, hi = bootstrap_mean(second - first, level=LEVEL)37 if name == "prompt-b": # the chapter says this lower end is exactly zero, so the run must show it38 assert lo == 0.0, "the prompt-b lower bound must be exactly zero"39 print(f"{name} minus baseline, difference: {100 * float((second - first).mean()):+.1f} points")40 print(f"{name} minus baseline, {100 * LEVEL:.2f}% interval: {interval_points(lo, hi)}")41 print(f"{name} minus baseline, n: {n}")42 if lo > 0:43 print(f"{name} minus baseline, reading: better at n = {n}")44 elif hi < 0:45 print(f"{name} minus baseline, reading: worse at n = {n}")46 else:47 print(f"{name} minus baseline, reading: no difference detected at n = {n}")
python held-out-variants.pycomparisons (m): 3
share of the 5% for each comparison, 0.05 / m: 0.0167
level of each interval, 1 - 0.05 / m: 98.33%
resamples beyond each end of an interval: 16.7
unreadable verdicts, baseline: 0
unreadable verdicts, prompt-b: 0
prompt-b minus baseline, difference: +10.0 points
prompt-b minus baseline, 98.33% interval: [+0.0, +22.5] points
prompt-b minus baseline, n: 40
prompt-b minus baseline, reading: no difference detected at n = 40
unreadable verdicts, k5: 0
k5 minus baseline, difference: +2.5 points
k5 minus baseline, 98.33% interval: [-12.5, +18.4] points
k5 minus baseline, n: 40
k5 minus baseline, reading: no difference detected at n = 40
unreadable verdicts, unfiltered: 0
unfiltered minus baseline, difference: -10.0 points
unfiltered minus baseline, 98.33% interval: [-27.5, +8.4] points
unfiltered minus baseline, n: 40
unfiltered minus baseline, reading: no difference detected at n = 40
Three comparisons share one held-out set, so each interval is set at a higher level than 95%, and Many comparisons explains why. The level is 98.33%. Each variant gets a difference, an interval, a count of paired items and a reading, all printed.
Held-out has 40 items, against 79 on dev. So its intervals are wider for the same difference, and a result on it can only be as sharp as its item count allows. The split keeps a result honest, and its cost is fewer items to spend.
The difference for prompt B is +10.0 points, with an interval of [+0.0, +22.5] points. The comparison has 40 paired items.
The k5 difference is +2.5 points, with an interval of [-12.5, +18.4] points. The unfiltered difference is -10.0 points, with an interval of [-27.5, +8.4] points.
Dev gave +2.5 points for the same prompt, and the two point estimates differ. The 40 held-out items can't tell which is nearer the truth. The smallest difference that they detect, at 98.33% confidence and 80% power, is 16.2 points, so both point estimates sit below what this set can detect. How many questions a difference needs works out the arithmetic.
By the rule, all three read as no difference detected. For prompt B the printed reading is no difference detected at n = 40. For k5 it is no difference detected at n = 40, and for unfiltered no difference detected at n = 40.
Prompt B is the close case. Its interval is [+0.0, +22.5] points, and its lower end is exactly zero. The reading rule says better only when the whole interval sits above zero, and this one reaches zero without clearing it, so by the rule the reading is no difference detected. Beyond each end of the interval lie 16.7 of the 2,000 resamples, so the call sits on the line. The seed was fixed before the run for that reason. A reader could argue that a point estimate of that size should count for something, and this section doesn't make that argument. The rule was set before the run, the comparison runs once, and nobody gets to look at these three again with a lower level or a different reading. If prompt B is to be tested further, it needs a fresh held-out set, sized by How many questions a difference needs and fixed before anyone sees its results, read on its own. Adding items to the 40 already read, and rereading the pool, would be a second look at a result already seen.
Working rule: Fix the variants, the level and the reading rule before you open the held-out set, and run each comparison once. An interval that reaches zero is no difference detected at that n. Don't rerun the comparison, lower the level or change the reading to get a different answer. Add held-out items only before they are scored.
多重比較
每一次比較都有可能讓人被誤導。對於一項實際上沒有差異的比較,95% 的區間仍然會因運氣而排除零,大約二十次中會出現一次。若進行三項這樣的比較,並且每一項都以 95% 回報,則至少有一項被讀成有差異的機率會升到百分之五以上。一章回報的比較越多,其中某一項是僥倖結果的可能性就越高。
常見的補救方式是 Bonferroni 校正,它把錯誤預算分配給各項比較。對同一項指標進行 m 次比較時,每個區間的水準設為 1 減去 0.05 除以 m。以三次比較為例,0.05 除以 3 為 0.0167,因此每個區間的水準是 98.33%,也就是保留集一節所使用的水準。這個 0.05 要分配到每個區間的兩側尾端,所以每一側尾端分得 0.05 除以 2m。無論各項比較之間有什麼共同點,這個算式都成立。其中任何一項呈現假差異的機率,不會超過各自機率的總和,而這個總和是 5%。
程式碼會由 m 算出這個水準並印出來,讀者可以對照比較的次數自行檢查。代價是區間變寬。在這個水準下,每個區間都比 95% 的區間更寬,所以差異必須更大,其區間才會離開零。各自機率的總和是任一假差異發生機率的上界,因此只要各項比較的假結果彼此重疊,這種校正就是保守的。本書採用這個方法,是因為它的規則簡短、容易陳述與檢查,讀者可以由比較的次數自行重新計算。
哪些比較構成這個次數?在本章中,成對比較一節裡的開發集比較自成一項,它以 95% 回報,因為它是以調校資料進行的示範範例。三項保留集比較構成一組比較家族,並以調整後的水準回報。同時看到兩者的讀者,不應把開發集的區間當成保留集的結果,而本章所依據的是保留集的結果。
比較的次數是設計的一部分。在執行任何比較之前,先算出本章將對同一項指標回報的比較次數。在結果出爐之後才加入的比較,應與該族群分開回報,標示為探索性分析,並另行計數。該族群的區間維持原先執行時的樣子,因為在結果出爐後重新設定,會改變這次執行所受的規則。
**實用原則:**在執行之前,先算出對同一項指標的比較次數,並把每個區間的水準設為 1 減去 0.05 除以該次數。在每個區間旁標示其水準。在結果出爐之後才加入的比較,應分開回報為探索性分析,且不重新設定該族群的區間。
一個差異需要多少題目
進行比較之前,應先問:要看出團隊在意大小的差異,需要多少題目。答案取決於一個量:兩個變體在逐題上有多不一致。兩個提示詞若只在少數題目上不一致,只需少量題目就能顯示出較小的差異。 成對公式能把這件事說得具體。以 z 信賴 表示信賴水準對應的雙尾常態值,以 z 檢定力 表示檢定力對應的常態值。則 n = (z 信賴 + z 檢定力)² · d 的變異數 / δ²,其中 d 的變異數是單一題目成對差異的變異數,δ 是要偵測的差異。這個變異數等於兩個變體不一致的題目所占比例,減去它們平均差異的平方。因此這項估計取決於不一致的程度,而不是通過率本身。本節採用 95% 信賴水準與 80% 檢定力。
1"""How many items a paired comparison needs to detect a difference of 5 and of 10 points, with 95% confidence and 80% power.23The main estimate reads the paired differences that prompt-b and the baseline show on the dev items: each item's d is4+1, 0 or -1 (P2 minus P1), and the items on which they disagree set the variance. The floor uses only the judge's5within-item sampling noise over five samples; it leaves the disagreement between variants out, so it is a lower bound.6"""7import lib # first: fixes the thread counts and seeds before anything else loads89from lib import variance10from lib.answers import chapter_items11from lib.calibration import readable_verdicts12from lib.judging import paired, passed13from lib.prompts import REF_SHA25614from lib.stats import pct15from lib.testset import load_items1617# n = (z_alpha + z_beta)^2 * var_d / delta^2, rounded up (variance.items_needed). var_d = p_discordant - mean(d)^2.18# Smallest difference that n items detect: (z_alpha + z_beta) * sqrt(var_d / n) (variance.smallest_difference_points).19BASELINE, PROMPT_B = "judge-answers-baseline", "judge-answers-prompt-b"20CHAPTER_ONE_ITEMS = 20 # the chapter 1 set, read below the way twenty-cannot-tell reads it2122items = [item.id for item in load_items(split="dev")] # eighty dev items, sorted by id23p1_found, p1_unreadable = readable_verdicts(BASELINE, items, REF_SHA256)24p2_found, p2_unreadable = readable_verdicts(PROMPT_B, items, REF_SHA256)25kept = [item_id for item_id in items if item_id not in {i for i, _ in p1_unreadable + p2_unreadable}]26first = passed({item_id: p1_found[item_id] for item_id in kept}) # 1.0 or 0.0 per item27second = passed({item_id: p2_found[item_id] for item_id in kept})28var_d = variance.paired_difference_variance(first, second)29print(f"share of discordant items, P2 and P1 disagree: {pct(int((first != second).sum()), len(kept))}")30print(f"assumption, the next change disagrees with the baseline on about as many items as prompt-b did "31 f"({100 * (first != second).mean():.1f}% of items)")3233# Floor: five samples of a 0/1 verdict with k PASS give the unbiased within-item variance k(5 - k) / (5 * 4).34rows = [row for row in variance.item_pass_counts("judge-answers-variance", items) if row.complete]35noise = sum(row.passes * (5 - row.passes) / 20 for row in rows) / len(rows)36print(f"floor, sampling noise only: dev items with five readable verdicts (n): {len(rows)}")37print(f"floor, sampling noise only: mean within-item variance of one verdict: {noise:.4f}")3839# Two variants judged separately: their two noises add and the item's difficulty cancels, so the floor is 2 * noise.40for label, var in (("main estimate", var_d), ("floor, sampling noise only", 2 * noise)):41 print(f"{label}, variance of one item's paired difference: {var:.4f}")42 for delta in (5.0, 10.0):43 print(f"{label}, items needed for a {delta:.0f}-point difference: {variance.items_needed(var, delta)}")44 print(f"{label}, smallest difference that {CHAPTER_ONE_ITEMS} items detect, 80% power and 95% confidence: "45 f"{variance.smallest_difference_points(var, CHAPTER_ONE_ITEMS):.1f} points")4647# The chapter 1 set, read as twenty-cannot-tell reads it: its share of discordant items and its interval width.48chapter_one = paired(BASELINE, PROMPT_B, [item.id for item in chapter_items()])49width_20 = variance.interval_width_points(var_d, CHAPTER_ONE_ITEMS)50print(f"chapter 1 set, share of items where the two verdicts differ: "51 f"{pct(int((chapter_one.first != chapter_one.second).sum()), chapter_one.n)}")52print(f"chapter 1 set, 95% interval width: {100 * (chapter_one.diff_hi - chapter_one.diff_lo):.1f} points")53print(f"main estimate, 95% width at {CHAPTER_ONE_ITEMS} items: {width_20:.1f} points")54print(f"reading: with {CHAPTER_ONE_ITEMS} items and a change that disagrees with the baseline on "55 f"{100 * (first != second).mean():.1f}% of items, as prompt-b did, the smallest difference detected with 80% power "56 f"at 95% confidence is {variance.smallest_difference_points(var_d, CHAPTER_ONE_ITEMS):.1f} points")
python sample-size.pyshare of discordant items, P2 and P1 disagree: 10.1% (8 of 79)
assumption, the next change disagrees with the baseline on about as many items as prompt-b did (10.1% of items)
floor, sampling noise only: dev items with five readable verdicts (n): 79
floor, sampling noise only: mean within-item variance of one verdict: 0.0051
main estimate, variance of one item's paired difference: 0.1006
main estimate, items needed for a 5-point difference: 316
main estimate, items needed for a 10-point difference: 79
main estimate, smallest difference that 20 items detect, 80% power and 95% confidence: 19.9 points
floor, sampling noise only, variance of one item's paired difference: 0.0101
floor, sampling noise only, items needed for a 5-point difference: 32
floor, sampling noise only, items needed for a 10-point difference: 8
floor, sampling noise only, smallest difference that 20 items detect, 80% power and 95% confidence: 6.3 points
chapter 1 set, share of items where the two verdicts differ: 10.0% (2 of 20)
chapter 1 set, 95% interval width: 30.0 points
main estimate, 95% width at 20 items: 27.8 points
reading: with 20 items and a change that disagrees with the baseline on 10.1% of items, as prompt-b did, the smallest difference detected with 80% power at 95% confidence is 19.9 points
這段程式碼讀取提示詞 B 與基準在開發集題目上的不一致情形。它的假設已印出,也是需要檢查的一點:下一個變更與基準不一致的題目數,大約和提示詞 B 相當。在此,不一致題目的比例為 10.1% (8 of 79)。
1"""The arithmetic behind the five-point figure in sample-size, and the limits of its inputs: the interval on the share of2discordant items (a percentile bootstrap over items, as lib.stats computes every interval), the held-out level of 98.33%,3and the difference that the 40 held-out items can detect. The disagreement is the dev items' paired run, as in sample-size."""4import lib # first: fixes the thread counts and seeds before anything else loads56from lib import variance7from lib.calibration import readable_verdicts8from lib.judging import passed9from lib.prompts import REF_SHA25610from lib.stats import bootstrap_mean, interval_pct11from lib.testset import load_items1213BASELINE = "judge-answers-baseline"14PROMPT_B = "judge-answers-prompt-b"15ALPHA_HELD_OUT = 0.05 / 3 # two-sided alpha for each of the three held-out comparisons (Bonferroni)16HELD_OUT_LEVEL = 1 - ALPHA_HELD_OUT1718items = [item.id for item in load_items(split="dev")] # eighty dev items, sorted by id19p1_found, p1_unreadable = readable_verdicts(BASELINE, items, REF_SHA256)20p2_found, p2_unreadable = readable_verdicts(PROMPT_B, items, REF_SHA256)21kept = [item_id for item_id in items if item_id not in {i for i, _ in p1_unreadable + p2_unreadable}]22first = passed({item_id: p1_found[item_id] for item_id in kept})23second = passed({item_id: p2_found[item_id] for item_id in kept})24discordant = (first != second).astype(float) # 1 where the two prompts disagree on an item, else 025var_d = variance.paired_difference_variance(first, second)26mean_d = float((second - first).mean())2728# The five-point figure, worked: n = (z_alpha + z_beta)^2 * var(d) / delta^229z2, delta2 = variance.z_sum() ** 2, 0.05 ** 230print(f"z sum squared, 95% confidence and 80% power: {z2:.2f}")31print(f"var(d), variance of one item paired difference: {var_d:.4f}")32print(f"delta squared, a five-point difference: {delta2:.4f}")33print(f"n for five points: {z2:.2f} x {var_d:.4f} / {delta2:.4f} = {z2 * var_d / delta2:.1f}, "34 f"rounded up to {variance.items_needed(var_d, 5.0)}")3536# The share rests on the discordant items alone, so its 95% interval is a range for the answer.37share_lo, share_hi = bootstrap_mean(discordant)38print(f"discordant share, 95% interval: {interval_pct(share_lo, share_hi)}")39for end, share in (("low", share_lo), ("high", share_hi)):40 # mean(d) stays at its observed value; only the share moves41 print(f"items needed for 5 points, {end} end of the share: {variance.items_needed(share - mean_d ** 2, 5.0)}")4243print(f"items needed for 5 points at {100 * HELD_OUT_LEVEL:.2f}% confidence: "44 f"{variance.items_needed_at(var_d, 5.0, ALPHA_HELD_OUT)}")4546# Only the count is read here: no held-out verdict is opened by this example.47held_out_n = len(load_items(split="held-out"))48print(f"held-out items (n): {held_out_n}")49print(f"smallest difference {held_out_n} held-out items detect at {100 * HELD_OUT_LEVEL:.2f}% and 80% power: "50 f"{variance.smallest_difference_at(var_d, held_out_n, ALPHA_HELD_OUT):.1f} points")
python sample-size-limits.pyz sum squared, 95% confidence and 80% power: 7.85
var(d), variance of one item paired difference: 0.1006
delta squared, a five-point difference: 0.0025
n for five points: 7.85 x 0.1006 / 0.0025 = 315.9, rounded up to 316
discordant share, 95% interval: [3.8%, 17.7%]
items needed for 5 points, low end of the share: 118
items needed for 5 points, high end of the share: 555
items needed for 5 points at 98.33% confidence: 422
held-out items (n): 40
smallest difference 40 held-out items detect at 98.33% and 80% power: 16.2 points
該範例中進行了五個百分點差異的演算過程。兩個 z 值之和的平方為 7.85。單一題目成對差異的變異數為 0.1006。五個百分點差異的平方為 0.0025。因此 n 為 7.85 x 0.1006 / 0.0025 = 315.9, rounded up to 316。
Miller(2024)以更一般的形式給出成對公式,其中含有各題目內部雜訊的項。此處的公式是每題只有一則回覆的情形,因此省去了題目內部的項。依該文的輸入所算出的範例,約為 969 題。這些輸入描述的是另一項實驗,所以本書自己的數字只取自所印出的執行結果。
檢定力也有相反的一面。在 80% 檢定力下,即使題目集的大小是依真實差異而定,仍有約五分之一的機會看不出該差異。這是固定題目預算的代價;無法負擔此代價的團隊,應接受較大的差異作為目標,或在執行前擴充題目集。
平均差異很小時,所需題目數與該比例成正比,因此不一致題目多一倍的變更,約需多一倍的題目。團隊應根據開發集上的成對執行估計該比例並載明,當變更比測量時所用的變更更大時,應再次檢查。
主要估計指出,五個百分點的差異需要 316 題,十個百分點的差異需要 79 題。下限只使用每題內部的抽樣雜訊,得出 32 與 8。應把下限視為所需題數的下界,而不是計畫。它略去了變體之間的不一致,而這種不一致是真實存在的。兩個提示詞改變的是助理說什麼,而不只是它說得有多穩定。下限只適用於在溫度 0.7 下每題抽樣一則回覆、且變體與 P1 一樣穩定的比較。只有兩個題目為其雜訊有所貢獻,所以它建立在極少的題目上。只要變更的不一致程度超過單靠抽樣所能解釋的範圍,依下限擬定的計畫就會不足。
同一項估計也回答了第 1 章題目集的問題。以二十題而言,在上述假設的不一致程度下,此檢定在 80% 檢定力與 95% 信賴水準下能偵測到的最小差異為 19.9 points。
第 1 章的題目集在 10.0% (2 of 20) 的題目上出現不一致。
其區間寬度為 30.0 points。在相同的不一致程度下,公式給出 27.8 points,兩者相近。二十題範例呈現了該題目集。在這樣的不一致程度下,二十題通常會漏掉十個百分點的差異;想看出五個百分點差異的團隊,需要數百題,而不是數十題。
這項估計僅依據不一致的題目,因此它只是數量級,而不是確切的題數。該比例的 95% 區間為 [3.8%, 17.7%]。在這個區間內,五個百分點的差異在低端需要 118 題,在高端需要 555 題。公式是常態近似,而判讀規則使用自助法,在不一致題目很少時,兩者只是大致相符。在保留集的 98.33% 水準下,相同的輸入對五個百分點的差異需要 422 題。
在實際運作中,常見的錯誤是憑感覺決定題目集大小,再把結果當成決定來讀。想要五個百分點進步的團隊若只跑數十題,通常會得到未偵測到差異的結果,並可能把它當成該變更無效的定論。應在執行前把大小與不一致的假設一併寫下,並依此報告結果。在載明的 n 之下未偵測到差異,是關於該題目集的發現,而不是變更失敗。
實用原則: 比較之前,先在開發集上進行成對比較,以估計兩個變體多常不一致,並把該比例載明為假設。針對你在意的差異,依比較將被判讀的水準計算所需題數,並根據該比例的區間給出範圍。若題目集較小,應報告它無法偵測到這種大小的差異,且不要把其未偵測到差異解讀為對該變更的否定。下限只作為下界使用,而且只適用於對回覆進行抽樣的比較。
本章,於本版次中
版次 2026.11.3範例於 2026年11 月 21 通過測試本章 5 個範例 個範例中有 5 個通過
譯自英文書,版次 2026.11.3。