判断一次改动是否更好,以及需要多少个问题
阅读约需 31分钟 分钟,6节中含5个示例。
读完本章,你可以利用配对比较上的区间,判断助手两个版本之间的差异是否超出噪声。你还可以算出,对于给定大小的差异,需要多少个问题才能让测试显示出它。
第 1 章对助手做了一次度量,并给出其通过率的区间。随后,它在二十个条目上比较了两个提示词。这两者之差的区间宽度为 30.0 points。该章以一个承诺收尾:后面的章会算出,一个差异需要多少个条目才能显示出来。本章履行这一承诺。修改提示词的团队需要三个答案。通过率的变动,是否出于测试中的偶然性之外的原因?这次比较是否有能力显示出差异?要显示出团队所关心的那个差异,又需要多少个问题?
这里的每个数字都来自同一个助手模型 Qwen3-4B-Instruct-2507,以及同一个评判者,即以温度零运行的同一个模型。区间涵盖的是抽取条目所带来的波动。第一节度量的是对回复进行采样所带来的波动,不使用区间。区间并不涵盖评判者的偏差,这部分由第 7 章检验。阅读每个结果时,请记住这一点。
单个条目内的方差
把同一个问题问两遍,模型可能给出不同的回答。本章的各项比较,每个条目和每个提示只使用一条已记录的回复,因此每个判定都只依据一条回复。温度较高时,模型会进行采样,同一条目的判定可能在两次采样之间发生变化。这对比较很重要。两个提示之间的差异,如果小于一次采样到下一次采样之间的波动,就根本算不上差异,而只看一次运行得到的比率会掩盖这种波动。
方差集包含助手对每个开发条目的五次采样回复,采样温度为 0.7。评判器在温度为零时阅读每条回复。当一个条目的五个判定并不全部相同时,该条目即为翻转。统计翻转的数量,就能在做出任何改动之前,衡量判定在采样之间的波动幅度。
1"""The flip rate over the dev variance set: the share of items whose judge verdict differs across their five samples. The2printed line 'flip rate' rests on the per-item PASS counts, which are printed with it: how many items passed 0, 1, 2, 3, 43or 5 of 5 samples, and the mean pass rate. The temperature-0 baseline's pass rate over the same items is printed for4comparison. An item with an unreadable verdict is printed by id and left out, never counted as FAIL."""5import lib # first: fixes the thread counts and seeds before anything else loads67from lib.calibration import readable_verdicts8from lib.judging import passed9from lib.prompts import REF_SHA25610from lib.stats import pct11from lib.testset import load_items12from lib.variance import VARIANCE_SAMPLES, item_pass_counts1314VARIANCE = "judge-answers-variance"15BASELINE = "judge-answers-baseline"16SAMPLES = len(VARIANCE_SAMPLES)1718items = [item.id for item in load_items(split="dev")] # eighty dev items, sorted by id19counts = item_pass_counts(VARIANCE, items)20excluded = [row for row in counts if not row.complete]21kept = [row for row in counts if row.complete]22for row in excluded:23 samples = ", ".join(str(sample) for sample in row.unreadable)24 print(f"unreadable, {row.item_id}: samples {samples} (left out, not counted as FAIL)")25print(f"left out of the flip rate (unreadable verdict): {len(excluded)}")2627n = len(kept)28print(f"variance items compared (n): {n}")29print(f"flip rate: {pct(sum(1 for row in kept if row.flaky), n)}")30spread = [sum(1 for row in kept if row.passes == passes) for passes in range(SAMPLES + 1)]31assert sum(spread) == n # every kept item falls in exactly one PASS count32for passes, count in enumerate(spread):33 print(f"items with {passes} of {SAMPLES} PASS: {count}")34print(f"mean per-item pass rate across samples: {pct(sum(row.passes for row in kept), n * SAMPLES)}")3536found, baseline_unreadable = readable_verdicts(BASELINE, [row.item_id for row in kept], REF_SHA256)37assert not baseline_unreadable # q-0043 is already left out above, so no kept item is unreadable under the baseline38baseline = passed(found)39print(f"baseline pass rate, same items (temperature 0): {pct(int(baseline.sum()), len(baseline))}")
python flip-rate.pyunreadable, q-0043: samples 1, 4 (left out, not counted as FAIL)
left out of the flip rate (unreadable verdict): 1
variance items compared (n): 79
flip rate: 2.5% (2 of 79)
items with 0 of 5 PASS: 16
items with 1 of 5 PASS: 1
items with 2 of 5 PASS: 0
items with 3 of 5 PASS: 0
items with 4 of 5 PASS: 1
items with 5 of 5 PASS: 61
mean per-item pass rate across samples: 78.5% (310 of 395)
baseline pass rate, same items (temperature 0): 78.5% (62 of 79)
代码按条目统计评判器通过了其五次采样中的几次。判定无法读取的条目会被排除在外,不计为失败,输出中会指明被排除的条目。
大多数条目不会变动。61 个条目五次采样全部通过,16 个条目五次全部失败。在翻转的条目中,1 个只通过一次采样,1 个通过四次采样。
翻转率为 2.5% (2 of 79)。
所有采样的平均通过率为 78.5% (310 of 395)。基线为每个条目在温度为零时的一条回复,其通过率为 78.5% (62 of 79)。两者相等,两个翻转的条目向相反方向拉动。这一点对 P1 成立,即方差集所使用的提示,而别的提示可能不那么稳定。
这只是一个范围很窄的结果。含有更多临界问题的集合会翻转得更多,而且翻转率统计的是条目数,本身没有区间。翻转的条目数这样少,很难说明下一次改动会有多频繁地引起翻转。
翻转率只衡量在条目固定的情况下,对这个助手的回复进行采样所产生的离散程度。两个提示之间的比较,还会随每个提示答对的条目而变动。最后一节把这种采样离散程度用作下界,并且只用于对回复进行采样的比较:差异需要多少个问题。
工作准则: 在信任一次运行的比率之前,先在你实际发布所用的温度下运行方差集,并把翻转率与平均通过率放在一起阅读。翻转率说明采样会使一个提示的判定变动多大。只有当某项比较同样对回复进行采样时,它才能为该比较给出界限。
对题目的自助法
第 1 章给出了通过率的区间,并用它来解读一个小型数据集。本节说明这一方法,因为本章其余部分以三种方式使用它:用于一个比率,用于一个配对差值,以及用于一个必须通过更严格水平的差值。
该方法是百分位自助法。数据集中的题目只是团队本可以编写的所有问题中的一个样本,因此换一个数据集,得到的比率会略有不同。要估计这种差异有多大,可以从数据集中有放回地抽取题目,抽取的数量与数据集所含题目数相同,这样有些题目会出现两次,有些则一次也不出现。对这一次抽取计算统计量,重复抽取许多次,然后保留结果中间的 95%。自助法不增加任何数据。它重复使用数据集中已有的题目,并显示:如果数据集是同一类抽取的另一个样本,统计量会移动多远。
重采样的单位是题目,这是需要牢记的规则。对于比率,每道题贡献它的判定结果。对于配对差值,每道题贡献一个数:提示 B 的判定结果减去基线的判定结果,取值为正一、零或负一。对于方差数据集,一道题的五个样本要放在一起,因为它们回答的是同一个问题。若把它们拆开抽取,就会让它们看起来像五个相互独立的问题,得到的区间会过窄。Miller (2024)对成组相关的问题也指出了同一点。该论文是预印本,也是一份从业者的笔记,而不是经过评审的方法,因此本书引用它是为了其中的理由,而不是把它当作证明。
本书固定了各项设置,以便结果可以复现。该方法抽取 2,000 次重采样,种子为零,并在每次调用内部创建一个全新的随机数生成器。因此,同样的数据总会给出同样的区间,无论此前运行过什么。种子是一种选择,而不是数据的属性。换一个种子,区间端点会略有移动,这正是要把设置写下来的原因。
该方法还依赖于一个前提:数据集是问题的公平样本。第 1 章已论证过这一点,而任何区间都无法检验它。数据集较小时,区间较粗略。在这种规模的数据集上,百分位区间的端点只是初步估计,应按近似值来解读。
工作准则: 对题目重采样,绝不在一道题内部对各次运行重采样,并让一道题的各个样本保持在一起。在代码中固定重采样次数和种子,使同样的数据给出同样的区间。当数据集所含题目很少时,称该区间为粗略区间,并在旁边注明。
配对比较
两个提示词回答同样的问题,因此它们的结果是配对的。配对比较对每道题用提示词 B 的判定减去基线的判定,再对整个集合取这些差值的平均。其区间来自对题目的自助法(bootstrap)。
配对之所以重要,是因为大多数题目上两者的结果一致。一道题无论两个提示词都通过还是都失败,无论题目难度如何,对差值都没有贡献。比较的离散程度来自两个提示词结果出现分歧的那些题目。非配对比较取两个各自独立的通过率,将它们并排放置。它在每一道题上都带着题目之间的差异,包括结果一致的那些题目,而这种差异可能掩盖真实存在的差别。配对的做法则抵消了两个提示词共有的那一部分。Miller (2024) 中为配对差值所做的论证也是同样的意思:只要两个模型在每道题上的得分正相关,配对差值的方差就会下降。
1"""Compares prompt-b (P2) with the baseline prompt (P1) on the eighty dev items: each pass rate, the paired difference with2its 95% interval, and how the items split between the two prompts. 'Better' is used only when the whole interval is above3zero (plan section 6.4); the interval here includes zero, so the reading is no difference detected."""4import lib # first: fixes the thread counts and seeds before anything else loads56from lib.calibration import readable_verdicts7from lib.judging import passed8from lib.prompts import REF_SHA2569from lib.stats import bootstrap_mean, interval_points, pct10from lib.testset import load_items1112BASELINE = "judge-answers-baseline"13PROMPT_B = "judge-answers-prompt-b"1415items = [item.id for item in load_items(split="dev")] # eighty dev items, sorted by id16p1_found, p1_unreadable = readable_verdicts(BASELINE, items, REF_SHA256)17p2_found, p2_unreadable = readable_verdicts(PROMPT_B, items, REF_SHA256)18for label, unreadable in (("P1 (baseline)", p1_unreadable), ("P2 (prompt-b)", p2_unreadable)):19 details = "".join(f' ({item_id}, "{first}")' for item_id, first in unreadable)20 print(f"unreadable verdicts, {label}: {len(unreadable)}{details}")21# An item without a readable verdict under either prompt is dropped from both sides: a paired difference needs the same items.22excluded = {item_id for item_id, _ in p1_unreadable + p2_unreadable}23kept = [item_id for item_id in items if item_id not in excluded]24first = passed({item_id: p1_found[item_id] for item_id in kept}) # 1.0 or 0.0 per item, in id order25second = passed({item_id: p2_found[item_id] for item_id in kept})2627n = len(kept)28lo, hi = bootstrap_mean(second - first) # 95%, one comparison: the default level29print(f"P1 (baseline) pass rate: {pct(int(first.sum()), n)}")30print(f"P2 (prompt-b) pass rate: {pct(int(second.sum()), n)}")31print(f"difference, P2 minus P1: {100 * float((second - first).mean()):+.1f} points")32print(f"95% interval of the difference: {interval_points(lo, hi)}")33print(f"paired items (n): {n}")34print(f"items only P1 passes: {int(((first == 1) & (second == 0)).sum())}")35print(f"items only P2 passes: {int(((first == 0) & (second == 1)).sum())}")36print(f"items both pass: {int(((first == 1) & (second == 1)).sum())}")37print(f"items neither passes: {int(((first == 0) & (second == 0)).sum())}")38if lo > 0:39 print(f"reading: P2 is better at n = {n}")40elif hi < 0:41 print(f"reading: P2 is worse at n = {n}")42else:43 print(f"reading: no difference detected at n = {n}")
python paired-prompt-b.pyunreadable verdicts, P1 (baseline): 1 (q-0043, "VERDICT: PARTIAL")
unreadable verdicts, P2 (prompt-b): 0
P1 (baseline) pass rate: 78.5% (62 of 79)
P2 (prompt-b) pass rate: 81.0% (64 of 79)
difference, P2 minus P1: +2.5 points
95% interval of the difference: [-3.8, +10.1] points
paired items (n): 79
items only P1 passes: 3
items only P2 passes: 5
items both pass: 59
items neither passes: 12
reading: no difference detected at n = 79
这里,提示词 B 在开发集题目上与基线对照运行。若某道题在任一提示词下的判定无法读取,则该题从两侧同时剔除,因为配对差值要求两侧使用相同的题目。打印出的各行给出每个通过率、差值、差值的区间以及配对题目的数量。
基线通过 78.5% (62 of 79)。提示词 B 通过 81.0% (64 of 79)。差值(提示词 B 减去基线)为 +2.5 points,95% 区间为 [-3.8, +10.1] points。
题目的分布显示了差值来自哪里。有 59 道题在两个提示词下都通过,有 12 道题在两个提示词下都失败。只有结果出现分歧的题目才会推动差值:有 5 道题只在提示词 B 下通过,有 3 道题只在基线下通过。
区间包含零,所以结论是 no difference detected at n = 79。区间从下降延伸到上升,这个集合无法说明哪一端更接近真实情况。未检测到差异,是关于一个特定规模的集合的发现,并不是断言两个提示词相等,文字中也不应这样表述。
工作准则: 在相同的题目上对各个变体做配对比较,并连同区间和 n 一起报告差值。只有整个区间都在零之上时,才称某个变体更好;只有整个区间都在零之下时,才称其更差。当区间包含零时,报告在该 n 下未检测到差异,并说明要检出这一差值需要多少题目,如一个差值需要多少道题所推算的那样。
留出集,只用一次
开发集的条目用于构建。团队可以在这些条目上调整提示词,阅读失败案例,再修改下一版。但开发集上的结果无法证明某项改动有效,因为这项改动本身就是由这些结果塑造出来的。留出集是不经调整的检验。在留出集上做比较只运行一次,所依据的规则在运行之前就已固定:变体、水平和读数方式。如果改变主意后重新运行,或者在看到数字之后才选定水平,区间就不再具有它所声称的含义。
本节在留出集上运行三项比较,每个变体都与基线相比。提示词 B 即配对比较中的那一个。变体 k5 检索五个节,而不是三个。变体 unfiltered 同时从文档的所有版本中检索,取三个节,而不是只检索被询问的那个版本。
1"""Compares prompt-b, k5 and unfiltered with the baseline on the held-out items, each variant once, at the level the2Bonferroni rule gives for three comparisons (98.33%). Each variant gets its paired difference, interval, n and a reading by3plan section 6.4. Nothing was chosen with these numbers: the variants, the level and the reading rule were fixed before4this run, and the run happens once."""5import lib # first: fixes the thread counts and seeds before anything else loads67from lib.calibration import readable_verdicts8from lib.judging import passed9from lib.prompts import REF_SHA25610from lib.stats import RESAMPLES, bootstrap_mean, interval_points11from lib.testset import load_items1213BASELINE = "baseline"14VARIANTS = ("prompt-b", "k5", "unfiltered")15M = len(VARIANTS) # one comparison per variant, each against the baseline16LEVEL = 1 - 0.05 / M # Bonferroni: 1 - 0.05/m, which is 98.33% for m = 3 (plan section 6.3)1718items = [item.id for item in load_items(split="held-out")] # sorted by id19print(f"comparisons (m): {M}")20print(f"share of the 5% for each comparison, 0.05 / m: {0.05 / M:.4f}")21print(f"level of each interval, 1 - 0.05 / m: {100 * LEVEL:.2f}%")22print(f"resamples beyond each end of an interval: {RESAMPLES * (1 - LEVEL) / 2:.1f}")23base_found, base_unreadable = readable_verdicts(f"judge-answers-{BASELINE}", items, REF_SHA256)24details = "".join(f' ({item_id}, "{first}")' for item_id, first in base_unreadable)25print(f"unreadable verdicts, baseline: {len(base_unreadable)}{details}")26for name in VARIANTS:27 found, unreadable = readable_verdicts(f"judge-answers-{name}", items, REF_SHA256)28 details = "".join(f' ({item_id}, "{first}")' for item_id, first in unreadable)29 print(f"unreadable verdicts, {name}: {len(unreadable)}{details}")30 # An item unreadable under the baseline or this variant is dropped from both sides of this pair only.31 excluded = {item_id for item_id, _ in base_unreadable + unreadable}32 kept = [item_id for item_id in items if item_id not in excluded]33 first = passed({item_id: base_found[item_id] for item_id in kept})34 second = passed({item_id: found[item_id] for item_id in kept})35 n = len(kept)36 lo, hi = bootstrap_mean(second - first, level=LEVEL)37 if name == "prompt-b": # the chapter says this lower end is exactly zero, so the run must show it38 assert lo == 0.0, "the prompt-b lower bound must be exactly zero"39 print(f"{name} minus baseline, difference: {100 * float((second - first).mean()):+.1f} points")40 print(f"{name} minus baseline, {100 * LEVEL:.2f}% interval: {interval_points(lo, hi)}")41 print(f"{name} minus baseline, n: {n}")42 if lo > 0:43 print(f"{name} minus baseline, reading: better at n = {n}")44 elif hi < 0:45 print(f"{name} minus baseline, reading: worse at n = {n}")46 else:47 print(f"{name} minus baseline, reading: no difference detected at n = {n}")
python held-out-variants.pycomparisons (m): 3
share of the 5% for each comparison, 0.05 / m: 0.0167
level of each interval, 1 - 0.05 / m: 98.33%
resamples beyond each end of an interval: 16.7
unreadable verdicts, baseline: 0
unreadable verdicts, prompt-b: 0
prompt-b minus baseline, difference: +10.0 points
prompt-b minus baseline, 98.33% interval: [+0.0, +22.5] points
prompt-b minus baseline, n: 40
prompt-b minus baseline, reading: no difference detected at n = 40
unreadable verdicts, k5: 0
k5 minus baseline, difference: +2.5 points
k5 minus baseline, 98.33% interval: [-12.5, +18.4] points
k5 minus baseline, n: 40
k5 minus baseline, reading: no difference detected at n = 40
unreadable verdicts, unfiltered: 0
unfiltered minus baseline, difference: -10.0 points
unfiltered minus baseline, 98.33% interval: [-27.5, +8.4] points
unfiltered minus baseline, n: 40
unfiltered minus baseline, reading: no difference detected at n = 40
三项比较共用同一个留出集,因此每个区间的水平都要高于 95%,原因见多重比较。该水平为 98.33%。每个变体都会得到一个差值、一个区间、一个配对条目数和一个读数,均已打印。
留出集有 40 个条目,而开发集有 79 个。因此,在差值相同的情况下,留出集上的区间更宽,其上的结果也只能达到条目数所允许的精度。这样划分使结果保持诚实,代价是可供使用的条目更少。
提示词 B 的差值为 +10.0 points,区间为 [+0.0, +22.5] points。该比较有 40 个配对条目。
k5 的差值为 +2.5 points,区间为 [-12.5, +18.4] points。unfiltered 的差值为 -10.0 points,区间为 [-27.5, +8.4] points。
同一个提示词在开发集上给出的是 +2.5 points,两个点估计并不相同。40 个留出集条目无法判断哪一个更接近真实值。在 98.33% 置信度和 80% 检验功效下,这些条目能够检测到的最小差值为 16.2 points,所以两个点估计都低于这个集合所能检测到的程度。一个差值需要多少道题给出了相关的计算。
按规则,三者的读数都是未检测到差异。提示词 B 打印出的读数为 no difference detected at n = 40。k5 为 no difference detected at n = 40,unfiltered 为 no difference detected at n = 40。
提示词 B 是临界的情形。它的区间为 [+0.0, +22.5] points,下端恰好为零。读数规则规定,只有整个区间都在零之上才算更好,而这个区间触及零却没有越过它,因此按规则,读数是未检测到差异。在区间每一端之外,各有 2,000 次重抽样中的 16.7 个,所以这一判定正处于临界线上。种子在运行之前就已固定,原因正在于此。有人可能会主张,如此大小的点估计应当有一定分量,但本节不提出这样的主张。规则在运行之前就已设定,比较只运行一次,没有人可以再用更低的水平或不同的读数方式重新审视这三项。如果要对提示词 B 做进一步检验,就需要一个新的留出集,其规模由一个差值需要多少道题确定,并在任何人看到其结果之前固定下来,单独读取。在已经读过的 40 个条目上追加条目,再重读整个条目池,就等于对已经看过的结果再看一次。
**工作准则:**在打开留出集之前,先固定变体、水平和读数规则,每项比较只运行一次。触及零的区间,就是在该 n 下未检测到差异。不要重新运行比较,不要降低水平,也不要改变读数方式以求得另一个答案。只有在留出集条目被打分之前,才可以追加。
多重比较
每一次比较都是一次被误导的机会。对于一个并无真实差异的比较,95% 区间仍会因运气而排除零,概率约为二十分之一。若进行三次这样的比较,并各以 95% 报告,则至少有一次读作存在差异的概率会升至百分之五以上。一章中报告的比较越多,其中某一个是侥幸结果的可能性就越大。
通常的补救办法是 Bonferroni 校正,它在各次比较之间分配误差预算。对同一指标做 m 次比较时,每个区间的水平设为 1 减去 0.05 除以 m。对于三次比较,0.05 除以 3 为 0.0167,因此每个区间的水平为 98.33%,这也是留出集一节所用的水平。这 0.05 要分摊到每个区间的两侧尾部,所以每侧尾部得到 0.05 除以 2m。无论这些比较有何共同之处,这一算术都成立。其中任何一个(至少一个)出现虚假差异的概率,不会超过各自单独概率之和,而这个和是 5%。
代码根据 m 计算该水平并将其打印出来,读者可以对照比较的次数加以核对。代价是区间变宽。在这一水平下,每个区间都比 95% 的区间更宽,因此差异需要更大,其区间才能越过零。各自单独概率之和是出现任何虚假差异的概率的上界,所以只要各次比较的虚假结果相互重叠,这一校正就是保守的。本书采用这种方法,是因为它的规则易于陈述和核对,读者可以根据比较的次数自行重新计算。
哪些比较构成这个计数?在本章中,配对一节里的开发集比较是单独存在的,它按 95% 报告,因为它是基于调参数据的演示示例。三个留出集比较构成一个族,按调整后的水平报告。同时看到两者的读者,不应把开发集的区间当作留出集的结果,而本章所依据的是留出集的结果。
计数是设计的一部分。在运行任何比较之前,先数出本章将要报告的同一指标的比较次数。结果出来之后才加入的比较,要与该族分开报告,标明为探索性比较,并有其自己的计数。该族的区间保持运行时的原样,因为在结果出来之后重新设定,就改变了这次运行所遵循的规则。
工作准则: 运行之前先数出同一指标的比较次数,并把每个区间的水平设为 1 减去 0.05 除以该次数。在每个区间旁标明其水平。结果出来之后才加入的比较,要分开报告并标明为探索性比较,该族的区间不再重新设定。
一个差异需要多少道题
在做比较之前,应当先问:要显示出团队所关心大小的差异,需要多少个条目。答案取决于一个量:两个变体在逐条目上的分歧有多大。仅在少数条目上存在分歧的两个提示词,只需很少的条目就能显示出较小的差异。
配对公式使这一点具体化。令 z-conf 为置信水平对应的双侧正态值,z-power 为检验功效对应的正态值。则 n = (z-conf + z-power)² · var(d) / δ²,其中 var(d) 是单个条目配对差异的方差,δ 是要检测的差异。方差等于两个变体产生分歧的条目所占的比例,减去二者平均差异的平方。因此,这一估计由分歧驱动,而不是由通过率本身驱动。本节采用 95% 的置信度和 80% 的功效。
1"""How many items a paired comparison needs to detect a difference of 5 and of 10 points, with 95% confidence and 80% power.23The main estimate reads the paired differences that prompt-b and the baseline show on the dev items: each item's d is4+1, 0 or -1 (P2 minus P1), and the items on which they disagree set the variance. The floor uses only the judge's5within-item sampling noise over five samples; it leaves the disagreement between variants out, so it is a lower bound.6"""7import lib # first: fixes the thread counts and seeds before anything else loads89from lib import variance10from lib.answers import chapter_items11from lib.calibration import readable_verdicts12from lib.judging import paired, passed13from lib.prompts import REF_SHA25614from lib.stats import pct15from lib.testset import load_items1617# n = (z_alpha + z_beta)^2 * var_d / delta^2, rounded up (variance.items_needed). var_d = p_discordant - mean(d)^2.18# Smallest difference that n items detect: (z_alpha + z_beta) * sqrt(var_d / n) (variance.smallest_difference_points).19BASELINE, PROMPT_B = "judge-answers-baseline", "judge-answers-prompt-b"20CHAPTER_ONE_ITEMS = 20 # the chapter 1 set, read below the way twenty-cannot-tell reads it2122items = [item.id for item in load_items(split="dev")] # eighty dev items, sorted by id23p1_found, p1_unreadable = readable_verdicts(BASELINE, items, REF_SHA256)24p2_found, p2_unreadable = readable_verdicts(PROMPT_B, items, REF_SHA256)25kept = [item_id for item_id in items if item_id not in {i for i, _ in p1_unreadable + p2_unreadable}]26first = passed({item_id: p1_found[item_id] for item_id in kept}) # 1.0 or 0.0 per item27second = passed({item_id: p2_found[item_id] for item_id in kept})28var_d = variance.paired_difference_variance(first, second)29print(f"share of discordant items, P2 and P1 disagree: {pct(int((first != second).sum()), len(kept))}")30print(f"assumption, the next change disagrees with the baseline on about as many items as prompt-b did "31 f"({100 * (first != second).mean():.1f}% of items)")3233# Floor: five samples of a 0/1 verdict with k PASS give the unbiased within-item variance k(5 - k) / (5 * 4).34rows = [row for row in variance.item_pass_counts("judge-answers-variance", items) if row.complete]35noise = sum(row.passes * (5 - row.passes) / 20 for row in rows) / len(rows)36print(f"floor, sampling noise only: dev items with five readable verdicts (n): {len(rows)}")37print(f"floor, sampling noise only: mean within-item variance of one verdict: {noise:.4f}")3839# Two variants judged separately: their two noises add and the item's difficulty cancels, so the floor is 2 * noise.40for label, var in (("main estimate", var_d), ("floor, sampling noise only", 2 * noise)):41 print(f"{label}, variance of one item's paired difference: {var:.4f}")42 for delta in (5.0, 10.0):43 print(f"{label}, items needed for a {delta:.0f}-point difference: {variance.items_needed(var, delta)}")44 print(f"{label}, smallest difference that {CHAPTER_ONE_ITEMS} items detect, 80% power and 95% confidence: "45 f"{variance.smallest_difference_points(var, CHAPTER_ONE_ITEMS):.1f} points")4647# The chapter 1 set, read as twenty-cannot-tell reads it: its share of discordant items and its interval width.48chapter_one = paired(BASELINE, PROMPT_B, [item.id for item in chapter_items()])49width_20 = variance.interval_width_points(var_d, CHAPTER_ONE_ITEMS)50print(f"chapter 1 set, share of items where the two verdicts differ: "51 f"{pct(int((chapter_one.first != chapter_one.second).sum()), chapter_one.n)}")52print(f"chapter 1 set, 95% interval width: {100 * (chapter_one.diff_hi - chapter_one.diff_lo):.1f} points")53print(f"main estimate, 95% width at {CHAPTER_ONE_ITEMS} items: {width_20:.1f} points")54print(f"reading: with {CHAPTER_ONE_ITEMS} items and a change that disagrees with the baseline on "55 f"{100 * (first != second).mean():.1f}% of items, as prompt-b did, the smallest difference detected with 80% power "56 f"at 95% confidence is {variance.smallest_difference_points(var_d, CHAPTER_ONE_ITEMS):.1f} points")
python sample-size.pyshare of discordant items, P2 and P1 disagree: 10.1% (8 of 79)
assumption, the next change disagrees with the baseline on about as many items as prompt-b did (10.1% of items)
floor, sampling noise only: dev items with five readable verdicts (n): 79
floor, sampling noise only: mean within-item variance of one verdict: 0.0051
main estimate, variance of one item's paired difference: 0.1006
main estimate, items needed for a 5-point difference: 316
main estimate, items needed for a 10-point difference: 79
main estimate, smallest difference that 20 items detect, 80% power and 95% confidence: 19.9 points
floor, sampling noise only, variance of one item's paired difference: 0.0101
floor, sampling noise only, items needed for a 5-point difference: 32
floor, sampling noise only, items needed for a 10-point difference: 8
floor, sampling noise only, smallest difference that 20 items detect, 80% power and 95% confidence: 6.3 points
chapter 1 set, share of items where the two verdicts differ: 10.0% (2 of 20)
chapter 1 set, 95% interval width: 30.0 points
main estimate, 95% width at 20 items: 27.8 points
reading: with 20 items and a change that disagrees with the baseline on 10.1% of items, as prompt-b did, the smallest difference detected with 80% power at 95% confidence is 19.9 points
这里的代码读取提示词 B 与基线在开发集条目上的分歧。它的假设已被打印出来,也是需要核对的一点:下一次改动与基线产生分歧的条目数,大致与提示词 B 相当。此处产生分歧的条目所占比例为 10.1% (8 of 79)。
1"""The arithmetic behind the five-point figure in sample-size, and the limits of its inputs: the interval on the share of2discordant items (a percentile bootstrap over items, as lib.stats computes every interval), the held-out level of 98.33%,3and the difference that the 40 held-out items can detect. The disagreement is the dev items' paired run, as in sample-size."""4import lib # first: fixes the thread counts and seeds before anything else loads56from lib import variance7from lib.calibration import readable_verdicts8from lib.judging import passed9from lib.prompts import REF_SHA25610from lib.stats import bootstrap_mean, interval_pct11from lib.testset import load_items1213BASELINE = "judge-answers-baseline"14PROMPT_B = "judge-answers-prompt-b"15ALPHA_HELD_OUT = 0.05 / 3 # two-sided alpha for each of the three held-out comparisons (Bonferroni)16HELD_OUT_LEVEL = 1 - ALPHA_HELD_OUT1718items = [item.id for item in load_items(split="dev")] # eighty dev items, sorted by id19p1_found, p1_unreadable = readable_verdicts(BASELINE, items, REF_SHA256)20p2_found, p2_unreadable = readable_verdicts(PROMPT_B, items, REF_SHA256)21kept = [item_id for item_id in items if item_id not in {i for i, _ in p1_unreadable + p2_unreadable}]22first = passed({item_id: p1_found[item_id] for item_id in kept})23second = passed({item_id: p2_found[item_id] for item_id in kept})24discordant = (first != second).astype(float) # 1 where the two prompts disagree on an item, else 025var_d = variance.paired_difference_variance(first, second)26mean_d = float((second - first).mean())2728# The five-point figure, worked: n = (z_alpha + z_beta)^2 * var(d) / delta^229z2, delta2 = variance.z_sum() ** 2, 0.05 ** 230print(f"z sum squared, 95% confidence and 80% power: {z2:.2f}")31print(f"var(d), variance of one item paired difference: {var_d:.4f}")32print(f"delta squared, a five-point difference: {delta2:.4f}")33print(f"n for five points: {z2:.2f} x {var_d:.4f} / {delta2:.4f} = {z2 * var_d / delta2:.1f}, "34 f"rounded up to {variance.items_needed(var_d, 5.0)}")3536# The share rests on the discordant items alone, so its 95% interval is a range for the answer.37share_lo, share_hi = bootstrap_mean(discordant)38print(f"discordant share, 95% interval: {interval_pct(share_lo, share_hi)}")39for end, share in (("low", share_lo), ("high", share_hi)):40 # mean(d) stays at its observed value; only the share moves41 print(f"items needed for 5 points, {end} end of the share: {variance.items_needed(share - mean_d ** 2, 5.0)}")4243print(f"items needed for 5 points at {100 * HELD_OUT_LEVEL:.2f}% confidence: "44 f"{variance.items_needed_at(var_d, 5.0, ALPHA_HELD_OUT)}")4546# Only the count is read here: no held-out verdict is opened by this example.47held_out_n = len(load_items(split="held-out"))48print(f"held-out items (n): {held_out_n}")49print(f"smallest difference {held_out_n} held-out items detect at {100 * HELD_OUT_LEVEL:.2f}% and 80% power: "50 f"{variance.smallest_difference_at(var_d, held_out_n, ALPHA_HELD_OUT):.1f} points")
python sample-size-limits.pyz sum squared, 95% confidence and 80% power: 7.85
var(d), variance of one item paired difference: 0.1006
delta squared, a five-point difference: 0.0025
n for five points: 7.85 x 0.1006 / 0.0025 = 315.9, rounded up to 316
discordant share, 95% interval: [3.8%, 17.7%]
items needed for 5 points, low end of the share: 118
items needed for 5 points, high end of the share: 555
items needed for 5 points at 98.33% confidence: 422
held-out items (n): 40
smallest difference 40 held-out items detect at 98.33% and 80% power: 16.2 points
五个百分点差异的具体算术过程在该示例中给出。两个 z 值之和的平方为 7.85。单个条目配对差异的方差为 0.1006。五个百分点差异的平方为 0.0025。因此 n 为 7.85 x 0.1006 / 0.0025 = 315.9, rounded up to 316。
Miller(2024)以更一般的形式给出了配对公式,其中含有每个条目内部噪声的项。这里的公式就是该形式在每个条目只有一条回复时的特例,条目内部的项被舍去。按该文的输入代入,其算例约为 969 个条目。这些输入描述的是另一项实验,因此本书自己的数字只来自打印出的那次运行。
功效也反过来起作用。在 80% 功效下,为某个真实差异而确定规模的集合,仍有大约五分之一的机会漏掉它。这是固定题量预算所付出的代价;付不起这一代价的团队,要么接受把更大的差异作为目标,要么在运行前扩充集合。
平均差异较小时,所需条目数与该比例成正比,因此在两倍数量的条目上产生分歧的改动,大约需要两倍的题量。团队应当通过开发集上的一次配对运行来估计该比例,将其明确写出,并在改动比测量时的那次更大时重新核对。
主估计表明,五个百分点的差异需要 316 个条目,十个百分点的差异需要 79 个。下限只使用每个条目内部的抽样噪声,给出的是 32 和 8。应将下限视为所需条目数的下界,而不是计划。它略去了变体之间的分歧,而这种分歧是真实存在的。两个提示词改变的是助手说什么,而不只是它说得有多稳定。下限只适用于这样的比较:每个条目在温度 0.7 下只采样一条回复,且各变体像 P1 一样稳定。只有两个条目会增加它的噪声,因此它建立在极少的条目之上。只要某个改动的分歧超出单靠抽样所能解释的范围,按下限制定的计划就会不够用。
同一个估计也回答了第 1 章那个集合的问题。在二十个条目下,以 80% 功效、95% 置信度检验所能检测到的最小差异为 19.9 points,这是在上文所假设的分歧之下。
第 1 章的集合在其 10.0% (2 of 20) 的条目上存在分歧。
其区间宽度为 30.0 points。公式在相同分歧下给出 27.8 points,二者相近。二十个条目的示例展示了该集合。在这样的分歧下,二十个条目通常会漏掉十个百分点的差异;而想看到五个百分点差异的团队,需要的是数百个条目,而不是几十个。
该估计只依据分歧条目,因此它给出的是数量级,而不是确切的计数。其对该比例的 95% 区间为 [3.8%, 17.7%]。在这一区间内,五个百分点的差异在低端需要 118 个条目,在高端需要 555 个。公式是正态近似,而读数规则使用的是自助法(bootstrap),当产生分歧的条目很少时,二者只是大致吻合。在留出集的 98.33% 水平上,相同的输入对五个百分点的差异需要 422 个条目。
在生产环境中,常见的错误是凭感觉选定集合规模,然后把结果当作决定来读。想要五个百分点提升的团队只运行几十个条目,通常会得到未检测到差异的结果,并可能把它当作该改动无效的裁决。应在运行前把规模连同分歧假设一并写下,并据此报告结果。在已声明的 n 下未检测到差异,是关于该集合的一项发现,而不是改动失败。
工作准则: 在比较之前,先在开发集上做一次配对比较,以估计两个变体产生分歧的频率,并把该比例作为假设明确写出。针对你所关心的差异,在比较将被解读的水平上计算所需的条目数,并根据该比例的区间给出其范围。如果集合更小,就报告它无法检测到这样大小的差异,不要把它的未检测到差异读成对该改动的否定。仅把下限当作下界,并且只用于对回复进行抽样的比较。
本章,在本版次中
版次 2026.11.3示例于 2026年11月21日 通过本章 5个示例 个示例中有 5 个通过
译自英文版图书,版次 2026.11.3。