找出评判器的判定在何处不再可靠
阅读约需 38分钟 分钟,4节中含3个示例。
读完本章,你能够检验评判器是否存在本书数据所能显示的两种偏差,即位置偏差和长度偏差。你能够说明每项检验显示了什么、又留下了什么未决的问题,并能够写下在何种条件下你不再信任它的某一条判定。
第 6 章用人工评分度量了评判器的错误,并在结尾提出一个问题:当成对评判器所看到的两条回复互换位置时,它会怎样表现。本章要问的是,在评判器所评的 sablefile 回复中,还有什么会改变一条判定。评判器读取对某个测试问题的回复,将其与参考答案对照,然后输出 PASS 或 FAIL,其设定见第 5 章。它的通过率就是团队会拿来汇报的那个数字,而只有当判定取决于回复本身时,这个数字才值得汇报。如果两个答案互换位置后判定随之改变,或者通过率随回复变长而上升,而回复所说的内容并无增加,那么这样的判定度量的就是答案以外的东西。建立在这种判定之上的通过率,可能看起来像是进步,实际上并无进步。
有两点提醒适用于全章,因此在此一次说明。第一点,评判器是 Qwen3-4B-Instruct-2507,本章的每一个比率都是在该模型上测得的。本章引用的已发表数字来自其他评判器,它们说明的是一个评判器可能发生什么,而不是这个评判器实际会怎样。第二点涉及规模。人工评分的成对样本和开发集条目都是较小的集合,因此每个比率都附有其计数,每项比较都附有区间。
本章依次讨论三个局限。第一是位置:成对判定是否取决于哪个答案先被展示。第二是长度:通过率是否随回复长度而上升。第三是自我偏好:评判器是否偏向由其自身模型写出的答案。前两项,本书可以在自己的记录上检验,因为这些记录包含检验所需的内容:每一对人工评分的样本都在两种顺序下被评判过,每一条回复的长度都可以计数。第三项则无法检验,相关一节会说明原因。最后一节把这三个局限转化为读者可以应用的规则,每条规则都与一个计数相联系。
位置
请评审者判断两条回复哪一条更好,它的回答可能取决于它先读到哪一条回复。如果一个团队只按一种顺序读取,就看不到这种依赖,其报告也会把顺序一并带进答案里。
这项检验很容易表述:把每一对回复评审两次,一次按原顺序,一次互换顺序。如果评审结论跟随它所偏好的那条回复,这个结论就是一致的;如果评审结论跟随的是位置,顺序一变结论就变,这说明评审者的选择既取决于回复所处的位置,也取决于回复本身的内容。评审者可能偏向先读到的回复,也可能偏向后读到的回复。这项检验只能通过顺序的变化显示这种偏向,不能说明偏向的成因,也无需说明,因为随顺序改变的结论,读者不应原样采用。第 5 章设定了两种顺序和评审结论,第 6 章则把评审者的选择与人工评分对照检查。
每一对经人工评分的回复都按两种顺序各评审一次,每一对都落入四类之一:一致、顺序改变、平局或不可读。示例的输出把顺序改变的回复对称为 flipped。不可读的回复对,是指在任一顺序下缺少评审结论。对于顺序改变的回复对,要问的是哪个位置在两种顺序下都胜出。
1"""How the pairwise judge's verdict follows the position it was shown in, on the HelpSteer2 pairs. Each pair is judged in both2orders and sorted as consistent, flipped, tie or unreadable. The flipped pairs are split by the position that won both orders,3and the consistent pairs are checked against the human preference."""4import lib # first: fixes the thread counts and seeds before anything else loads56from lib.bias import flip_direction7from lib.judging import both_orders, human_pairs8from lib.stats import bootstrap_mean, interval_pct, pct910AB = "judge-hs2-pairs-ab"11BA = "judge-hs2-pairs-ba"12SPLITS = ("dev", "held-out")1314pairs = human_pairs() # all 150 pairs, sorted by id, each with its split and the human's preferred response15orders = both_orders(AB, BA, list(pairs)) # every pair judged in both orders, sorted by id1617for split in SPLITS:18 in_split = [o for o in orders if pairs[o.pair_id].split == split]19 for kind in ("consistent", "flipped", "tie", "unreadable"):20 # the denominator is the split's pairs, so the counts add up to the split's size21 print(f"{kind}, {split}: {sum(1 for o in in_split if o.kind == kind)} of {len(in_split)} pairs")2223for split in SPLITS:24 flipped = [o for o in orders if o.kind == "flipped" and pairs[o.pair_id].split == split]25 # the letter A's position won both times when flip_direction says 'first'; B's when it says 'second'26 first = sum(1 for o in flipped if flip_direction(o) == "first")27 print(f"flipped, first position won both orders, {split}: {pct(first, len(flipped))}")2829# both splits pooled, one 0 or 1 per flipped pair: 1.0 when the first position won both orders30flipped_all = [1.0 if flip_direction(o) == "first" else 0.0 for o in orders if o.kind == "flipped"]31print(f"flipped, first position won both orders, both splits: {pct(int(sum(flipped_all)), len(flipped_all))}")32lo, hi = bootstrap_mean(flipped_all) # percentile bootstrap over the flipped pairs, as every interval in the book33print(f"first-position share interval, both splits: {interval_pct(lo, hi)}")3435for split in SPLITS:36 consistent = [o for o in orders if o.kind == "consistent" and pairs[o.pair_id].split == split]37 # in a consistent pair the two orders agree, so ab is the original response the judge picked38 agree = sum(1 for o in consistent if o.ab == pairs[o.pair_id].preferred)39 print(f"consistent pairs agreeing with the human, {split}: {pct(agree, len(consistent))}")
python position-flips.pyconsistent, dev: 60 of 75 pairs
flipped, dev: 15 of 75 pairs
tie, dev: 0 of 75 pairs
unreadable, dev: 0 of 75 pairs
consistent, held-out: 56 of 75 pairs
flipped, held-out: 19 of 75 pairs
tie, held-out: 0 of 75 pairs
unreadable, held-out: 0 of 75 pairs
flipped, first position won both orders, dev: 20.0% (3 of 15)
flipped, first position won both orders, held-out: 26.3% (5 of 19)
flipped, first position won both orders, both splits: 23.5% (8 of 34)
first-position share interval, both splits: [8.8%, 38.2%]
consistent pairs agreeing with the human, dev: 81.7% (49 of 60)
consistent pairs agreeing with the human, held-out: 83.9% (47 of 56)
多数评审结论在两种顺序下保持不变。在开发集上,60 of 75 pairs 对一致,15 of 75 pairs 对随顺序改变。在留出集上,56 of 75 pairs 对一致,19 of 75 pairs 对随顺序改变。
平局算作可读的评审结论,因此仍留在顺序改变率的分母中。这里开发集上的平局为 0 of 75 pairs 对,不可读的结论为 0 of 75 pairs 对,所以开发集中每一对在两种顺序下都给出了评审结论。
在两个数据划分合计的范围内,顺序改变率为 22.7% (34 of 150)。其分母是在两种顺序下都读到结论的回复对,因此任一顺序下结论不可读的回复对不计入。
顺序改变偏向第二个位置,应当读取合并后的计数,因为各划分内的计数很小。在两个划分合计的范围内,第一个位置在顺序改变中所占的份额为 23.5% (8 of 34)。其区间为 [8.8%, 38.2%],且该区间完全落在二分之一以下。因此,对这些回复对而言,合并计数能把这种偏向与均匀分布区分开来。合并份额建立在数量不多的顺序改变之上,所以只要有少数几对朝任一方向变动,就会使它移动几个百分点。仅就开发集而言,这一份额为 20.0% (3 of 15),在留出集上为 26.3% (5 of 19)。
其他研究在别的评审者身上也发现了这种偏向,而且这些偏向的方向并不一致。Wang 等人(2023)是一篇预印本,报告称在使用不同模型的设置中,GPT-4 偏向第一个位置,ChatGPT 偏向第二个位置。Zheng 等人(2023)发表于 NeurIPS 2023 Datasets and Benchmarks 赛道,报告称在使用默认提示词时,GPT-4 在 65.0% 的比较中于两种顺序下给出了相同的结论。这些都是针对其他评审者的已发表设置。它们告诉读者该测量什么,并不能说明对一个 4B 评审者应当预期什么。
一致的回复对,其结论才可以当作一种选择来读,所以示例把它们与人类偏好对照检查。在开发集上,评审者在一致的回复对上与人类相符的比率为:
81.7% (49 of 60)。
在留出集上,这一比率为:
83.9% (47 of 56)。
顺序改变的回复对得不到这样的检查,因为顺序改变的结论并不能说明评审者选择了哪一条回复。如果一个团队只报告一致回复对上的符合率,它测量的只是这组数据中争议较小的部分。顺序改变的部分才是评审者的选择存疑之处,需要单独计数。
这种偏向对团队如何进行比较有实际影响。假设一个团队总是把候选回复放在第二个位置,从不互换。如果评审者偏向第二个位置,候选回复就会得到一份与其答案无关的加成。总是把候选回复放在第一位的团队,则会产生相反的误差。对每一对回复都互换顺序,并报告顺序改变的情况,是把这种偏向从比较中排除,而不是寄希望于它并不存在。
**工作准则:**在读取任何成对比较的评审结论之前,先按两种顺序各评审一次,并在结论计数旁报告顺序改变率。把顺序改变的回复对算作对两条回复都没有结论,而不是算作先出现或后出现的那条回复获胜。当顺序改变偏向某一方向时,要说明偏向哪个方向、有多少对、区间如何,不要假定别的评审者也有同样的偏向。
这部分尚未翻译,因此以英文显示。
Length
Length asks a different question. Position is about which reply the judge sees first; length is about the reply itself. The question is whether the judge gives a longer reply a higher score for being longer. Published studies describe that bias, so a test in this book has to look for it, and a clean result would need a test that could have shown the bias.
Zheng and colleagues (2023) reported that all the models they tested may be prone to verbosity bias, and that GPT-4 defends against it much better than the others. In one attack, a repetitive list was added to 23 answers. The failure rate was 91.3% for two of the models and 8.7% for GPT-4. Those figures come from other judges and a different attack. They are a reason to measure length here, and they say nothing about the size of the effect in this book.
The test is simpler than the attack. It takes the dev replies that the baseline assistant answered, sorts them into three bands by word count, and prints the judge's pass rate for each band. The edges that set the bands are fixed in the example. Other edges would make other bands, and the picture could change with them.
1"""The reference-guided judge's PASS rate by the length of the assistant's reply, on the dev items the baseline answered. The2reply lengths are in words, in three bands with the edges printed below. Each band's rate is printed with its count. Declined3replies and unreadable verdicts are left out, and each is printed with its count."""4import lib # first: fixes the thread counts and seeds before anything else loads56from lib.bias import LENGTH_EDGES, band_outcomes, read_verdicts, reply_lengths7from lib.stats import pct8from lib.testset import load_items910ASSISTANT = "assistant-baseline" # the reply the judge read, and the one whose length is measured11JUDGE = "judge-answers-baseline"1213dev = [item.id for item in load_items(split="dev")] # 80 dev items, sorted by id14lengths = reply_lengths(ASSISTANT, dev) # the answered items only: declined replies are left out here15verdicts = read_verdicts(JUDGE, dev) # None where the judge's first line is not a verdict1617print(f"band edges, in words: {LENGTH_EDGES[0]} and {LENGTH_EDGES[1]}")18print(f"declined, left out: {len(dev) - len(lengths)} of {len(dev)} dev items")19unreadable = [item_id for item_id in lengths if verdicts[item_id] is None]20print(f"unreadable verdict, left out: {', '.join(unreadable) if unreadable else 'none'}")2122for band, outcomes in band_outcomes(lengths, verdicts):23 print(f"{band}: {pct(int(outcomes.sum()), outcomes.size)}")
python length-and-verdicts.pyband edges, in words: 15 and 30
declined, left out: 23 of 80 dev items
unreadable verdict, left out: q-0043
under 15 words: 94.1% (16 of 17)
15 to 29 words: 83.3% (15 of 18)
30 words or more: 85.7% (18 of 21)
The edges fall at 15 and 30 words. Declined replies are left out of the bands, which is 23 of 80 dev items. The judge's first line is read as its verdict. When that line is not a verdict, the verdict is unreadable, and the example prints such an item and leaves it out of the bands. It is never counted as FAIL. Here the unreadable verdict is on q-0043.
The bands are built from answered replies only, so their denominators are smaller than the set. The second-model comparison in the last section uses a different set of items, counting every dev item the judge read for both assistants and counting a correct decline as a pass under the answer rubric in Chapter 5. The two sets are not the same, and their rates should not be set side by side.
The rates do not climb with length. The middle band passed least often, and the longest band passed more often than the middle one, though less often than the shortest. A reader who fits one straight line through three bands would get a slope, and these bands do not support that line.
The shortest band passed 94.1% (16 of 17).
The middle band passed 83.3% (15 of 18).
The longest band passed 85.7% (18 of 21).
The comparison the book reports is the longest band minus the shortest. Its order is fixed by length, so it is set before anyone reads the rates. The highest-rate band minus the lowest-rate band would be chosen after the rates are seen, and it could never fall below zero. The gap is -8.4 points, interval [-28.6, +8.1] points.
The method is percentile bootstrap over items, each band resampled alone, 2000 resamples, 95% level. The two bands hold different items, so each is resampled on its own, and the bands' counts are the ones in their rates above. The band counts are small, so the interval is wide. The interval includes zero, so the result is no difference detected between the longest and the shortest band.
No difference detected does not mean that length is harmless. It says that bands of this size cannot separate a gap from zero. They cannot pick out a few points of bias, because one reply moves a band's rate by several points. The bootstrap over items that intervals and sample size describes is the same resampling. The two bands here are resampled on their own, an interval for two separate groups, which that chapter does not build.
A longer reply is also not always a padded one. A reply that covers more of the question may pass more often for that reason, and a rate that climbs with length would not, by itself, show padding. Telling the two apart needs replies in which length and content come apart, such as a complete answer and the same answer with filler added. The bands here cannot separate them, because the replies in each band differ in more than their length.
Working rule: Don't name a length effect, or its absence, from a few bands. Compare the longest band with the shortest, with the gap's interval and each band's count. When the interval includes zero, say "no difference detected" with those counts, and stop there. If length could matter to your product, build a test in which length changes and the content stays the same, and measure that.
自我偏好:本书无法检验的问题
长度一节探讨了评判者是否会因回复篇幅较长而给予更高评价。本节探讨的是,评判者是否会因回复由谁撰写而给予更高评价。本书无法回答这一问题,其自身的设置恰好说明了原因。
Panickssery 等人(2024)的预印本将自我偏好定义为:LLM 评估者倾向于给自己生成的摘要打出比其他模型的摘要更高的分数,而人类标注者认为这两组摘要的质量相当。作者称其因果性结果只是初步证据,并指出相关性本身并不能证明因果论断。这一问题值得对任何评判者提出,而答案必须在该评判者身上实际测量。
检验需要这样的回答:按人类评分者的判断质量相当,且分别由评判者自身所用的模型和另一个模型生成,或被标注为分别出自二者之手。同一评判者随后会在每种署名之下阅读同样的回答,两个通过率的差异便只能来自撰写者。这一设计的轮廓清晰,但本书的记录并未实施它。
本书只有一个评判者,阅读两个撰写者的回答。基线助手与评判者是同一个模型,即 Qwen3-4B-Instruct-2507。另一组回答来自较小的模型 Qwen3-0.6B,由同一评判者阅读。本书没有对这些回答做质量匹配的人类评分,而这位评判者是衡量其质量的唯一尺度,因此在撰写者变化的同时,这一比较无法保持质量不变。
在同样的开发集条目上,基线的通过率为 78.5% (62 of 79),第二个模型的通过率为 40.5% (32 of 79),差值为 -38.0 points, interval [-48.1, -26.6] points, n = 79。
直接来看,第二个模型的回答通过的频率要低得多。质量较低会产生这样的差距,评判者偏向自身所用模型同样会产生这样的差距。本书的数据无法区分二者,因为衡量质量的唯一尺度就是这位评判者。所以,这一差距只是在这位评判者眼中两个助手之间的差异,既不是自我偏好的结果,也不能说明不存在自我偏好。
若报告称评判者对自身所用的模型是公平的,就必须说明该检验并未在本书的记录上进行。对此避而不谈,读者会把它当作一个明确无偏的结果,而这里并非如此。团队能够提出的论断要窄得多:记录显示了两个助手之间的差距,但对评判者是否偏向自身所用的模型,记录没有显示任何信息。
工作准则: 当评判者与助手是同一个模型时,应在通过率旁注明这一点,而不要将自我偏好报告为已检验。除非上述检查已就位,否则不要用评判者来比较其自身所用的模型与另一个模型;该检查要求质量匹配的回答、由两个模型生成或被标注为出自两个模型,并由同一评判者阅读。第 6 章建立了此类检查所需的人类评分(人工评分)。
何时不应再信任一项判定
这些限度只有成为读者可以应用的规则才有用,而每条规则都需要一个计数来核对。只查看表格的读者得到的是限度,而不是限度背后的理由,上面各节已经说明了每一行的理由。限度表把它们按组打印为带标签的行:顺序变化率;长度差距及其方法;不可读判定;同模型计数;以及第二模型对比。顺序变化率不包括在任一顺序下都不可读的配对,而上文的位置示例则把其划分中的每一对都计算在内。请从表格顶部开始阅读,因为每一行回答一个问题,没有哪一行会回答另一行的问题。
1"""The judge's limits as one table, one labelled line per limit, each computed as in position-flips and length-and-verdicts.2The length line compares the longest band with the shortest, fixed in advance by length order: each band holds different3items, so each is resampled on its own. The unreadable count is over every answer-judge recording on every item."""4import lib # first: fixes the thread counts and seeds before anything else loads56from lib.bias import band_outcomes, paired_passes, read_verdicts, reply_lengths, same_model_count, unreadable_count7from lib.judging import both_orders, human_pairs8from lib.stats import RESAMPLES, bootstrap_difference, bootstrap_mean, interval_points, pct9from lib.testset import load_items1011JUDGE = "judge-answers-baseline"12ANSWER_JUDGES = tuple(f"judge-answers-{variant}" for variant in ("baseline", "k5", "prompt-b", "second-model", "unfiltered"))13items = [item.id for item in load_items()]14dev = [item.id for item in load_items(split="dev")]1516# Order-change rate: flipped pairs over the pairs read in both orders. A tie is a readable verdict; an unreadable pair is left out.17pairs = human_pairs()18read_both = [o for o in both_orders("judge-hs2-pairs-ab", "judge-hs2-pairs-ba", list(pairs)) if o.kind != "unreadable"]19print(f"order-change rate: {pct(sum(1 for o in read_both if o.kind == 'flipped'), len(read_both))}")2021# Length gap: the longest band's PASS rate minus the shortest band's, fixed by length order, on the dev items.22shortest, _, longest = band_outcomes(reply_lengths("assistant-baseline", dev), read_verdicts(JUDGE, dev))23lo, hi = bootstrap_difference(shortest[1], longest[1])24print(f"length gap, longest band minus shortest band: {100 * (longest[1].mean() - shortest[1].mean()):+.1f} points, "25 f"interval {interval_points(lo, hi)}")26print(f"length gap method: percentile bootstrap over items, each band resampled alone, {RESAMPLES} resamples, 95% level")27print(f"shortest band, {shortest[0]}: {pct(int(shortest[1].sum()), shortest[1].size)}")28print(f"longest band, {longest[0]}: {pct(int(longest[1].sum()), longest[1].size)}")2930unreadable, total = unreadable_count(ANSWER_JUDGES, items)31print(f"unreadable verdicts, five answer-judge recordings: {unreadable} of {total}")32print(f"items per answer-judge recording: {len(items)}")33worst = max((unreadable_count([name], items) for name in ANSWER_JUDGES), key=lambda found: found[0])34print(f"most unreadable in one recording: {worst[0]} of {worst[1]}")3536same, judged = same_model_count(ANSWER_JUDGES)37print("self-preference test on these recordings: not possible")38print(f"same-model recordings: {same} of {judged}")3940# Second model beside the baseline, on the dev items both judged: a None verdict in either leaves the item out.41both, base_pass, second_pass = paired_passes(JUDGE, "judge-answers-second-model", dev)42print(f"baseline PASS rate, same dev items: {pct(int(base_pass.sum()), len(both))}")43print(f"second-model PASS rate, same dev items: {pct(int(second_pass.sum()), len(both))}")44diff_lo, diff_hi = bootstrap_mean(second_pass - base_pass)45print(f"second-model minus baseline, same dev items: {100 * (second_pass - base_pass).mean():+.1f} points, "46 f"interval {interval_points(diff_lo, diff_hi)}, n = {len(both)}")
python verdict-limits.pyorder-change rate: 22.7% (34 of 150)
length gap, longest band minus shortest band: -8.4 points, interval [-28.6, +8.1] points
length gap method: percentile bootstrap over items, each band resampled alone, 2000 resamples, 95% level
shortest band, under 15 words: 94.1% (16 of 17)
longest band, 30 words or more: 85.7% (18 of 21)
unreadable verdicts, five answer-judge recordings: 2 of 600
items per answer-judge recording: 120
most unreadable in one recording: 1 of 120
self-preference test on these recordings: not possible
same-model recordings: 4 of 5
baseline PASS rate, same dev items: 78.5% (62 of 79)
second-model PASS rate, same dev items: 40.5% (32 of 79)
second-model minus baseline, same dev items: -38.0 points, interval [-48.1, -26.6] points, n = 79
每个比率都要对照它自己的分母来读。各行并不共用同一个分母,而分母较小的比率只因一个条目就会大幅波动。表格没有说明判定器是否总体上与人的判断一致。那是第 6 章的工作,判定器可能通过这里的每一行检查,却仍在要紧的配对上与人工评分不一致。这些限度是一项判定在值得进行校准之前必须满足的条件,而不是校准的替代品。
下面的规则依照这些行而来,每一条都建立在其中一行之上。
- 顺序变化。 当成对判定随顺序而变时,不要把它当作结果来读。在两种划分上统计顺序变化,并把顺序变化率与判定计数一并打印。
- 不可读判定。 把它们打印出来,将其排除在外,绝不把它们算作 FAIL。表格在五份答案判定器记录中统计到 2 of 600 项不可读判定:这五份记录是基线、k5、prompt-b、第二模型和未过滤变体,每份在 120 个条目上读取。单份记录损失最多的是 1 of 120 项。这是经验法则,而不是测得的临界值:当某份记录中每 100 项判定里有超过 1 项不可读时,应先修正提示,再从该记录读取比率。
- 长度。 不要仅凭几个长度区间就断言存在长度效应,也不要仅凭它们就断言不存在该效应。长度行把最长区间与最短区间相比较,并给出其区间。当该区间包含零时,请连同计数一起写“未检测到差异”,到此为止。
- 同模型。 当判定器与助手是同一个模型时,比率旁边应带有这一事实。这里有 4 of 5 份被判定的记录属于这种情形。本书无法在这些记录上检验自我偏好,因此不要用判定器来比较它自己的模型与另一个模型,如自我偏好一节所述。
- 质量匹配。 两个助手通过率之间的差距是助手之间的差异,而不是判定器的偏差。第二模型一行就是示例,这也是自我偏好一节无论正反都不给出结果的原因。
- 迁移。 比率所依赖的校准是在其他行上测得的,因此只能把它带到迁移假设所允许的范围。
满足停止条件的判定不会被丢弃。顺序变化的配对或不可读的判定,对该条目而言成为无判定。它被打印、计数,并被排除在它本会进入的比率之外,不可读判定就是这样处理的,随后该比率会报告它排除了多少个条目。同模型记录不是停止条件,它是在比率旁边加以报告的。
在生产中,这些限度表现为比率因错误的原因而变动。假设某团队更改了助手的提示,通过率上升。在团队把功劳归于提示之前,限度表会提出三个问题。判定是否随顺序而变,或者无法读取?回复是否变长了,长度行的区间能否把差距与零区分开?判定器是否就是撰写这些回复的模型?前两个问题若答案为是,则这一上升在报告之前需要单独核查。第三个问题若答案为是,则报告这一上升时要在旁边注明这一事实。
工作准则: 当一项判定随顺序而变或无法读取时,不再信任它,并在报告中说明。当判定器就是其所评分答案的生成模型时,把这一点报告在比率旁边,并且不要用判定器去比较它自己的模型与另一个模型,如自我偏好一节所述。在发布通过率之前,像第 6 章那样对照人工评分检查判定器,并把顺序变化率、不可读计数和同模型这一事实一并打印在比率旁边。通过率对答案的描述,只限于判定器与人的判断一致的程度,因此比率应始终连同它的校准一起呈现。
本章,在本版次中
版次 2026.11.3示例于 2026年11月21日 通过本章 3个示例 个示例中有 3 个通过
译自英文版图书,版次 2026.11.3。