人工与 LLM 的相关性判断
阅读约需 76分钟 分钟,13节中含11个示例。
学完本章,你将能够撰写评判指南,用 kappa 和加权 kappa 衡量两位评判者的一致程度,从混淆矩阵中读出 LLM 评判者的错误,并判断 LLM 的判断在哪些场合可以替代人工判断,在哪些场合不能。
本书中的每个分数都建立在判断之上:人们阅读在商店搜索框中输入的查询,与它返回的某件商品,并对这一配对打分。这些判断就是 WANDS 的标签,每个标签都是最多三位受过培训的标注者的多数投票结果,而且存在空缺:在测试查询上,BM25 的前 10 名结果中有 22 件商品无人评判(见统计未评判的商品)。这些判断还要耗费人力,而且 WANDS 标注者之间的一致程度在四个月的校准期间有所提高(见预期评判者会有分歧)。LLM 可以为给定的任何查询和商品打分。在商店让 LLM 担任评判者之前,需要弄清它与商店的评判者有多大程度一致,以及它给出的分数是否会让人对排序器做出同样的决定。
LLM 的分数来自一份录制记录:recordings/llm_judgments.py 在本地以温度 0 和固定的随机种子,将开源模型 gemma4:12b-it-q4_K_M 运行了一次,并保存了它的回答,因此示例通过 lib.recorded.llm_judgments() 读取这些回答,而无需调用模型。我们选择这个模型,是因为它是开源的,采用 Apache 2.0 许可证,并且可在本地运行,所以任何人都可以重新运行这份录制。JSON 模式(schema)将每个回答限定为三种标签之一。录制时用两种提示词为每个配对打分,即 short 和 guidelines(见撰写评判指南),配对取自两个集合,本章称之为评判池。
用于在排序器之间做出取舍的分数仍然基于人工评分。本章在测试查询上计划进行的比较,是第 3 章的三项比较:用各提示词的 LLM 分数,按第 3 章的阈值 0.0167 重新计分,进行配对置换检验。这三项比较是:语义检索对比 BM25,混合检索对比语义检索,以及重排序后的混合检索对比混合检索。
撰写评判准则
本章把给出判断的人,无论是真人还是 LLM,都称为评判者,而评级是代表这一判断的数字。第 1 章把判断定义为人给出的标签;本章把它扩展到 LLM 给出的标签。判断取决于其背后的指示。评判准则说明每个评级的含义、如何在各评级之间做选择,以及如何处理介于两级之间的情形。
WANDS 公布了其标注人员使用的准则:共八页,采用 MIT 许可(标注准则)。对于每个查询与商品的配对,准则列出了评判者所能看到的内容:查询,以及商品的 ID、名称、图片和网页。准则定义了三个评级,并附有两条说明:一条要求评判者先理解查询,即“强烈建议先用谷歌搜索”;另一条要求忽略拼写错误,按配对的原样进行标注。随后准则以示例进行讲解,为每个评级给出查询、商品和理由:对于 wire basket with dividers(商店自己的训练查询之一),没有隔板的线篮属于部分相关;而对于 salon chair,沙龙防疲劳垫属于不相关,因为顾客要找的是沙龙椅,而不是垫子。准则也会随着评判者的使用而变化。WANDS 的作者称,其标注人员之间的一致性在四个月里不断提高,他们给出的原因包括:每天有讨论相互冲突的标注的例行安排,以及根据标注人员的意见改进准则(Chen 等,2022,第 5 节)。某个网络搜索引擎的评估员准则也是如此,其 2025 年 9 月 11 日版本长达 182 页(搜索质量评估员准则)。
录制过程中使用的两个提示词就是给 LLM 的评判准则。short 仅包含 WANDS 的三条定义;guidelines 保留这三条定义,并增加五条规则,因此请按录制发送时的样子将其打印出来:
1"""Prints the model, the word count of each prompt and the guidelines prompt verbatim, from the recording's manifest. The prose's point rests on the rules at the end of that prompt."""2from lib.recorded import manifest34recording = manifest("llm-judgments") # the recording's model, settings and both prompts, verbatim5short, guidelines = recording["prompts"]["short"], recording["prompts"]["guidelines"]67print("model:", recording["model"])8print("what the model saw:", recording["sees"])9print("words in the short prompt:", len(short.split()))10print("words in the guidelines prompt:", len(guidelines.split()))11print()12print(guidelines)
python print-judging-guidelines.pymodel: gemma4:12b-it-q4_K_M
what the model saw: lib.store text plus attributes; no images
words in the short prompt: 59
words in the guidelines prompt: 206
You judge product search results for an online furniture and home-goods store. A shopper typed the query and was shown the product. Decide how well the product matches what the shopper asked for.
Exact: the product fully matches the search query.
Partial: the product only matches the target entity of the query, but does not satisfy the modifiers for the query.
Irrelevant: the product is not relevant to the query.
Guidelines:
- First find the target entity: the kind of product the query asks for (a sofa, a lamp, a rug).
- Then find the modifiers: everything else the query asks for, such as colour, size, material, style, brand or a feature.
- Exact needs the right kind of product and every modifier. Partial is the right kind of product with at least one modifier missing or contradicted. Irrelevant is the wrong kind of product, even when it shares words with the query, such as an accessory for the product asked for.
- Read misspelled queries as the shopper meant them.
- Judge only from the product's text and attributes. If a modifier isn't mentioned, decide from what is written, and don't assume it is met.
Query: {query}
Product:
{product}
Answer with the label in JSON.
guidelines 提示词是两者中较长的一个,有 206 个词,而 short 提示词为 59 个词。它的前两条规则阐明了 WANDS 对部分相关的定义:先找出实体,再看修饰语。配件规则解决了 WANDS 自己的标签出现分歧的一种情形:对于 clear shower curtain,WANDS 把同一套浴帘挂钩列出了两次,一次标为部分相关,一次标为不相关(预期评判者会有分歧)。拼写错误规则沿用了 WANDS 的说明。最后一条规则在 WANDS 的准则中没有对应内容,因为该商店的数据不含图片,所以录制时只向 LLM 提供文本。两个提示词都没有 WANDS 准则所依赖的带解析示例。
工作准则: 把准则写成定义、一套流程、一个为每个评级附上理由的带解析示例,以及针对评判者曾有分歧的每种情形的一条规则。记录分歧,当某一类分歧反复出现时就修订准则。给 LLM 评判者的准则应与给真人的相同,再加上一条针对它看不到的内容的规则,然后衡量它的实际表现,因为你写下的规则并不等于评判者遵循的规则。
用 kappa 度量超出随机水平的一致性
在信任第二个评判者之前,先度量它与第一个评判者的一致程度。两个评判者对同一批配对给出相同等级的比例,即观察一致率,是最直观的数字,但它会让结果显得偏高。两个相互独立地随机猜测的评判者,也会在一部分配对上偶然一致;当两者都经常使用同一个标签时,偶然一致的配对更多。Cohen's kappa 对观察一致率做随机校正:它把观察一致率与这样一种情形下的期望一致率相比较,即每个评判者保持各自的标签占比,但彼此独立地选择标签。kappa 为 1 表示完全一致,为 0 则表示“没有超出随机预期的一致”(scikit-learn 文档):
kappa = (observed - expected) / (1 - expected)
expected = sum over grades g of (share of pairs judge A gives g) × (share of pairs judge B gives g)
用于度量一致性的训练池包含人工针对训练查询评判过的配对,按标签均匀抽取:每个查询每种标签最多 4 个,共 550 个配对。测试池包含将由 LLM 评分、用于给商店的各排序器打分的配对:对每个测试查询,取第 1 章四种排序器各自前 10 名中的每一件商品,无论是否有人工评判过,共 590 个配对。这四种排序器是 BM25、基于 Snowflake/snowflake-arctic-embed-m-v1.5 嵌入的语义检索、二者的混合,以及由交叉编码器 Alibaba-NLP/gte-reranker-modernbert-base 对混合结果重排序的排序器。对 LLM 评判者的每次度量只对其所在的池成立,因此本章每次都会指明所用的池。
问题在于,该录制中的 LLM 对一个配对的评级,与商店的评判者的评级有多常一致。使用 short 提示(仅含 WANDS 的三条定义)在训练池上即可回答这一问题。先手工构建一个 3×3 的计数表,由它得出 kappa,再用 scikit-learn 的 cohen_kappa_score 核对结果:
1"""Builds the 3×3 table of the store's grades against the short prompt's on the train pool, then computes observed agreement, chance agreement and kappa from it. The prose's point rests on the printed kappa by hand, which matches the scikit-learn line."""2import numpy as np3from sklearn.metrics import cohen_kappa_score45from lib.agreement import graded_pairs67pairs = graded_pairs("train") # up to 4 human-judged pairs per label per train query8# the grades are nullable integers; plain ones compare and count the way the formulas below expect9human, llm = pairs["human"].astype(int), pairs["short"].astype(int)10grades = [2, 1, 0] # exact, partial, irrelevant11counts = np.array([[((human == row) & (llm == column)).sum() for column in grades] for row in grades])12print("pairs:", counts.sum())13print("rows are the human grades 2, 1, 0 and columns the LLM's, short prompt")14print(counts)1516observed = np.trace(counts) / counts.sum() # the share of pairs on the diagonal17human_share, llm_share = counts.sum(axis=1) / counts.sum(), counts.sum(axis=0) / counts.sum()18expected = (human_share * llm_share).sum() # agreement if the two judged independently19kappa = (observed - expected) / (1 - expected)20print(f"observed agreement, short prompt: {observed:.3f}")21print(f"agreement expected by chance, short prompt: {expected:.3f}")22print(f"kappa by hand, short prompt: {kappa:.3f}")23print(f"kappa from scikit-learn, short prompt: {cohen_kappa_score(human, llm):.3f}")2425other = pairs["guidelines"].astype(int)26print(f"observed agreement, guidelines prompt: {(human == other).mean():.3f}")27print(f"kappa from scikit-learn, guidelines prompt: {cohen_kappa_score(human, other):.3f}")
python kappa-by-hand.pypairs: 550
rows are the human grades 2, 1, 0 and columns the LLM's, short prompt
[[102 85 9]
[ 22 132 46]
[ 3 42 109]]
observed agreement, short prompt: 0.624
agreement expected by chance, short prompt: 0.337
kappa by hand, short prompt: 0.432
kappa from scikit-learn, short prompt: 0.432
observed agreement, guidelines prompt: 0.633
kappa from scikit-learn, guidelines prompt: 0.454
LLM 与商店的评判者对 0.624 的训练配对给出了相同的等级,而仅凭随机就会产生 0.337 的一致。kappa 先从观察一致率中减去这部分随机占比,再除以高于随机水平的剩余空间,对 short 得到 0.432,与 scikit-learn 的 0.432 相同。
guidelines 与评判者一致的频率比 short 略高,在 0.633 的配对上一致,kappa 为 0.454。
这算好吗?把 kappa 称为“中等”或“相当高”的量表,是特定领域的惯例。Artstein 和 Poesio 的综述指出,为某一特定目的选择合适的一致性水平,“仍然近乎一门玄学”(Artstein and Poesio, 2008)。
已发表的数字只提供背景,而不是门槛。WANDS 的标注者在最初两个月内彼此的 kappa 为 0.467,在第四个月为 0.826(Chen et al., 2022, Table 2)。
在 TREC(Text REtrieval Conference 的共享评测)的配对分层样本上,Thomas 等人发现,GPT-4 相对于评估者的 kappa 低至 0.20,高至 0.64,取决于提示。两组受过训练的人工评估者对 TREC ad hoc 文档进行第二次评判时,kappa 达到 0.52(Cormack et al., 1998, as reported by Thomas et al., 2024)。每个数字都来自不同的样本,而且 kappa 取决于样本的标签构成。
工作准则: 报告 kappa,并在旁边给出观察一致率,不要只报告观察一致率。在人工针对训练查询评判过的配对上度量它,并把所用的任何阈值标明为经验法则。当评判者多于两个或存在缺失标签时,使用 Krippendorff's alpha,它将观察到的不一致与随机不一致相比较,可处理任意数量的观察者和不完整的数据(Krippendorff, 2011)。
用加权 kappa 给接近的判断部分得分
Kappa 对每一次分歧一视同仁。商店的等级是有序的,完全相关高于部分相关,部分相关高于不相关,把完全相关的商品判为部分相关,比判为不相关的错误要小。在 nDCG@10 中(给一个排序打分),前者损失的增益只有后者的一半:差一个等级,而不是两个。加权 kappa 按距离给每次分歧一个权重,再把加权后的分歧与偶然情况下预期的加权分歧相比较(Artstein and Poesio, 2008, section 2.6)。采用线性权重时,相差一个等级的分歧权重为 0.5,相差两个等级的分歧权重为 1:
weighted kappa = 1 - (sum of weight × observed count) / (sum of weight × count expected by chance)
如果每次分歧的权重都是 1,这个公式给出的就是普通 kappa。scikit-learn 的 cohen_kappa_score 就是这样计算这三种 kappa 的:默认权重为 1,使用 weights="linear" 时权重为等级之间的距离,使用 weights="quadratic" 时权重为距离的平方(scikit-learn source)。对权重做整体缩放不会改变结果。
对于这两个提示词,本示例在衡量超出偶然的一致性中均匀抽取的训练对上,手工构造线性权重,然后用 scikit-learn 核对每个结果:
1"""Computes linear weighted kappa by hand for both prompts on the train pool and checks each against scikit-learn. The prose's point rests on the two linear weighted kappa lines, which match scikit-learn's, and on the one-grade-apart shares beside them."""2import numpy as np3from sklearn.metrics import cohen_kappa_score, confusion_matrix45from lib.agreement import PROMPTS, graded_pairs67pairs = graded_pairs("train") # up to 4 human-judged pairs per label per train query8grades = [2, 1, 0] # exact, partial, irrelevant9cost = np.abs(np.subtract.outer(grades, grades)) / 2 # 0 on the diagonal, 0.5 one grade apart, 1 two apart10print("disagreement weights, rows and columns 2, 1, 0")11print(cost)1213for prompt in PROMPTS:14 human, llm = pairs["human"].astype(int), pairs[prompt].astype(int)15 # labels fixes the rows and columns to 2, 1, 0 in that order, not the order the grades first appear16 counts = confusion_matrix(human, llm, labels=grades) # rows human, columns LLM17 expected = np.outer(counts.sum(axis=1), counts.sum(axis=0)) / counts.sum() # counts if independent18 weighted = 1 - (cost * counts).sum() / (cost * expected).sum()19 near = cost == 0.5 # cells one grade apart20 print()21 # row 0 is exact and column 2 is irrelevant, so these two corners are the far misses22 print(f"pairs two grades apart, {prompt} prompt: {counts[0, 2] + counts[2, 0]}")23 print(f"disagreements one grade apart, share, {prompt} prompt: {counts[near].sum() / counts[cost > 0].sum():.3f}")24 print(f"one grade apart, share expected by chance, {prompt} prompt: {expected[near].sum() / expected[cost > 0].sum():.3f}")25 print(f"kappa, {prompt} prompt: {cohen_kappa_score(human, llm):.3f}")26 print(f"linear weighted kappa by hand, {prompt} prompt: {weighted:.3f}")27 print(f"linear weighted kappa from scikit-learn, {prompt} prompt: "28 f"{cohen_kappa_score(human, llm, weights='linear'):.3f}")
python weighted-kappa-by-hand.pydisagreement weights, rows and columns 2, 1, 0
[[0. 0.5 1. ]
[0.5 0. 0.5]
[1. 0.5 0. ]]
pairs two grades apart, short prompt: 12
disagreements one grade apart, share, short prompt: 0.942
one grade apart, share expected by chance, short prompt: 0.742
kappa, short prompt: 0.432
linear weighted kappa by hand, short prompt: 0.523
linear weighted kappa from scikit-learn, short prompt: 0.523
pairs two grades apart, guidelines prompt: 30
disagreements one grade apart, share, guidelines prompt: 0.851
one grade apart, share expected by chance, guidelines prompt: 0.664
kappa, guidelines prompt: 0.454
linear weighted kappa by hand, guidelines prompt: 0.531
linear weighted kappa from scikit-learn, guidelines prompt: 0.531
加权后,short 的 kappa 从 0.432 升到 0.523,guidelines 的 kappa 从 0.454 升到 0.531,两者都与 scikit-learn 的结果一致。之所以上升,是因为相差一个等级的分歧出现得比偶然情况下更频繁:short 的这类分歧占 0.942,而偶然预期为 0.742;guidelines 的占 0.851,偶然预期为 0.664。
两个提示词在严重偏差上也有所不同,即相差两个等级的配对:short 为 12,guidelines 为 30。哪一个提示词的一致性更好,要由区间来回答。
工作准则: 对于有序等级,在普通 kappa 之外报告线性加权 kappa,并单独统计严重偏差。每次分歧的权重应按它所损失的增益差来定。采用本书的线性增益时,就是线性权重。采用 0、1 和 3 的指数增益时(选择指标),差值为 1、2 和 3,scikit-learn 的两个选项都给不出这样的权重,因此要像示例构造线性矩阵那样自行构造这个矩阵。
结合标签构成解读 kappa
Kappa 的校正取决于各评判者使用每个标签的频率,因此同一评判者在标签构成不同的样本上会得到不同的 kappa。当某一个标签占据主导时,观察一致率可能很高,而 kappa 仍然很低。Artstein 和 Poesio 把这称为流行率问题(prevalence problem):“标注者可能在很高比例的条目上达成一致,而且标注在很大程度上确实正确,但可靠性系数仍然很低”(Artstein and Poesio, 2008)。
这次记录对训练池做了均衡抽样:每个训练查询的每种标签最多取 4 个配对,因此池中的完全相关(exact)配对和不相关(irrelevant)配对多于商店判断中的数量。要了解这会对 kappa 产生什么影响,可以给每个配对加权,使训练池具有全部训练查询判断的标签构成,然后重新测量:
1"""Measures each prompt's agreement with the store's judges on the train pool, as drawn and reweighted to the label mix of all the train judgments. The prose's point rests on the kappa lines for each mix and on the higher-kappa lines, which change with the mix."""2import numpy as np3from sklearn.metrics import cohen_kappa_score45from lib.agreement import LABELS, PROMPTS, graded_pairs6from lib.store import load_labels, load_queries78pairs = graded_pairs("train") # drawn evenly: up to 4 pairs per label per train query9drawn = pairs["human"].value_counts(normalize=True)10labels = load_labels()11# every judged label for the train queries, not only the pairs that were drawn12natural = labels[labels["query_id"].isin(load_queries("train").index)]["grade"].value_counts(normalize=True)13top = graded_pairs("test")["human"].dropna().value_counts(normalize=True) # the rankers' top 10s, judged pairs14for grade, label in LABELS.items():15 print(f"{label} share, as drawn: {drawn[grade]:.3f}")16 print(f"{label} share, in the train judgments: {natural[grade]:.3f}")17 print(f"{label} share, judged test-pool pairs: {top[grade]:.3f}")1819# each pair counts for its label's share in the judgments divided by its share as drawn20weight = pairs["human"].map(natural / drawn).astype(float) # count each pair as if drawn in that mix21kappas = {}22for prompt in PROMPTS:23 human, llm = pairs["human"].astype(int), pairs[prompt].astype(int)24 print()25 print(f"observed agreement, {prompt} prompt, as drawn: {(human == llm).mean():.3f}")26 print(f"kappa, {prompt} prompt, as drawn: {cohen_kappa_score(human, llm):.3f}")27 print(f"observed agreement, {prompt} prompt, train judgments' mix: {np.average(human == llm, weights=weight):.3f}")28 # sample_weight counts each pair that many times; it is not the linear weights of the last section29 kappas[prompt] = cohen_kappa_score(human, llm), cohen_kappa_score(human, llm, sample_weight=weight)30 print(f"kappa, {prompt} prompt, train judgments' mix: {kappas[prompt][1]:.3f}")31print()32for position, mix in enumerate(["as drawn", "train judgments' mix"]):33 # on a tie, max returns the first prompt, short34 print(f"higher kappa, {mix}: {max(PROMPTS, key=lambda prompt: kappas[prompt][position])}")
python kappa-under-two-label-mixes.pyexact share, as drawn: 0.356
exact share, in the train judgments: 0.234
exact share, judged test-pool pairs: 0.543
partial share, as drawn: 0.364
partial share, in the train judgments: 0.613
partial share, judged test-pool pairs: 0.427
irrelevant share, as drawn: 0.280
irrelevant share, in the train judgments: 0.153
irrelevant share, judged test-pool pairs: 0.030
observed agreement, short prompt, as drawn: 0.624
kappa, short prompt, as drawn: 0.432
observed agreement, short prompt, train judgments' mix: 0.635
kappa, short prompt, train judgments' mix: 0.370
observed agreement, guidelines prompt, as drawn: 0.633
kappa, guidelines prompt, as drawn: 0.454
observed agreement, guidelines prompt, train judgments' mix: 0.573
kappa, guidelines prompt, train judgments' mix: 0.345
higher kappa, as drawn: guidelines
higher kappa, train judgments' mix: short
部分相关(partial)在按抽样原样的训练池中占 0.364,在训练判断中占 0.613,因此重新加权会更重地计入部分相关配对,而更轻地计入完全相关和不相关配对。
在判断的标签构成下,short 的观察一致率由 0.624 变为 0.635,kappa 由 0.432 变为 0.370。对于 guidelines,观察一致率由 0.633 变为 0.573,kappa 由 0.454 变为 0.345。
按抽样原样,kappa 较高的提示词是 guidelines;在判断的标签构成下,则是 short。
已发表的数字都带有各自的标签构成。我们找到的唯一一项关于 WANDS 上 LLM 评判者的研究,在三种标签上均匀抽取了 5,000 个查询与商品的配对。它报告的准确率从表现最好的开源模型(700 亿参数)的 0.726,到表现最好的闭源模型的 0.837,均为各自的最佳设置,即为每个配对检索 16 个示例(worked examples)(Sachdev et al., 2025, Table 3)。商店的提示词没有示例。该准确率是均衡样本上的观察一致率,与训练池的情形相同,因此这两个数字都不能迁移到商店判断的标签构成上。
商店将请 LLM 判断的配对,是各排序器排在前 10 位的商品,即测试池。在其已判断的配对中,0.543 为完全相关,0.427 为部分相关,0.030 为不相关。这是第三种构成,也是另一类配对,解读 LLM 的错误方向一节会对其进行测量。
工作准则: 报告每个 kappa 时都要同时报告标签构成,并且不要比较来自不同标签构成样本的 kappa,无论是你自己的还是已发表的。我们建议按你使用评判者的方式抽取一致性样本:如果它将用于给排序器的前 10 位结果评分,就从训练查询的前 10 位结果中抽取配对。
解读 LLM 错误的方向
Kappa 说明两位评判者的一致程度,而不说明二者如何不同。混淆矩阵展示两位评判者的差异:它是一张计数表,每一行对应商店的一个等级,每一列对应 LLM 的一个等级,行列均按完全相关、部分相关、不相关的顺序排列。对角线以上,表示 LLM 给出的等级低于商店的评判者;对角线以下,表示高于商店的评判者。错误多落在对角线以下的评判者是宽松的,多落在对角线以上的则是严格的。
若干研究发现 LLM 评判者偏宽松。TREC 2024 的 RAG 赛道评估检索增强生成(根据检索到的文本写出答案),在该赛道上,人工评估者总体上显得比开源 LLM 评判者 UMBRELA 更严格(Upadhyay et al., 2024a)。Arabzadeh 和 Clarke 报告称,基于 GPT-4o 构建的 UMBRELA(Upadhyay et al., 2024b),在 TREC 2019 至 2021 年的 Deep Learning 赛道中,标为完全相关的文档比人工多出 26% 以上,分别为 3,063 篇和 2,429 篇(Arabzadeh and Clarke, 2025a)。Yu 等人报告称,LLM 评判者"一贯给出偏高的相关性分数"(Yu et al., 2026)。
并非所有研究都得出相同结论。Thomas 等人用其 32 个提示中的一个,在按标签分层抽取的样本上发现,LLM 比 TREC 的评估者更倾向于把一个查询与文档对判为不相关,比例为 44% 对 33%(Thomas et al., 2024)。
为每个提示在每个样本池上各做一个矩阵,可以看出 LLM 偏向哪个方向的错误,以及这一方向在从均衡的训练对转到各排序器的前 10 名时是否依然成立。训练池按标签均匀抽取,而测试池中被评判的对,是 LLM 要评分的第 1 章四个排序器的前 10 名。测试池矩阵报告的是测试查询上的情况,不用于选择提示:
1"""Prints the confusion matrix of the store's grades against each prompt's grades, on the train pool and on the judged test pairs, with the counts of lower and higher grades. The prose's point rests on the 'stricter' line for each pool, read with the largest error printed beside it."""2import pandas as pd3from sklearn.metrics import cohen_kappa_score45from lib.agreement import LABELS, PROMPTS, graded_pairs67order = list(LABELS.values()) # rows: the store's grades; columns: the LLM's; both exact first8for pool in ["train", "test"]:9 pairs = graded_pairs(pool).dropna(subset=["human"]) # in the test pool, the pairs people judged10 human = pairs["human"].astype(int)11 print(f"\njudged pairs, {pool} pool: {len(pairs)}\nmean human grade, {pool} pool: {human.mean():.3f}")12 for prompt in PROMPTS:13 llm, name = pairs[prompt].astype(int), f"{pool} pool, {prompt} prompt"14 table = pd.crosstab(human.map(LABELS).rename("human"), llm.map(LABELS).rename(f"llm, {name}"))15 table = table.reindex(index=order, columns=order, fill_value=0)16 errors = {f"{row} called {column}": table.at[row, column] for row in order for column in order if row != column}17 # on a tie, max returns the first cell, in row order18 largest = max(errors, key=errors.get)19 print(table)20 print(f"kappa, {name}: {cohen_kappa_score(human, llm):.3f}\nmean llm grade, {name}: {llm.mean():.3f}")21 print(f"llm grade higher than human, {name}: {(llm > human).sum()}")22 print(f"llm grade lower than human, {name}: {(llm < human).sum()}")23 # a tie counts as not stricter24 print(f"stricter than the store's judges, {name}: {'yes' if (llm < human).sum() > (llm > human).sum() else 'no'}")25 print(f"largest error, {name}: {largest} ({errors[largest]})")
python llm-confusion-matrix.py
judged pairs, train pool: 550
mean human grade, train pool: 1.076
llm, train pool, short prompt exact partial irrelevant
human
exact 102 85 9
partial 22 132 46
irrelevant 3 42 109
kappa, train pool, short prompt: 0.432
mean llm grade, train pool, short prompt: 0.933
llm grade higher than human, train pool, short prompt: 67
llm grade lower than human, train pool, short prompt: 140
stricter than the store's judges, train pool, short prompt: yes
largest error, train pool, short prompt: exact called partial (85)
llm, train pool, guidelines prompt exact partial irrelevant
human
exact 118 53 25
partial 30 98 72
irrelevant 5 17 132
kappa, train pool, guidelines prompt: 0.454
mean llm grade, train pool, guidelines prompt: 0.862
llm grade higher than human, train pool, guidelines prompt: 52
llm grade lower than human, train pool, guidelines prompt: 150
stricter than the store's judges, train pool, guidelines prompt: yes
largest error, train pool, guidelines prompt: partial called irrelevant (72)
judged pairs, test pool: 564
mean human grade, test pool: 1.512
llm, test pool, short prompt exact partial irrelevant
human
exact 252 53 1
partial 57 177 7
irrelevant 2 6 9
kappa, test pool, short prompt: 0.571
mean llm grade, test pool, short prompt: 1.521
llm grade higher than human, test pool, short prompt: 65
llm grade lower than human, test pool, short prompt: 61
stricter than the store's judges, test pool, short prompt: no
largest error, test pool, short prompt: partial called exact (57)
llm, test pool, guidelines prompt exact partial irrelevant
human
exact 261 41 4
partial 76 133 32
irrelevant 3 4 10
kappa, test pool, guidelines prompt: 0.470
mean llm grade, test pool, guidelines prompt: 1.521
llm grade higher than human, test pool, guidelines prompt: 83
llm grade lower than human, test pool, guidelines prompt: 77
stricter than the store's judges, test pool, guidelines prompt: no
largest error, test pool, guidelines prompt: partial called exact (76)
在训练池上,LLM 比商店的评判者更严格吗?使用 short 时:yes。使用 guidelines 时:yes。使用 short 时,它给出更低等级的有 140 对,更高等级的有 67 对;使用 guidelines 时,更低的有 150 对,更高的有 52 对。最大误差在使用 short 时为 exact called partial (85),在使用 guidelines 时为 partial called irrelevant (72)。
在测试池上,使用 short 时的答案是 no,使用 guidelines 时是 no。使用 short 时,LLM 给出更高等级的有 65 对,更低等级的有 61 对;使用 guidelines 时,更高的有 83 对,更低的有 77 对。它的平均等级在使用 short 时为 1.521,在使用 guidelines 时为 1.521,而评判者的平均等级为 1.512。它的最大误差在使用 short 时为 partial called exact (57),在使用 guidelines 时为 partial called exact (76)。
Kappa 在使用 short 时为 0.571,在使用 guidelines 时为 0.470,而训练池上分别为 0.432 和 0.454。方向、最大误差和 kappa 都随样本池而变,正如标签构成所警示的那样。比较 LLM 看到了什么找到了一种同时适用于两个样本池的模式。
错误的方向也不能说明排序器的得分会发生什么变化。宽松的评判者会把更多商品算作相关,这会提高 precision@10(统计相关商品)。nDCG@10 除以由同一组等级构建的理想前 10 名(为一个排序打分),因此提高等级也可能同时抬高理想值,净效果可能朝任一方向变化,并因排序器而异。排序器顺序的检验在测试池上对此做了测量。
工作准则: 先看混淆矩阵,再看 kappa,所用的查询与文档对要按你将来使用评判者的方式抽取,并为该样本池指明方向和最大误差。不要把一个样本池的方向带到另一个样本池,也不要据此预测排序器的 nDCG@10。应在相同的等级下相互比较排序器,切勿把 LLM 等级下的得分与人工等级下的得分相比较。
对比 LLM 所见与评判者所见
评判者只能依据所见的证据打分。WANDS 的评判者看到的是查询词以及产品的 ID、名称、图片和网页,而标注指南要求他们先弄懂查询的含义:“强烈建议先上网搜索”(标注指南)。录制记录向 LLM 展示了每个产品的名称、类别、品类、描述和属性,正如 lib.agreement.as_the_llm_saw 所写出的那样。商店的样本把描述截断为约 400 个字符(数据卡),且不含图片,因此录制时发给 LLM 的只有文本。部分分歧可能源于这一差距,而不是 LLM 的评判能力;这是我们的推断。其中一个推论可以检验:如果 LLM 依据的是文字所写的内容,那么对于文本中没有写明查询所求内容的 exact 产品,它应给出更低的等级;对于文本中写明了所求内容的 partial 产品,它应给出更高的等级。
这项检查取一个训练对、测试查询 clear shower curtain,以及两个池中所有已评判的 exact 和 partial 对(录制记录)。它在 LLM 所见的内容中查找每个查询词,并像第 2 章的检查那样把内容拆分为小写单词:
1"""Checks whether the LLM's grades follow the words in each product's text, for one train pair, for the clear shower curtain products, and across both pools. The prose's point rests on the 'share graded lower' and 'share graded higher' lines, which compare pairs whose text has every query word with pairs that miss one."""2import re34from lib.agreement import PROMPTS, as_the_llm_saw, graded_pairs5from lib.store import load_products, load_queries678def words(text):9 """Lowercase word tokens, as chapter 2 matches query words against product text."""10 # runs of letters, digits and underscores, so 7qt is one word and '7 qt' is two11 return set(re.findall(r"\w+", text.lower()))121314products, queries = load_products(), load_queries()15# query 163 is a train query, and product 20718 is the pair the prose follows16print("query:", queries.at[163, "query"]) # a train query17print(as_the_llm_saw(products.loc[20718]))18print("grades from people, short, guidelines:", *graded_pairs("train").set_index(["query_id", "product_id"]).loc[(163, 20718)])1920curtains = graded_pairs("test").query("query_id == 246") # clear shower curtain: every product in a top 1021# whole words only, so 'unclear' doesn't count as saying clear22says = curtains["product_id"].map(lambda product_id: "clear" in words(as_the_llm_saw(products.loc[product_id])))23for kind, group in [("say clear", curtains[says]), ("never say clear", curtains[~says])]:24 print(f"\nclear shower curtain products that {kind}: {len(group)}")25 for column, judge in [("human", "people"), ("short", "the short prompt"), ("guidelines", "the guidelines prompt")]:26 print(f"products that {kind}, graded exact by {judge}: {(group[column] == 2).sum()}")2728for pool in ["train", "test"]:29 pairs = graded_pairs(pool).dropna(subset=["human"]) # in the test pool, the pairs people judged30 # subset: every query word appears in the text the LLM saw31 pairs["every word"] = [words(queries.at[query_id, "query"]) <= words(as_the_llm_saw(products.loc[product_id]))32 for query_id, product_id in zip(pairs["query_id"], pairs["product_id"])]33 # share of each group that the LLM grades below people (exact) or above them (partial), for each prompt34 for label, grade, every_word, direction in [("exact", 2, False, "lower"), ("exact", 2, True, "lower"),35 ("partial", 1, True, "higher"), ("partial", 1, False, "higher")]:36 group = pairs[(pairs["human"] == grade) & (pairs["every word"] == every_word)]37 kind = f"{label} {pool} pairs {'with every query word' if every_word else 'missing a query word'}"38 print(f"\n{kind}: {len(group)}")39 for prompt in PROMPTS:40 moved = group[prompt] < grade if direction == "lower" else group[prompt] > grade41 print(f"share graded {direction}, {prompt} prompt, {kind}: {moved.mean():.3f}")
python llm-grades-follow-written-words.pyquery: dual monitor stand
Name: abramowski monitor stand
Class: Desktop Organizers & Desk Pads
Category:
Description: the monitor stand riser can help !
Attributes: color: white; material: wood; style: modern; width: 23.5; height: 4.64; depth: 9.2
grades from people, short, guidelines: 2 1 1
clear shower curtain products that say clear: 9
products that say clear, graded exact by people: 5
products that say clear, graded exact by the short prompt: 7
products that say clear, graded exact by the guidelines prompt: 7
clear shower curtain products that never say clear: 9
products that never say clear, graded exact by people: 1
products that never say clear, graded exact by the short prompt: 0
products that never say clear, graded exact by the guidelines prompt: 0
exact train pairs missing a query word: 149
share graded lower, short prompt, exact train pairs missing a query word: 0.584
share graded lower, guidelines prompt, exact train pairs missing a query word: 0.483
exact train pairs with every query word: 47
share graded lower, short prompt, exact train pairs with every query word: 0.149
share graded lower, guidelines prompt, exact train pairs with every query word: 0.128
partial train pairs with every query word: 8
share graded higher, short prompt, partial train pairs with every query word: 0.875
share graded higher, guidelines prompt, partial train pairs with every query word: 0.875
partial train pairs missing a query word: 192
share graded higher, short prompt, partial train pairs missing a query word: 0.078
share graded higher, guidelines prompt, partial train pairs missing a query word: 0.120
exact test pairs missing a query word: 128
share graded lower, short prompt, exact test pairs missing a query word: 0.305
share graded lower, guidelines prompt, exact test pairs missing a query word: 0.281
exact test pairs with every query word: 178
share graded lower, short prompt, exact test pairs with every query word: 0.084
share graded lower, guidelines prompt, exact test pairs with every query word: 0.051
partial test pairs with every query word: 42
share graded higher, short prompt, partial test pairs with every query word: 0.548
share graded higher, guidelines prompt, partial test pairs with every query word: 0.667
partial test pairs missing a query word: 199
share graded higher, short prompt, partial test pairs missing a query word: 0.171
share graded higher, guidelines prompt, partial test pairs missing a query word: 0.241
对于训练查询 dual monitor stand,商店的评判者把 abramowski monitor stand 判为 exact,而两个提示词都把它判为 partial(2 1 1,依此顺序);LLM 所见的内容中没有任何文字说明该支架可容纳两台显示器。
对于 clear shower curtain,LLM 的 exact 等级随该词而变。在 LLM 能读到 clear 一词的测试池产品中,共 9 个,评判者把 5 判为 exact,short 提示词判 7,guidelines 提示词判 7。在从未出现该词的 9 个产品中,评判者把 1 判为 exact,short 提示词判 0,guidelines 提示词判 0。
按池统计,这一规律成立。使用 short 提示词时,文本缺少某个查询词的 exact 对被 LLM 打出更低等级的比例,在训练池为 0.584,在测试池为 0.305;文本包含全部查询词的 exact 对,对应比例分别为 0.149 和 0.084。
在测试池中,文本包含全部查询词的对被打出更高等级的比例为 0.548,而缺少某个词的对为 0.171。在测试池中,guidelines 提示词的划分也是如此:exact 对为 0.281,对照 0.051;partial 对为 0.667,对照 0.241。
两个池中常见的情形不同。训练池有 149 个缺词的 exact 对和 8 个含全部词的 partial 对;测试池分别有 128 和 42。依我们的理解,这就是错误方向在两个池之间不同的原因。这项检查比较粗略:复数形式或拼写错误也会被算作缺词。guidelines 提示词明确要求降级:“如果没有提到某个修饰词,就根据所写内容判断,不要假定它已满足。”short 提示词没有这样的规则,也同样会降级。
工作准则: 把评判者所依据的、能转换为文本的每个字段(如属性和完整描述)都提供给 LLM 评判者,并写明它看不到什么。在把某个分歧算作 LLM 的错误之前,先检查证据是否在它所见的内容之中。如果评判者的证据是视觉性的,就要像本例在两个池中所做的那样,衡量 LLM 是否给 exact 产品打出更低的等级;或者,若你的模型支持图片,就把图片提供给它。
衡量提示词对标注结果的影响
LLM 评判器是模型与提示词的组合。Thomas 等人对同一提示词的 42 种改写进行了测试,发现平均 kappa 的分布从 0.50 到 0.72,其中 0.50 被他们称为中等一致性。他们提醒说,任何单个提示词的得分,都只是评判器可能达到的范围中的一次抽样(Thomas et al., 2024)。
Arabzadeh 和 Clarke 在 TREC 深度学习(Deep Learning)数据集上,使用三个 LLM 和 72 个提示词进行了实验,发现二元方法和成对方法对提示词措辞最不敏感,而分级相关性判断(即本店所采用的类型)对措辞高度敏感(Arabzadeh and Clarke, 2025b)。没有哪一个提示词对所有 LLM 都最优,不过 UMBRELA 在不同 LLM 上的分级评估中始终表现良好。
这两个提示词共用同一套定义,guidelines 则在此基础上增加了五条规则(撰写评判指南)。记录中的每个评判池都显示了两个提示词在哪些地方不一致,而训练池则显示能否分辨哪个提示词与本店评判者的一致性更好。同一查询的样本对并不相互独立,因此区间估计对整条训练查询进行重采样,正如第 3 章的自助法对查询进行重采样一样:
1"""Counts where the two prompts disagree, their kappa with each other, and how far their kappas with people differ across resampled train queries. The prose's point rests on the 95% interval line, which includes 0 for this recording."""2import numpy as np3import pandas as pd4from sklearn.metrics import cohen_kappa_score56from lib.agreement import LABELS, graded_pairs7from lib.store import load_products, load_queries8910def kappa_gap(sample):11 """Kappa with people under the guidelines prompt, minus kappa under the short prompt."""12 human = sample["human"].astype(int)13 return cohen_kappa_score(human, sample["guidelines"].astype(int)) - cohen_kappa_score(human, sample["short"].astype(int))141516order = list(LABELS.values()) # exact, partial, irrelevant17for pool in ["train", "test"]:18 # the test pool includes pairs nobody judged, so these counts cover every pair19 pairs = graded_pairs(pool)20 print(f"\n{pool} pairs where the prompts disagree: {(pairs['short'] != pairs['guidelines']).sum()} of {len(pairs)}\n"21 f"kappa between the prompts, {pool} pool: {cohen_kappa_score(pairs['short'], pairs['guidelines']):.3f}")22 table = pd.crosstab(pairs["short"].map(LABELS).rename(f"short, {pool}"), pairs["guidelines"].map(LABELS).rename("guidelines"))23 table = table.reindex(index=order, columns=order, fill_value=0)24 cells = {f"{row} under short, {column} under guidelines": table.at[row, column] for row in order for column in order if row != column}25 print(f"{table}\nlargest disagreement, {pool} pool: {max(cells, key=cells.get)} ({max(cells.values())})")2627pairs, products, queries = graded_pairs("train"), load_products(), load_queries()28moved = pairs[(pairs["short"] == 1) & (pairs["guidelines"] == 0)] # partial under one prompt, irrelevant under the other29print("\n".join(f"moved train pairs judged {label} by people: {(moved['human'] == grade).sum()}" for grade, label in LABELS.items()))30# the first five queries in order, not a random sample31for query_id, product_id in moved[["query_id", "product_id"]].drop_duplicates("query_id").head(5).itertuples(index=False):32 print(f" {queries.at[query_id, 'query']:<28} {products.at[product_id, 'name'][:50]}")3334# a fixed seed, so the interval is the same on every run35rng, train_queries = np.random.default_rng(2026), pairs["query_id"].unique()36by_query = {query_id: group for query_id, group in pairs.groupby("query_id")}37# each resample draws train queries with replacement, so some repeat and others are left out38gaps = [kappa_gap(pd.concat([by_query[query_id] for query_id in rng.choice(train_queries, len(train_queries))]))39 for _ in range(2000)] # resample whole train queries40low, high = np.percentile(gaps, [2.5, 97.5])41print(f"kappa with people, guidelines minus short: {kappa_gap(pairs):.3f}\n95% interval over resampled train queries: {low:.3f} to {high:.3f}")
python prompt-disagreement.py
train pairs where the prompts disagree: 122 of 550
kappa between the prompts, train pool: 0.668
guidelines exact partial irrelevant
short, train
exact 122 4 1
partial 31 153 75
irrelevant 0 11 153
largest disagreement, train pool: partial under short, irrelevant under guidelines (75)
test pairs where the prompts disagree: 93 of 590
kappa between the prompts, test pool: 0.712
guidelines exact partial irrelevant
short, test
exact 303 11 1
partial 42 174 37
irrelevant 0 2 20
largest disagreement, test pool: partial under short, exact under guidelines (42)
moved train pairs judged exact by people: 18
moved train pairs judged partial by people: 31
moved train pairs judged irrelevant by people: 26
toddler couch fold out eulah 79 '' cotton rolled arm sofa bed
70s inspired furniture angels moroccan window curtains , oriental wavy cu
7qt slow cooker berghoff international eclipse 7 qt non-stick cove
gnome fairy garden fairy house tea pot fairy garden
wall sconce with usb port 2 piece wall sconce set
kappa with people, guidelines minus short: 0.022
95% interval over resampled train queries: -0.042 to 0.086
两个提示词在 122 of 550 个训练样本对和 93 of 590 个测试样本对上意见不一致,二者之间的 kappa 在训练池上为 0.668,在测试池上为 0.712。
最大的分歧在不同评判池中有所不同。在训练池上是 partial under short, irrelevant under guidelines (75);在测试池(即各排序器的前 10 名)上是 partial under short, exact under guidelines (42)。
在 short 判为部分相关、而 guidelines 判为不相关的训练样本对中,本店的评判者判为不相关的有 26 个,这与 guidelines 一致;判为部分相关的有 31 个,这与 short 一致;判为完全匹配的有 18 个。示例列出的变动样本对中,有针对 toddler couch fold out 的沙发床,以及针对 70s inspired furniture 的窗帘。记录保存的是标签而不是理由,因此我们根据这类名称推断,这种变动源于 guidelines 中把产品类型错误判为不相关的那条规则。
按与本店评判者的 kappa 衡量,guidelines 比 short 高出 0.022,对训练查询重采样得到的 95% 区间为 -0.042 to 0.086。该区间包含 0,因此训练池无法区分这两个提示词,本店将两者都保留。
工作准则: 把提示词视为评判器的一部分。逐字记录每个提示词,并同时记录模型版本、设置和答案模式,本次记录在其清单和脚本中就是这样做的。提示词一旦改动,就把它当作另一个评判器,重新衡量其一致性。至少运行两个提示词,只有两者都得出同一结论时,才信任基于 LLM 评分的结论。选择某个提示词而不选另一个时,应以训练样本对为依据,并且只有当二者差值的区间不包含 0 时才可如此选择。
这部分尚未翻译,因此以英文显示。
Fill unjudged pairs with LLM grades
We consider filling the holes in the human judgments the least risky use of an LLM judge. Keep every human grade, and use the LLM only for pairs nobody judged, which a metric would otherwise count as grade 0 (Count unjudged products). The LLM then decides only what was a guess before.
Of the 590 pairs in the top 10s of chapter 1's four rankers, 26 are unjudged, and BM25's top 10s hold 22 of them, against 3 for semantic retrieval and 6 for the hybrid.
Let each prompt fill them, rescore the four rankers on nDCG@10 (Score one ranking), and rerun chapter 3's planned comparisons on the filled grades:
1"""Fills the pairs nobody judged with each prompt's grades, then rescores the four rankers and the planned verdicts with the filled grades. The prose's point rests on the 'that change' lines, which list the rankers and verdicts that move when the holes are filled."""2from lib.agreement import LABELS, PLANNED, PROMPTS, chapter_one_runs, fill_unjudged, graded_pairs, llm_qrels, planned_verdicts3from lib.metrics import evaluate4from lib.store import load_products, load_queries, qrels567def order(scores):8 """Rankers by mean nDCG@10 over the test queries, best first."""9 return sorted(scores, key=lambda name: scores[name].mean(), reverse=True)101112products, queries, people = load_products(), load_queries("test"), qrels("test")13runs = chapter_one_runs(products, queries)14pairs = graded_pairs("test") # every product in the top 10 of the four runs, per test query15unjudged = pairs[pairs["human"].isna()]16print(f"pairs in the test pool: {len(pairs)}\nof them, unjudged: {len(unjudged)}")17for name, run in runs.items():18 # people[query_id] holds the human judgments for a query, so a product missing from it was never judged19 missing = sum(product_id not in people[query_id] for query_id, ranking in run.items() for product_id in ranking[:10])20 print(f"unjudged products in the top 10s, {name}: {missing}")21lights = unjudged[unjudged["query_id"] == 133] # outdoor light fixtures22print(f"unjudged products for outdoor light fixtures: {len(lights)}\n"23 f"of them, with bulb in the name: {lights['product_id'].map(products['name']).str.contains('bulb').sum()}")24threshold = 0.05 / len(PLANNED) # chapter 3's rule25human = {name: evaluate(run, people)["ndcg@10"] for name, run in runs.items()} # per-query scores26print(f"order, people's grades only: {' > '.join(order(human))}")2728for prompt in PROMPTS:29 counts = ", ".join(str((unjudged[prompt] == grade).sum()) for grade in LABELS) # grades 2, 1 and 030 print(f"\nunjudged pairs graded exact, partial, irrelevant, {prompt} prompt: {counts}")31 print(f"unjudged bulbs graded irrelevant, {prompt} prompt: {(lights[prompt] == 0).sum()}")32 filled = {name: evaluate(run, fill_unjudged(people, llm_qrels(prompt)))["ndcg@10"] for name, run in runs.items()}33 print("\n".join(f"{name} ndcg@10, filled by the {prompt} prompt: {filled[name].mean():.3f}" for name in runs))34 print(f"order, filled by the {prompt} prompt: {' > '.join(order(filled))}")35 # compares the two orders place by place36 moved = [name for name, place in zip(order(filled), order(human)) if name != place]37 print(f"rankers that change place when filled, {prompt} prompt: {', '.join(moved) or 'none'}")38 changed = [pair for pair, established in planned_verdicts(filled, threshold).items()39 if established != planned_verdicts(human, threshold)[pair]]40 print(f"verdicts that change when filled, {prompt} prompt: {'; '.join(' minus '.join(pair) for pair in changed) or 'none'}")
python fill-unjudged-pairs.pypairs in the test pool: 590
of them, unjudged: 26
unjudged products in the top 10s, bm25: 22
unjudged products in the top 10s, semantic: 3
unjudged products in the top 10s, hybrid: 6
unjudged products in the top 10s, reranked: 6
unjudged products for outdoor light fixtures: 8
of them, with bulb in the name: 8
order, people's grades only: semantic > hybrid > reranked > bm25
unjudged pairs graded exact, partial, irrelevant, short prompt: 4, 17, 5
unjudged bulbs graded irrelevant, short prompt: 0
bm25 ndcg@10, filled by the short prompt: 0.754
semantic ndcg@10, filled by the short prompt: 0.833
hybrid ndcg@10, filled by the short prompt: 0.821
reranked ndcg@10, filled by the short prompt: 0.813
order, filled by the short prompt: semantic > hybrid > reranked > bm25
rankers that change place when filled, short prompt: none
verdicts that change when filled, short prompt: none
unjudged pairs graded exact, partial, irrelevant, guidelines prompt: 5, 9, 12
unjudged bulbs graded irrelevant, guidelines prompt: 7
bm25 ndcg@10, filled by the guidelines prompt: 0.745
semantic ndcg@10, filled by the guidelines prompt: 0.834
hybrid ndcg@10, filled by the guidelines prompt: 0.821
reranked ndcg@10, filled by the guidelines prompt: 0.813
order, filled by the guidelines prompt: semantic > hybrid > reranked > bm25
rankers that change place when filled, guidelines prompt: none
verdicts that change when filled, guidelines prompt: none
The prompts disagree about the unjudged pairs. Counted as exact, partial and irrelevant, short grades them 4, 17, 5, and guidelines 5, 9, 12. They part on the unjudged products for outdoor light fixtures in BM25's top 10 (Score every test query): 8 of the 8 have bulb in their names.
guidelines calls 7 of the bulbs irrelevant, and short calls 0. Chapter 1 argued that a bulb isn't a light fixture. Which reading the store's judges would take, nobody knows until they judge the bulbs.
With the holes filled, BM25 scores 0.754 with short and 0.745 with guidelines, against 0.725 on human grades alone, and semantic retrieval scores 0.833 and 0.834, against 0.830.
The order on human grades alone is semantic > hybrid > reranked > bm25. The rankers that change place when short fills the holes are none, and when guidelines does, none. The planned comparisons' verdicts that change are none under short and none under guidelines.
Under human grades, the rankers' mean judged@10, the share of each top 10 that people judged, runs from 0.927 for BM25 to 0.990 for semantic retrieval (Compare runs in a harness).
How far can the filled grades be trusted? The closest evidence is the judged pairs of the same pool, where the LLM's kappa with the store's judges is 0.571 for short and 0.470 for guidelines. On those judged pairs, its grade is lower than the judges' on 61 pairs and higher on 65 with short, and lower on 77 and higher on 83 with guidelines (Read the direction of the LLM's errors).
Working rule: Fill the unjudged pairs in your rankers' top 10s with LLM grades as a diagnostic, under every prompt you run, and report the filled score beside the human-only one with the number of pairs filled. If filling changes the order of two rankers or a verdict, have people judge those pairs. Never let a filled grade replace a human one.
检查 LLM 评分是否以相同方式对排序器排序
两组判断即使在许多标签上不一致,仍可能对系统给出相近的排名。Voorhees 用不同的相关性判断集合为 TREC 的运行结果打分,发现“使用不同相关性判断集合得到的系统排名之间,相关性非常高”(Voorhees, 1998)。LLM 评判者也表现出同样的分离。在 TREC 2023 年 Deep Learning 赛道的 42 组 LLM 标签中,kappa 从约 0.06 到 0.29 不等,而系统排名的相关性始终在约 0.83 到 0.95 之间(Rahmani et al., 2025, Table 3)。
这种相关性就是 Kendall's tau。当两种排序对一对系统的先后判断相同时,这一对是一致对;当两种排序对它的先后判断相反时,这一对是不一致对。tau 等于一致对数减去不一致对数,再除以全部对数,取值从表示顺序相同的 1 到表示顺序完全相反的 −1。SciPy 的 kendalltau 计算的是 τ_b,这种形式还能处理并列的情况(SciPy documentation)。
Yilmaz 等人指出,tau 在 0.9 以上的排名,常被视为实际上等价,至少在经验上如此(Yilmaz et al., 2008)。Sanderson 和 Soboroff 发现,用这样的阈值来下结论,其可靠程度不及该领域的设想(Sanderson and Soboroff, 2007)。
按 nDCG@10 为第 1 章的四个排序器打分,分别使用人工评分和每个提示词对应的 LLM 评分,就能看出 LLM 对它们的排序是否与人一致。LLM 只评判了测试池,即这些排序器前 10 名中的商品,因此还有一个对照,只用测试池中的商品对,以人工评分为它们打分。四个排序器共有 6 对,所以交换一对会使 tau 变化 0.333,tau 只能取七个值。这很粗略。一个在单个商品对上与人相差这么远的评判者,仍可能与人以相同的方式对排序器排序。它会这样吗?请先给出你的答案:
1"""Scores the four rankers under human grades, under people's grades on the LLM's pairs as a control, and under each prompt, then counts the swapped pairs for each set. The prose's point rests on the 'swapped under' lines, read with the tau lines beside them."""2from itertools import combinations34from scipy.stats import kendalltau56from lib.agreement import chapter_one_runs, llm_qrels7from lib.metrics import evaluate8from lib.store import load_products, load_queries, qrels910products, queries, people = load_products(), load_queries("test"), qrels("test")11runs = chapter_one_runs(products, queries)12short, guidelines = llm_qrels("short"), llm_qrels("guidelines")13pooled = {query_id: {product_id: grade for product_id, grade in people[query_id].items() if product_id in short[query_id]}14 for query_id in people} # people's grades for the pairs the LLM judged, and no others15judgment_sets = {"people": people, "people on the LLM's pairs": pooled, "the short prompt": short,16 "the guidelines prompt": guidelines}17means = {name: {ranker: evaluate(run, judgments)["ndcg@10"].mean() for ranker, run in runs.items()}18 for name, judgments in judgment_sets.items()}19for name, scores in means.items():20 print(f"order under {name}: {' > '.join(sorted(scores, key=scores.get, reverse=True))}")21 for ranker, score in scores.items():22 print(f"{ranker} ndcg@10 under {name}: {score:.3f}")2324orders = [sorted(scores, key=scores.get, reverse=True) for scores in means.values()]25steady = [ranker for position, ranker in enumerate(orders[0]) if all(order[position] == ranker for order in orders)]26print(f"rankers in the same place under every set of grades: {', '.join(steady) or 'none'}")27control = means["people on the LLM's pairs"] # the same pool, human grades28for name in ["the short prompt", "the guidelines prompt"]:29 print("\n".join(f"{ranker}, {name} minus people on the LLM's pairs: {means[name][ranker] - control[ranker]:+.3f}" for ranker in runs))30# people's scores are the reference that every other set of grades is compared with31pairs, reference = list(combinations(runs, 2)), means["people"]32# one swap turns a concordant pair into a discordant one, which moves tau by 2 over the number of pairs33print(f"\npairs of rankers: {len(pairs)}\nchange in tau from one swapped pair: {2 / len(pairs):.3f}")34for name, scores in list(means.items())[1:]:35 swapped = [(first, second) for first, second in pairs # discordant: opposite signs36 if (reference[first] - reference[second]) * (scores[first] - scores[second]) < 0]37 print(f"swapped under {name}: {'; '.join(f'{first} and {second}' for first, second in swapped) or 'none'}")38 # concordant minus discordant, over all pairs; the discordant count is the number of swaps39 print(f"tau by hand, people against {name}: {(len(pairs) - 2 * len(swapped)) / len(pairs):.3f}")40 # SciPy's tau_b, which handles ties; .statistic is the tau value41 print(f"tau from scipy, people against {name}: {kendalltau(list(reference.values()), list(scores.values())).statistic:.3f}")
python kendall-tau-four-rankers.pyorder under people: semantic > hybrid > reranked > bm25
bm25 ndcg@10 under people: 0.725
semantic ndcg@10 under people: 0.830
hybrid ndcg@10 under people: 0.812
reranked ndcg@10 under people: 0.798
order under people on the LLM's pairs: semantic > hybrid > reranked > bm25
bm25 ndcg@10 under people on the LLM's pairs: 0.795
semantic ndcg@10 under people on the LLM's pairs: 0.904
hybrid ndcg@10 under people on the LLM's pairs: 0.885
reranked ndcg@10 under people on the LLM's pairs: 0.869
order under the short prompt: reranked > semantic > hybrid > bm25
bm25 ndcg@10 under the short prompt: 0.832
semantic ndcg@10 under the short prompt: 0.903
hybrid ndcg@10 under the short prompt: 0.900
reranked ndcg@10 under the short prompt: 0.906
order under the guidelines prompt: reranked > semantic > hybrid > bm25
bm25 ndcg@10 under the guidelines prompt: 0.805
semantic ndcg@10 under the guidelines prompt: 0.878
hybrid ndcg@10 under the guidelines prompt: 0.876
reranked ndcg@10 under the guidelines prompt: 0.886
rankers in the same place under every set of grades: bm25
bm25, the short prompt minus people on the LLM's pairs: +0.037
semantic, the short prompt minus people on the LLM's pairs: -0.001
hybrid, the short prompt minus people on the LLM's pairs: +0.016
reranked, the short prompt minus people on the LLM's pairs: +0.037
bm25, the guidelines prompt minus people on the LLM's pairs: +0.010
semantic, the guidelines prompt minus people on the LLM's pairs: -0.026
hybrid, the guidelines prompt minus people on the LLM's pairs: -0.008
reranked, the guidelines prompt minus people on the LLM's pairs: +0.017
pairs of rankers: 6
change in tau from one swapped pair: 0.333
swapped under people on the LLM's pairs: none
tau by hand, people against people on the LLM's pairs: 1.000
tau from scipy, people against people on the LLM's pairs: 1.000
swapped under the short prompt: semantic and reranked; hybrid and reranked
tau by hand, people against the short prompt: 0.333
tau from scipy, people against the short prompt: 0.333
swapped under the guidelines prompt: semantic and reranked; hybrid and reranked
tau by hand, people against the guidelines prompt: 0.333
tau from scipy, people against the guidelines prompt: 0.333
人工评分给出 semantic > hybrid > reranked > bm25。对照组,即人工评分在 LLM 的商品对上,保持了这一顺序,tau 为 1.000,所以把评分限制在测试池内不会使任何排序器的位置改变。它确实抬高了每个得分,BM25 从 0.725 升到 0.795,因为理想的前 10 名只能包含入池的商品。
short 下的顺序是 reranked > semantic > hybrid > bm25,guidelines 下的顺序是 reranked > semantic > hybrid > bm25。short 交换了 semantic and reranked; hybrid and reranked,guidelines 交换了 semantic and reranked; hybrid and reranked。
short 的 tau 手算为 0.333,用 SciPy 算得 0.333;guidelines 的 tau 为 0.333。
与对照组相比,LLM 评分使四个排序器的变化幅度和方向都不相同。在 short 下,BM25 变化 +0.037,语义检索变化 -0.001,混合排序器变化 +0.016,重排序的混合排序器变化 +0.037。在 guidelines 下,变化分别为 +0.010、-0.026、-0.008 和 +0.017。评判者抬高了一些排序器,又压低了另一些,这使它们彼此越过对方的位置。
在 short 下,语义检索、混合排序器和重排序的混合排序器几乎并列,分别为 0.903、0.900 和 0.906。tau 不考虑两个排序器得分相差多远,所以近乎并列的排序器之间的一次交换,与最好和最差排序器之间的一次交换,计数相同。
tau 对顶部交换和底部交换也一视同仁,Yilmaz 等人指出了这一弱点(Yilmaz et al., 2008)。Yilmaz 等人谈到他们比较的 TREC 运行结果时指出,这些系统“性能非常相似。因此,Kendall’s τ 统计量可能会受这些系统影响”(Yilmaz et al., 2012)。
在每组评分下都保持位置不变的排序器是 bm25。顺序的差异只出现在人工评分给出 0.830、0.812 和 0.798 的三个排序器之间,而这些差距没有任何一项计划内比较能够确立(重新运行计划内比较)。
工作准则: 报告 tau 时要附上被交换的排序器对列表,不要单独给出。排序器只有寥寥几个时,一次交换就会使 tau 变化 0.333,不要拿它与 0.9 比较。然后用每组评分下的计划内比较检验被交换的排序器对之间是否存在差异。
用 LLM 评分重跑计划内比较
决策依据的是本书的规则所给出的结论,而不是tau 所比较的排序。如果使用 LLM 评分,商店会得到相同的结论吗?可以预期问题出现在排名靠前的部分:Otero 等人发现,“基于 LLM 的判断在对表现最好的系统排序时并不公平”,并且“在统计差异方面的假阳性率极高”(Otero et al., 2025)。
第 3 章的三项计划内比较使用的阈值为 0.0167,区间水平为 0.9833。在测试池上,分别用人工评分和每种提示词的评分重跑这三项比较,并加入第 3 章中重排序混合检索与语义检索的探索性比较,即可看出商店是否会上线相同的排序器:
1"""Runs chapter 3's three planned comparisons, and the exploratory one, under human grades and under each prompt's grades on the test pool. The prose's point rests on the 'verdicts that differ' lines and the count of disagreeing checks."""2from lib.agreement import PLANNED, PROMPTS, chapter_one_runs, llm_qrels # PLANNED: chapter 3's, each stage against the last3from lib.metrics import evaluate4from lib.significance import compare, verdict5from lib.store import load_products, load_queries, qrels67EXPLORATORY = [("reranked", "semantic")] # chosen after seeing test means8threshold = 0.05 / len(PLANNED) # chapter 3's threshold, so that a verdict means the same either way9print(f"planned comparisons: {len(PLANNED)}\np-value threshold: {threshold:.4f}\ninterval level: {1 - threshold:.4f}")1011runs = chapter_one_runs(load_products(), load_queries("test"))12judgment_sets = {"people": qrels("test"), **{f"{prompt} prompt": llm_qrels(prompt) for prompt in PROMPTS}}13established, disagreements = {}, {judges: 0 for judges in judgment_sets}14for judges, judgments in judgment_sets.items():15 ndcg = {ranker: evaluate(run, judgments)["ndcg@10"] for ranker, run in runs.items()}16 for after, before in PLANNED + EXPLORATORY:17 # the interval's level matches the threshold, so each interval and p-value follow one rule18 result, name = compare(ndcg[after], ndcg[before], confidence=1 - threshold), f"{after} minus {before}, {judges}"19 print(f"\n{name}: {result['mean difference']:.3f}\n{name}, permutation p: {result['permutation p']:.3f}")20 # the exploratory comparison gets no t-test, interval or verdict21 if (after, before) in PLANNED:22 print(f"{name}, t-test p: {result['t-test p']:.3f}\n{name}, interval: {result['ci low']:.3f} to {result['ci high']:.3f}")23 checks = verdict(result, threshold)24 established[(judges, after, before)] = checks["established"]25 disagree = [check for check in ["t-test", "interval"] if not checks[f"{check} agrees"]]26 disagreements[judges] += bool(disagree)27 print(f"{name}, established: {'yes' if checks['established'] else 'no'}\n"28 f"{name}, disagrees with the permutation test: {' and '.join(disagree) or 'neither'}")2930print()31for judges, count in disagreements.items():32 print(f"planned comparisons where the t-test or interval disagrees, {judges}: {count}")33for prompt in PROMPTS:34 differ = [pair for pair in PLANNED if established[(f"{prompt} prompt", *pair)] != established[("people", *pair)]]35 print(f"verdicts that differ from people's, {prompt} prompt: {'; '.join(f'{after} minus {before}' for after, before in differ) or 'none'}")
python planned-comparisons-llm-grades.pyplanned comparisons: 3
p-value threshold: 0.0167
interval level: 0.9833
semantic minus bm25, people: 0.105
semantic minus bm25, people, permutation p: 0.001
semantic minus bm25, people, t-test p: 0.003
semantic minus bm25, people, interval: 0.047 to 0.205
semantic minus bm25, people, established: yes
semantic minus bm25, people, disagrees with the permutation test: neither
hybrid minus semantic, people: -0.018
hybrid minus semantic, people, permutation p: 0.125
hybrid minus semantic, people, t-test p: 0.123
hybrid minus semantic, people, interval: -0.048 to 0.006
hybrid minus semantic, people, established: no
hybrid minus semantic, people, disagrees with the permutation test: neither
reranked minus hybrid, people: -0.014
reranked minus hybrid, people, permutation p: 0.382
reranked minus hybrid, people, t-test p: 0.375
reranked minus hybrid, people, interval: -0.053 to 0.020
reranked minus hybrid, people, established: no
reranked minus hybrid, people, disagrees with the permutation test: neither
reranked minus semantic, people: -0.032
reranked minus semantic, people, permutation p: 0.048
semantic minus bm25, short prompt: 0.070
semantic minus bm25, short prompt, permutation p: 0.010
semantic minus bm25, short prompt, t-test p: 0.014
semantic minus bm25, short prompt, interval: 0.016 to 0.146
semantic minus bm25, short prompt, established: yes
semantic minus bm25, short prompt, disagrees with the permutation test: neither
hybrid minus semantic, short prompt: -0.002
hybrid minus semantic, short prompt, permutation p: 0.851
hybrid minus semantic, short prompt, t-test p: 0.861
hybrid minus semantic, short prompt, interval: -0.029 to 0.036
hybrid minus semantic, short prompt, established: no
hybrid minus semantic, short prompt, disagrees with the permutation test: neither
reranked minus hybrid, short prompt: 0.006
reranked minus hybrid, short prompt, permutation p: 0.727
reranked minus hybrid, short prompt, t-test p: 0.714
reranked minus hybrid, short prompt, interval: -0.029 to 0.042
reranked minus hybrid, short prompt, established: no
reranked minus hybrid, short prompt, disagrees with the permutation test: neither
reranked minus semantic, short prompt: 0.003
reranked minus semantic, short prompt, permutation p: 0.847
semantic minus bm25, guidelines prompt: 0.073
semantic minus bm25, guidelines prompt, permutation p: 0.045
semantic minus bm25, guidelines prompt, t-test p: 0.057
semantic minus bm25, guidelines prompt, interval: 0.006 to 0.183
semantic minus bm25, guidelines prompt, established: no
semantic minus bm25, guidelines prompt, disagrees with the permutation test: interval
hybrid minus semantic, guidelines prompt: -0.001
hybrid minus semantic, guidelines prompt, permutation p: 0.929
hybrid minus semantic, guidelines prompt, t-test p: 0.945
hybrid minus semantic, guidelines prompt, interval: -0.045 to 0.037
hybrid minus semantic, guidelines prompt, established: no
hybrid minus semantic, guidelines prompt, disagrees with the permutation test: neither
reranked minus hybrid, guidelines prompt: 0.010
reranked minus hybrid, guidelines prompt, permutation p: 0.657
reranked minus hybrid, guidelines prompt, t-test p: 0.609
reranked minus hybrid, guidelines prompt, interval: -0.024 to 0.076
reranked minus hybrid, guidelines prompt, established: no
reranked minus hybrid, guidelines prompt, disagrees with the permutation test: neither
reranked minus semantic, guidelines prompt: 0.009
reranked minus semantic, guidelines prompt, permutation p: 0.581
planned comparisons where the t-test or interval disagrees, people: 0
planned comparisons where the t-test or interval disagrees, short prompt: 0
planned comparisons where the t-test or interval disagrees, guidelines prompt: 1
verdicts that differ from people's, short prompt: none
verdicts that differ from people's, guidelines prompt: semantic minus bm25
| 指标 | 评分 | 语义检索减 BM25 | 混合检索减语义检索 | 重排序减混合检索 |
|---|---|---|---|---|
| 平均差 | 人工 | 0.105 | -0.018 | -0.014 |
| 平均差 | short |
0.070 | -0.002 | 0.006 |
| 平均差 | guidelines |
0.073 | -0.001 | 0.010 |
| 区间 | 人工 | 0.047 to 0.205 | -0.048 to 0.006 | -0.053 to 0.020 |
| 区间 | short |
0.016 to 0.146 | -0.029 to 0.036 | -0.029 to 0.042 |
| 区间 | guidelines |
0.006 to 0.183 | -0.045 to 0.037 | -0.024 to 0.076 |
| 置换检验 p 值 | 人工 | 0.001 | 0.125 | 0.382 |
| 置换检验 p 值 | short |
0.010 | 0.851 | 0.727 |
| 置换检验 p 值 | guidelines |
0.045 | 0.929 | 0.657 |
| t 检验 p 值 | 人工 | 0.003 | 0.123 | 0.375 |
| t 检验 p 值 | short |
0.014 | 0.861 | 0.714 |
| t 检验 p 值 | guidelines |
0.057 | 0.945 | 0.609 |
| 是否成立 | 人工 | yes | no | no |
| 是否成立 | short |
yes | no | no |
| 是否成立 | guidelines |
no | no | no |
商店唯一成立的结果能否在 LLM 评分下保持,取决于提示词。在 guidelines 下,结论不同的是 semantic minus bm25,也就是第 3 章确立的那一项结果。在 short 下,与人工评分结论不同的是 none。
在人工评分下,t 检验或区间与置换检验不一致的计划内比较有 0 项;在 short 下有 0 项;在 guidelines 下有 1 项。对于 guidelines 下的语义检索减 BM25,不一致的检验是interval。
比较排序器的顺序发现了两对互换的排序器。其中一对,即重排序混合检索与混合检索,属于计划内比较,而上表显示它在任何一组评分下都不成立。另一对,即重排序混合检索与语义检索,属于探索性比较。在人工评分下,重排序混合检索减语义检索为 -0.032(置换检验 p 值 0.048)。在 LLM 评分下,short 为 0.003,guidelines 为 0.009。第 3 章是在看到测试均值之后才选定这项比较的,因此无论结果如何,它都不能确立任何结论。
LLM 评判者还可能偏向使用 LLM 的排序器:Balog 等人报告了“LLM 评判者对基于 LLM 的排序器表现出显著偏向的首个实证证据”(Balog et al., 2025)。第 1 章的排序器不使用 LLM,因此本章无法衡量这种偏向。本书对第 8 章和第 9 章中基于 LLM 的排序器,采用人工评分来打分。
工作准则: 绝不能仅凭 LLM 评分得出的结论上线变更。用 LLM 评分来决定哪些候选值得评判,再依据人工评分做出决定。当两种提示词给出不同的结论时,LLM 无法裁定这项比较,必须由人来裁定。
用人工样本校正 LLM 评分
一条折中的路径同时使用两种评分:由 LLM 为每条查询评分,由人工判断一份随机抽取的查询样本,再用这份样本校正 LLM 的误差。预测驱动推断(prediction-powered inference)借助人工样本,对模型的预测进行严格校正,其置信区间“无需对提供预测的机器学习算法做任何假设”(Angelopoulos et al., 2023)。就均值而言,这一思路的简单形式是:取 LLM 在全部查询上的均值,再加上人工样本中“人工评分减 LLM 评分”的平均值:
corrected = mean over all queries of the LLM score + mean over the human sample of (human score − LLM score)
在搜索评估中,Oosterhuis 等人用这种方式构建指标的区间,并提醒说“生成的相关性标注并非不受(系统性)误差的影响”(Oosterhuis et al., 2024)。Takehi 等人则在预算限定下,让 LLM 挑选交由人工判断的配对,再用这些人工判断对 LLM 进行校准(Takehi et al., 2025)。
假设商店有能力为 10 of 30 条测试查询取得人工判断。该示例抽取 2,000 组这一规模的随机集合,并对每一组校正 LLM 在各个提示词下对第 1 章中各排序器之间两个差距的估计:语义检索减 BM25,以及混合检索减语义检索。随后,它把每个校正后的估计与仅凭人工样本得到的结果相比较。离散度(spread)是某个估计在各次抽样上的标准差。这一模拟中没有任何人为选择:
1"""Draws random sets of human-judged test queries and corrects the LLM's estimate of two ranker gaps with each set. The prose's point rests on the 'smaller than the human sample's' lines, read with the correlation lines beside them."""2import numpy as np3import pandas as pd45from lib.agreement import PROMPTS, chapter_one_runs, llm_qrels6from lib.metrics import evaluate7from lib.store import load_products, load_queries, qrels89JUDGED, DRAWS = 10, 2000 # test queries people judge in each draw, and the number of draws10runs = chapter_one_runs(load_products(), load_queries("test"))11people = pd.DataFrame({ranker: evaluate(run, qrels("test"))["ndcg@10"] for ranker, run in runs.items()})12rng = np.random.default_rng(2026)13draws = [rng.choice(people.index, JUDGED, replace=False) for _ in range(DRAWS)] # which queries people judge14print(f"queries judged by people in each draw: {JUDGED} of {len(people)}\ndraws: {DRAWS:,}")1516for after, before in [("semantic", "bm25"), ("hybrid", "semantic")]:17 gap_people = people[after] - people[before] # per-query gaps under people's grades18 print(f"\n{after} minus {before}, people, all queries: {gap_people.mean():.3f}")19 # the spread: the standard deviation over the draws, with numpy's default divisor20 alone = np.std([gap_people[judged].mean() for judged in draws]) # the human sample used alone21 print(f"{after} minus {before}, human sample alone, spread: {alone:.3f}")22 for prompt in PROMPTS:23 llm = {ranker: evaluate(run, llm_qrels(prompt))["ndcg@10"] for ranker, run in runs.items()}24 gap_llm, name = llm[after] - llm[before], f"{after} minus {before}, {prompt} prompt"25 # the LLM's mean over all queries, plus the human sample's average of human minus LLM26 corrected = [gap_llm.mean() + (gap_people[judged] - gap_llm[judged]).mean() for judged in draws]27 print(f"{name}, LLM only: {gap_llm.mean():.3f}")28 print(f"{name}, corrected, mean over draws: {np.mean(corrected):.3f}")29 print(f"{name}, corrected, spread: {np.std(corrected):.3f}")30 print(f"{name}, corrected spread smaller than the human sample's: {'yes' if np.std(corrected) < alone else 'no'}")31 # corrcoef returns a 2×2 matrix; [0, 1] is the correlation between the two lists32 print(f"{name}, correlation of per-query gaps with people's: {np.corrcoef(gap_llm, gap_people)[0, 1]:.3f}")
python correct-llm-scores-with-humans.pyqueries judged by people in each draw: 10 of 30
draws: 2,000
semantic minus bm25, people, all queries: 0.105
semantic minus bm25, human sample alone, spread: 0.045
semantic minus bm25, short prompt, LLM only: 0.070
semantic minus bm25, short prompt, corrected, mean over draws: 0.104
semantic minus bm25, short prompt, corrected, spread: 0.036
semantic minus bm25, short prompt, corrected spread smaller than the human sample's: yes
semantic minus bm25, short prompt, correlation of per-query gaps with people's: 0.641
semantic minus bm25, guidelines prompt, LLM only: 0.073
semantic minus bm25, guidelines prompt, corrected, mean over draws: 0.104
semantic minus bm25, guidelines prompt, corrected, spread: 0.039
semantic minus bm25, guidelines prompt, corrected spread smaller than the human sample's: yes
semantic minus bm25, guidelines prompt, correlation of per-query gaps with people's: 0.692
hybrid minus semantic, people, all queries: -0.018
hybrid minus semantic, human sample alone, spread: 0.016
hybrid minus semantic, short prompt, LLM only: -0.002
hybrid minus semantic, short prompt, corrected, mean over draws: -0.017
hybrid minus semantic, short prompt, corrected, spread: 0.023
hybrid minus semantic, short prompt, corrected spread smaller than the human sample's: no
hybrid minus semantic, short prompt, correlation of per-query gaps with people's: 0.168
hybrid minus semantic, guidelines prompt, LLM only: -0.001
hybrid minus semantic, guidelines prompt, corrected, mean over draws: -0.017
hybrid minus semantic, guidelines prompt, corrected, spread: 0.024
hybrid minus semantic, guidelines prompt, corrected spread smaller than the human sample's: no
hybrid minus semantic, guidelines prompt, correlation of per-query gaps with people's: 0.342
对于语义检索减 BM25,仅用 LLM 在 short 下估计为 0.070,而基于人工评分的结果是 0.105。校正之后,该估计在各次抽样上的平均值为 0.104,因此校正消除了 LLM 的偏差。
其离散度为 0.036,而仅用人工样本时为 0.045。校正后的离散度是否小于人工样本的离散度?在 short 下为 yes,在 guidelines 下为 yes。LLM 对其余查询的评分提供了额外信息,因为它对各查询所得的差距与人工所得的差距之间的相关系数为 0.641。
对于混合检索减语义检索,校正后的离散度在 short 下为 0.023,在 guidelines 下为 0.024,而仅用人工样本时为 0.016。它是否小于人工样本的离散度?在 short 下为 no,在 guidelines 下为 no。在 short 下,各查询差距的相关系数为 0.168。
校正能够修正评分的水平,但只有在 LLM 的逐查询评分与人工评分相吻合时,才会缩小不确定性;而在这里,两个差距中较为接近的那个,二者吻合得最差。
工作准则: 每当你报告基于 LLM 评分的分数时,都应让人工判断一份随机抽取的查询样本,并报告校正后的估计。在这份样本上,检查 LLM 的逐查询差距与人工差距的相关程度。相关性较弱时,应把预算用于判断更多的查询。
生产环境中会出什么问题
**评判器会在没有通知的情况下发生变化。**用 LLM 做评判的存储会反复运行它,因此模型版本、量化(模型权重的存储精度)或提示词措辞只要有一项不同,就是另一个评判器。录制数据中的 manifest.json 固定了模型的摘要值以及两份提示词的原文。答案模式和上下文长度则位于 recordings/llm.py 和 recordings/llm_judgments.py 中,所以也要把它们记录下来。其中任何一项发生变化时,都要在同一份人工样本上重新运行一致性检查。
**针对评判器调优的系统,在该评判器上得分很高。**Clarke 和 Dietz 构造了一个利用 LLM 评判器的运行结果,它在自动评估下的排名("5th")显著高于人工评估下的排名("28th")(Clarke and Dietz, 2024)。如果排序器是依据 LLM 评分调优的,或者建立在同一个 LLM 之上,就需要用人工评分来给出它的核心得分。
**商品文本可以左右评判器。**Alaofi 等人发现,LLM 评判器可能被植入无关文本中的查询词误导,也可能被写入其中的指令误导(Alaofi et al., 2024)。卖家可以写出一段对评判器"说话"的描述。每次评估都要保留一份人工样本,这样评判器一旦被带偏,就会表现为一致性下降。
**在测试集上校准,就是在测试集上调优。**不断改写提示词,直到它与测试查询上的人工评分一致,这样得到的一致性就毫无意义。应当在训练查询上选择提示词,并在测试查询上报告一致性。
**人工评分也不是真相。**在 TREC Podcast 赛道中,LLM 与 TREC 评估员分歧最大的那些样本对,经专家重新评判后,更多时候站在 LLM 一边(Mansour et al., 2026)。应把人工评分视为你所选定的参照,在归咎于任何一方之前,先阅读这些分歧。
**工作准则:**固定评判器,每次评估都保留人工样本,任何环节发生变化时重新运行一致性检查,并且绝不用调优时所用的评判器给排序器打分。
本章做出的决定
除非后续章节给出理由,否则评测库保留这些决定。
- 每个标题分数都用人工评分。 LLM 评分只用作诊断:填补空缺、筛选候选和核对结论。第 8 章和第 9 章中基于 LLM 的排序器,都按人工评分计分。
- 关于评判器的每个发现都要写明所用的评测池。 kappa 与线性加权 kappa(旁附观测一致率)、严重偏差和标签构成,先在人工评判过的训练对上度量,再对测试池中已评判的对报告。方向和最大误差在每个评测池上分别读取,只有排序器的前 10 名才能说明排序器的分数。
- 两个提示词,逐字记录。 训练池无法区分二者,所以评测库两个都保留,只有两个提示词都给出同一结论时,才信任来自 LLM 评分的结论。
- 用 LLM 评分填补未评判的对,仅作诊断, 并与只用人工评分的分数并列报告。在测试查询上,填补不改变任何排序器的名次,也不改变任何结论。
- 只在相同评分下比较分数。 在同一评测池上,LLM 评分对各排序器 nDCG@10 的改变幅度并不相同,方向也不一致,因此 LLM 评分下的分数绝不与人工评分下的分数相比。
- 先看 tau 及其交换对,再按第 3 章的阈值做计划内的比较。 只有四个排序器时,tau 较为粗略,由结论来决定。
- 来自 LLM 评分的分数要附带一份人工抽样的查询, 用来修正该分数,并附带相关性,说明 LLM 的评分是否缩小了不确定性。
- 本章表明: LLM 评判器可以填补空缺和筛选候选,但它的评分对各排序器的影响并不均匀,因此重要的结论仍需要人工评分。
本章,在本版次中
版次 2026.11.10示例于 2026年11月19日 通过本章 11个示例 个示例中有 11 个通过
译自英文版图书,版次 2026.11.10。