衡量搜索
阅读约需 81分钟 分钟,13节中含12个示例。
读完本章,你将能够从零开始计算标准的排序指标,并用 scikit-learn 加以核对;在比较排序器之前选定一个指标;并判断两个排序器在你的测试查询上的差距是已经确立,还是可能出于偶然。
第 1 章最后给出了一个排序,其中语义检索是需要超越的排序器。在商店的测试查询上(这些搜索没有被用来调优任何排序器),BM25 的 nDCG@10 为 0.725,语义检索为 0.830,混合排序器为 0.812,经过重排的混合排序器为 0.798。在信任这一排序之前,必须回答两个问题。这些指标衡量的是否是购物者在意的东西,它们的计算是否正确?此外,这些差距是真实的,还是在 30 条查询上仅凭偶然就可能出现?本章回答这两个问题,并以本书关于“已确立的差异”的规则和一个应用该规则的测试框架作结。
这部分尚未翻译,因此以英文显示。
Count the relevant products in the top k
Most ranking metrics start from one question: how many of the results a shopper sees are relevant? Precision at k, written precision@k, is the share of the top k results that are relevant. Recall at k, recall@k, is the share of all relevant products that appear in the top k. Manning and colleagues describe precision as the share of retrieved documents that are relevant and recall as the share of relevant documents that are retrieved, and they note that raising one usually costs the other (Manning et al., 2008, section 8.3). Both need a yes-or-no idea of relevant. In this book a product is relevant to a query when its judgment is exact, grade 2. A product nobody judged for the query counts as not relevant.
BM25's top 10 for the store's test query clear shower curtain mixes clear curtains, curtains that aren't clear, and shower curtain hooks. How many of those ten are exact, and what is the most any test query could score? The example answers both by hand and checks the count against lib.metrics:
1"""Counts the exact products in BM25's top 10 for one query, computes precision and recall by hand, and checks them against lib.metrics, then finds the best score each test query could reach. The prose's point rests on the "exact in the top 10" and "exact products judged" lines, and on the ceiling lines after the blank line."""2import pandas as pd34from lib.lexical import LexicalSearch5from lib.metrics import precision_at_k, recall_at_k6from lib.store import load_products, load_queries, qrels78queries, judgments = load_queries("test"), qrels("test")9ranking = list(LexicalSearch(load_products()).search(queries.at[246, "query"], k=10).index) # clear shower curtain10# grade 2 is exact; a product the judges never graded is absent from this dict, so it is not exact11exact = {product_id for product_id, grade in judgments[246].items() if grade == 2}1213found = sum(product_id in exact for product_id in ranking[:10])14print("exact in the top 10:", found)15print("exact products judged:", len(exact))16print(f"precision@10 by hand: {found / 10:.3f}")17print(f"recall@10 by hand: {found / len(exact):.3f}")18print(f"precision@10 from lib.metrics: {precision_at_k(ranking, judgments[246], 10):.3f}")19print(f"recall@10 from lib.metrics: {recall_at_k(ranking, judgments[246], 10):.3f}")2021# the best any ranking can do: a top 10 that holds as many exact products as the query has22exact_counts = pd.Series({query_id: sum(grade == 2 for grade in per_query.values())23 for query_id, per_query in judgments.items()})24# a top 10 can hold at most 10 exact products, and no more than the query has25best_precision = exact_counts.clip(upper=10) / 1026# only 10 results are shown, so at most 10 of a query's exact products can be retrieved27best_recall = exact_counts.clip(upper=10) / exact_counts28print()29print("queries where precision@10 can't reach 1:", (best_precision < 1).sum())30print("fewest exact products for a query:", exact_counts.min())31print("queries with that few:", (exact_counts == exact_counts.min()).sum())32print(f"best precision@10 for those queries: {best_precision.min():.3f}")33print("queries where recall@10 can't reach 1:", (best_recall < 1).sum())34most = exact_counts.idxmax()35print("query with the most exact products:", queries.at[most, "query"])36print("its exact products:", exact_counts[most])37print(f"its best recall@10: {best_recall[most]:.3f}")
python precision-recall-by-hand.pyexact in the top 10: 4
exact products judged: 12
precision@10 by hand: 0.400
recall@10 by hand: 0.333
precision@10 from lib.metrics: 0.400
recall@10 from lib.metrics: 0.333
queries where precision@10 can't reach 1: 7
fewest exact products for a query: 4
queries with that few: 4
best precision@10 for those queries: 0.400
queries where recall@10 can't reach 1: 22
query with the most exact products: outdoor light fixtures
its exact products: 170
its best recall@10: 0.059
lib.metrics agrees with the hand count. Of the top 10, 4 are exact, so precision@10 is 0.400. Against the 12 exact products the judges marked for this query, recall@10 is 0.333. precision_at_k and recall_at_k give the same two values.
Neither metric has one ceiling for every query. A query with fewer than 10 exact products can't fill its top 10 with them, so precision@10 can't reach 1. 7 test queries are like that, and the 4 with only 4 exact products can reach a precision@10 of 0.400 at best.
A query with more than 10 exact products can't reach full recall in 10 results. That holds for 22 of the test queries. The query with the most, outdoor light fixtures, has 170 exact products, so its recall@10 tops out at 0.059. A mean over the test queries mixes queries that can score 1 with queries that can't.
Recall has a second problem, because its denominator counts only the exact products the judges found. WANDS judged candidates gathered for each query, not the whole catalog (chapter 1), so some exact products were never judged. Zobel found that judgments gathered this way overestimate recall (Know where the judgments came from).
Working rule: Use precision@10 when you care how much of the first page is right, and read it next to each query's ceiling. Don't make recall@10 a headline in product search, where a query can have more than a hundred relevant products, as outdoor light fixtures has. We recommend recall at a deep cutoff, such as the top 100, for one job only: checking that retrieval passes enough relevant products to a reranker. Even then, read it as an overestimate.
这部分尚未翻译,因此以英文显示。
Score the first relevant result with reciprocal rank
Precision counts a whole page, but some shoppers want one product: a replacement part, or a model they already know. For them, what matters is how far down the first good result sits. Reciprocal rank (RR) scores a query as 1 divided by the position of the first relevant result, or 0 if there is none, and the mean of RR over queries is the mean reciprocal rank (MRR). TREC, the Text REtrieval Conference, runs shared evaluations of search systems. MRR was the official measure of its question-answering track at TREC-8, where each question scored the reciprocal of the rank of the first correct answer, or 0 when none of the five answers was correct. The track chose it partly because it stays between 0 and 1 and averages well (Voorhees, 1999).
Averaging RR is disputed. Fuhr points out that under RR "the difference between ranks 1 and 2 is the same as that between ranks 2 and ∞", so RR isn't an interval scale, one on which equal steps mean equal differences, and he argues that its mean shouldn't be computed (Fuhr, 2017, section 2.1). Sakai sees no reason to forbid it (Sakai, 2020). Moffat argues that such metrics are more meaningful in their current form than in the intervalized versions proposed for them (Moffat, 2022). We use MRR with that caveat: it is a coarse summary.
RR depends on where you draw the line for relevant, so score BM25's ranking for clear shower curtain twice: once counting only exact products, once counting partial ones too. Then compute MRR for BM25 and semantic retrieval over the test queries:
1"""Writes reciprocal rank by hand for one query under two grade thresholds, then computes MRR for BM25 and semantic retrieval and checks it against lib.metrics. The prose's point rests on the two rr lines for clear shower curtain and the two MRR lines."""2import pandas as pd34from lib.lexical import LexicalSearch5from lib.metrics import evaluate6from lib.semantic import SemanticSearch7from lib.store import load_products, load_queries, qrels8910# relevant=2 counts only exact products; relevant=1 counts partial ones too11def reciprocal_rank(ranking, judgments, relevant=2):12 """1 / the position of the first relevant product, or 0 if the ranking has none."""13 # positions count from 1, as a shopper counts results14 for position, product_id in enumerate(ranking, start=1):15 # a product nobody judged has grade 0, so it never counts as relevant16 if judgments.get(product_id, 0) >= relevant:17 return 1 / position18 return 0.0192021queries, judgments = load_queries("test"), qrels("test")22# each run goes 100 products deep, so RR can look past the top 10 for the first exact product23runs = {"bm25": LexicalSearch(load_products()).run(queries, k=100), "semantic": SemanticSearch().run(queries, k=100)}2425ranking = runs["bm25"][246] # clear shower curtain26print(f"rr, exact counts as relevant: {reciprocal_rank(ranking, judgments[246]):.3f}")27print(f"rr, partial or better counts: {reciprocal_rank(ranking, judgments[246], relevant=1):.3f}")2829for name, run in runs.items():30 rr = pd.Series({query_id: reciprocal_rank(run[query_id], judgments[query_id]) for query_id in judgments})31 print(f"{name} mrr: {rr.mean():.3f}")32 print(f"{name} mrr from lib.metrics: {evaluate(run, judgments)['rr'].mean():.3f}")33 print(f"{name} queries with rr of 1:", (rr == 1).sum())
python reciprocal-rank-by-hand.pyrr, exact counts as relevant: 0.500
rr, partial or better counts: 1.000
bm25 mrr: 0.650
bm25 mrr from lib.metrics: 0.650
bm25 queries with rr of 1: 16
semantic mrr: 0.787
semantic mrr from lib.metrics: 0.787
semantic queries with rr of 1: 22
The grade threshold changes RR for this one query. BM25's first result for clear shower curtain is a partial match and its second is exact, so RR is 0.500 when only exact products count, and 1.000 when partial ones count too. Whenever you quote RR, say which grades count. The mean over the test queries matches the rr column of lib.metrics.evaluate: BM25 scores 0.650 and semantic retrieval 0.787. That column looks for the first exact product anywhere in the run, not only in the top 10.
RR has a blind spot. It is 1 for 22 test queries under semantic retrieval and for 16 under BM25, and on those queries it sees nothing below the first result. A ranker that puts one exact product first and nine irrelevant ones after it scores the same as a perfect top 10.
Working rule: Report MRR as the headline only when shoppers need one right product, as for a model number or a named product such as kohler whitehaven farmhouse kitchen sink. For queries where shoppers browse a page of options, such as wooden chair outdoor, use nDCG@10 as the headline and report MRR beside it.
由折损增益构建 nDCG
与精确率和 RR 不同,第 1 章用来为单个排序打分的 nDCG@10 会读取等级。每个结果把自己的等级作为增益,再乘以一个随位置增大而缩小的折损,这些值之和就是折损累积增益(DCG)。归一化 DCG(nDCG)用一个排序的 DCG 除以理想排序的 DCG。它的两个组成部分各有多个版本,而这些版本会改变数值。第一个是理想排序。提出 DCG 的 Järvelin 和 Kekäläinen 以“检索主题的召回基础,而不是某种 IR 技术的结果”为依据来构建它(Järvelin and Kekäläinen, 2002)。在实践中,这意味着理想排序是针对该查询已标注的所有商品给出的最佳顺序,而不仅仅是某个排序器返回的商品的最佳顺序。第二个是折损:他们提出把文档的得分除以其名次的对数,而各版本的区别在于采用哪种对数,以及从哪个名次开始折损。
lib.metrics 中的公式如下,其中 grade(i) 是位置 i 上商品的等级:
DCG@k = sum over positions i from 1 to k of grade(i) / log2(i + 1)
nDCG@k = DCG@k / DCG@k of the ideal ranking
这并不是唯一在用的版本。Järvelin 和 Kekäläinen 对名次低于对数底数的结果不做折损,因此底数为 2 时,前两个位置按全值计入。Burges 等人使用指数增益 2^grade − 1,并采用 log(1 + rank) 作为折损(Burges et al., 2005)。评估工具之间也有差异:ir-measures 文档为 pytrec_eval 和 gdeval 列出了不同的 nDCG 选项(ir-measures)。因此,每当引用一个 nDCG 数值时,都要指明所用的变体。
请从零写出 DCG 和 nDCG,并在第 1 章的 BM25、语义和混合运行结果上,用 lib.metrics 对它们进行核对。其中有一个得分好得令人难以置信:
1"""Writes DCG and nDCG from scratch and checks them against lib.metrics on every test query of three runs, then prints one query's grades beside its best possible grades. The prose's point rests on the largest-difference line and on the ndcg@10 line for luau string lights, read with the grade lines above it."""2import math34from lib.fusion import reciprocal_rank_fusion5from lib.lexical import LexicalSearch6from lib.metrics import ndcg_at_k7from lib.semantic import SemanticSearch8from lib.store import load_products, load_queries, qrels91011def dcg(grades, k):12 """Each grade divided by log2 of its position plus one, added up over the top k."""13 # positions count from 1, so the first result is divided by log2(2)14 return sum(grade / math.log2(position + 1) for position, grade in enumerate(grades[:k], start=1))151617def ndcg(ranking, judgments, k=10):18 """DCG of the ranking over DCG of the best order of every judged product (unjudged counts as 0)."""19 # the ideal ranking: every judged product, best grade first20 ideal = dcg(sorted(judgments.values(), reverse=True), k)21 # a product nobody judged counts as grade 0 in the ranking22 return dcg([judgments.get(product_id, 0) for product_id in ranking], k) / ideal232425queries, judgments = load_queries("test"), qrels("test")26bm25 = LexicalSearch(load_products()).run(queries, k=100)27semantic = SemanticSearch().run(queries, k=100)28runs = {"bm25": bm25, "semantic": semantic, "hybrid": reciprocal_rank_fusion([bm25, semantic])}2930gaps = [abs(ndcg(run[query_id], judgments[query_id]) - ndcg_at_k(run[query_id], judgments[query_id], 10))31 for run in runs.values() for query_id in judgments]32print("rankings checked:", len(gaps))33print(f"largest difference from lib.metrics: {max(gaps):.3f}")3435ranking, per_query = bm25[461], judgments[461] # luau string lights36print()37print("query:", queries.at[461, "query"])38print("grades in the top 10:", [per_query.get(product_id, 0) for product_id in ranking[:10]])39print("best possible grades:", sorted(per_query.values(), reverse=True)[:10])40print("exact products judged:", sum(grade == 2 for grade in per_query.values()))41print("exact in the top 10:", sum(per_query.get(product_id, 0) == 2 for product_id in ranking[:10]))42print(f"ndcg@10: {ndcg(ranking, per_query):.3f}")
python ndcg-from-scratch.pyrankings checked: 90
largest difference from lib.metrics: 0.000
query: luau string lights
grades in the top 10: [2, 2, 2, 2, 1, 1, 1, 1, 1, 1]
best possible grades: [2, 2, 2, 2, 1, 1, 1, 1, 1, 1]
exact products judged: 4
exact in the top 10: 4
ndcg@10: 1.000
在全部 90 个排序上,手写版本与 lib.metrics 的差异从未超过 0.000。
对于 luau string lights,BM25 的前 10 名包含 4 个精确匹配商品,得分为 1.000。这并不是错误。该查询只有 4 个已标注的精确匹配商品,因此最好的前 10 名由这些商品加上部分匹配商品组成,而 BM25 找到了这一顺序。nDCG@10 为 1 的含义是“与已标注商品的最佳顺序一样好”,而不是“十个精确匹配商品”。
工作准则: 使用线性增益、log2(i + 1) 折损的形式,截断位置取 10,理想排序由所有已标注商品构建,并且每当引用数值时都要说明该变体。只在使用相同标注评分的运行之间比较 nDCG:新的标注会改变理想排序,进而改变每一个得分。
与 scikit-learn 的 ndcg_score 保持一致
将指标与独立计算该指标的库对照检验。在 scikit-learn 1.9.1 中,ndcg_score 采用与 lib.metrics 相同的变体:以等级作为增益,每个增益除以 log2(rank + 1),而用于归一化的 DCG 是将同一组等级按最佳顺序排列后、在 k 处截断得到的 DCG(scikit-learn 源码)。其文档引用了 Järvelin 和 Kekäläinen 的论文,但代码实现的是后来的 rank + 1 形式(scikit-learn 文档)。有三个细节会使直接调用的结果与你自己的结果不一致:
- 它只对你传入的商品打分,因此仅用这些商品构造理想排序。
- 默认情况下,它会对得分相同的商品的增益取平均值(scikit-learn 文档)。这等于对并列商品的所有排列顺序取平均(McSherry and Najork, 2008)。
- 它要求每个查询占一行,且各行长度相同,并返回各行的平均值(scikit-learn 源码)。
先看直接调用的情形:ndcg_score 接收 BM25 对 clear shower curtain 返回的前 10 个商品的等级,并附上使它们保持 BM25 顺序的得分。然后按 lib.metrics 看待查询的方式构造每个查询的行,并比较这三次运行:
1"""Compares scikit-learn's ndcg_score with lib.metrics for one query, first with only the top 10 passed and then with every judged product in the row, and then on every test query of three runs. The prose's point rests on the ndcg@10 lines for clear shower curtain and on the 5-product lines."""2from sklearn.metrics import ndcg_score34from lib.fusion import reciprocal_rank_fusion5from lib.lexical import LexicalSearch6from lib.metrics import ndcg_at_k7from lib.semantic import SemanticSearch8from lib.store import load_products, load_queries, qrels91011def sklearn_ndcg(ranking, judgments, k=10):12 """One row for one query: every judged product, plus any retrieved product nobody judged."""13 # every judged product once, then the retrieved products nobody judged14 products = list(dict.fromkeys([*judgments, *ranking]))15 position = {product_id: i for i, product_id in enumerate(ranking)}16 grades = [[judgments.get(product_id, 0) for product_id in products]]17 # ndcg_score ranks by score, so the negated position puts rank 1 highest18 scores = [[-position.get(product_id, len(ranking)) for product_id in products]] # not retrieved: below all19 return ndcg_score(grades, scores, k=k)202122queries, judgments = load_queries("test"), qrels("test")23bm25 = LexicalSearch(load_products()).run(queries, k=100)24semantic = SemanticSearch().run(queries, k=100)25runs = {"bm25": bm25, "semantic": semantic, "hybrid": reciprocal_rank_fusion([bm25, semantic])}2627top = bm25[246][:10] # clear shower curtain28# scores 10 down to 1 put the top 10 in BM25's order, so the first product scores highest29naive = ndcg_score([[judgments[246].get(product_id, 0) for product_id in top]], [list(range(10, 0, -1))], k=10)30print(f"ndcg@10, only the top 10 passed: {naive:.3f}")31print(f"ndcg@10 from lib.metrics: {ndcg_at_k(top, judgments[246], 10):.3f}")32print(f"ndcg@10, every judged product passed: {sklearn_ndcg(top, judgments[246]):.3f}")3334gaps = [abs(sklearn_ndcg(run[query_id], judgments[query_id]) - ndcg_at_k(run[query_id], judgments[query_id], 10))35 for run in runs.values() for query_id in judgments]36print("rankings compared:", len(gaps))37print(f"largest difference: {max(gaps):.3f}")3839short = bm25[246][:5] # a ranking shorter than k40print(f"5-product ranking, lib.metrics: {ndcg_at_k(short, judgments[246], 10):.3f}")41print(f"5-product ranking, scikit-learn: {sklearn_ndcg(short, judgments[246]):.3f}")
python ndcg-matches-scikit-learn.pyndcg@10, only the top 10 passed: 0.901
ndcg@10 from lib.metrics: 0.615
ndcg@10, every judged product passed: 0.615
rankings compared: 90
largest difference: 0.000
5-product ranking, lib.metrics: 0.539
5-product ranking, scikit-learn: 0.629
只传入前 10 个商品时,ndcg_score 返回 0.901,而 lib.metrics 返回的 0.615,原因在于它只除以这十个商品自身的最佳顺序,因此看不到 BM25 漏掉的那些完全匹配的商品。lib.metrics 则除以该查询下已评判的 253 个商品的最佳顺序。示例中的 sklearn_ndcg 用一个办法修正了这一点。每一行包含所有已评判的商品,以及任何被检索到但无人评判的商品(等级为 0)。该办法按排名为检索到的商品打分,并给其余每个商品一个低于所有这些得分的相同分数。采用这一办法后,两者在全部 90 个排序上结果一致(最大差值 0.000)。请对每个查询各调用一次 ndcg_score,再自行对结果取平均,因为每个查询的行长度不同。
这个办法有一个局限,较短的运行结果会暴露它。ndcg_score 把某次运行未检索到的商品视为并列占据前 k 名内的空位,并按平均增益为这些空位计分。lib.metrics 则对空位不给任何分数,因为用户在那里什么也看不到。把 BM25 的排序截短为 5 个商品,两者就会不一致:lib.metrics 得到 0.539,而 ndcg_score 得到 0.629。短结果确实会出现:LexicalSearch 只返回得分高于 0 的商品。
工作准则: 报告你自己计算的 nDCG,并将 ndcg_score 用作独立检验,每个查询的行中包含所有已评判的商品,得分取自排名。在比较两者之前,先检查每次运行是否至少包含 k 个商品。如果你改为传入排序器的原始得分,请先按固定顺序打破并列,例如按 product_id 排序,否则凡是得分相同之处,两者都会不一致。
比较之前先选定指标
每个指标都是对一类购物者的建模。Precision@10 假定购物者会浏览整个首页,并统计页面上好商品的数量。RR 假定购物者在看到第一个好商品时就停止。nDCG@10 假定注意力沿页面向下递减,并且精确匹配的商品比部分匹配的商品更有价值:按本书的分级,价值是后者的两倍。指数增益 2^grade − 1(如 Burges et al. (2005) 所用)会把分级转换为 0、1 和 3 的增益,因此一个精确匹配的商品相当于三个部分匹配的商品。根据经验,当两个排序器相差很大时,这些模型的结论是一致的。当两个排序器接近时,指标本身就可能决定胜者,这就是要在查看结果之前选定指标的原因。Fuhr 要求事先陈述假设,形式为“We want to compare methods A and B using metric X”(Fuhr, 2017, section 2.6)。
以语义检索(需要超越的排序器)和混合检索为例,第 1 章未能把后者与前者区分开。选择指标是调整评估的一部分,因此要在 50 条训练查询上做出选择,并把测试查询留作报告之用。用四种方式为两个排序器打分:
1"""Scores semantic retrieval and the hybrid four ways on the train queries and names the higher scorer for each metric. The prose's point rests on the "higher" lines, the count of metrics each ranker wins, and the largest gap."""2from lib.fusion import reciprocal_rank_fusion3from lib.lexical import LexicalSearch4from lib.metrics import evaluate5from lib.semantic import SemanticSearch6from lib.store import load_products, load_queries, qrels78queries, judgments = load_queries("train"), qrels("train") # choosing a metric is tuning: use train9bm25 = LexicalSearch(load_products()).run(queries, k=100)10semantic = SemanticSearch().run(queries, k=100)11runs = {"semantic": semantic, "hybrid": reciprocal_rank_fusion([bm25, semantic])}1213exponential = {query_id: {product_id: 2 ** grade - 1 for product_id, grade in per_query.items()}14 for query_id, per_query in judgments.items()} # gains 0, 1, 3 instead of 0, 1, 215means = {}16for name, run in runs.items():17 scores = evaluate(run, judgments)18 means[name] = {"precision@10": scores["precision@10"].mean(), "mrr": scores["rr"].mean(),19 "ndcg@10": scores["ndcg@10"].mean(),20 "ndcg@10 with exponential gains": evaluate(run, exponential)["ndcg@10"].mean()}2122print("train queries:", len(judgments))23winners = []24for metric in means["semantic"]:25 for name in runs:26 print(f"{name} {metric}: {means[name][metric]:.3f}")27 # max keeps the first name it meets, so on an exact tie the order of runs decides28 winners.append(max(runs, key=lambda name: means[name][metric]))29 print(f"higher {metric}: {winners[-1]}")30for name in runs:31 print(f"metrics won by {name}:", winners.count(name))32print(f"largest gap: {max(abs(means['semantic'][m] - means['hybrid'][m]) for m in means['semantic']):.3f}")
python metric-choice-on-train.pytrain queries: 50
semantic precision@10: 0.502
hybrid precision@10: 0.488
higher precision@10: semantic
semantic mrr: 0.673
hybrid mrr: 0.710
higher mrr: hybrid
semantic ndcg@10: 0.772
hybrid ndcg@10: 0.774
higher ndcg@10: hybrid
semantic ndcg@10 with exponential gains: 0.709
hybrid ndcg@10 with exponential gains: 0.708
higher ndcg@10 with exponential gains: semantic
metrics won by semantic: 2
metrics won by hybrid: 2
largest gap: 0.037
这四个指标对胜者的判断并不一致。语义检索在其中 2 个指标上领先,混合检索在 2 个指标上领先,两者之间没有任何差距大于 0.037。当两个排序器如此接近时,由指标决定差距落向哪一边。第 1 章选定的核心指标 nDCG@10 偏向 hybrid。这些是训练查询,所以以上结果都不能说明哪个排序器更好。
因此,选择应当依据购物者,而不是依据结果。商店展示的是一页商品,其判断是分级的,所以 nDCG@10 与线性增益仍作为核心指标,一如第 1 章的选择,precision@10 和 MRR 与之并列报告。
工作准则: 在比较排序器之前,选定一个核心指标及其增益和截断位置,并把它们写下来。当判断是分级的、且购物者浏览整页时,我们推荐使用线性增益的 nDCG@10。当精确匹配远比近似匹配有价值时(例如必须适配的备件),改用指数增益;当大多数查询只有一个正确答案时,改用 MRR。在核心指标之外同时报告其他指标,并且在看到核心指标偏向哪个排序器之后,绝不更改它。
阅读逐查询差异,而不只是均值
对各查询的核心指标取均值,得到的是对整个商店的概括,而购物者看到的只是某一个查询的结果。TREC Robust 评测任务正是围绕这一点设立的。其组织者发现,针对平均值进行调优,可能会改善本来就表现良好的查询,有时还以牺牲表现较差的查询为代价(Voorhees, 2004)。因此,在相信均值的变化之前,应先查看逐查询差异:每个查询对应一个数值,即新排序器的得分减去旧排序器的得分。这些差异也是显著性检验所使用的证据。
平均来看,第 1 章中商店的混合检索运行结果明显高于 BM25。下面在 nDCG@10 上逐查询比较两者:
1"""Compares the hybrid and BM25 query by query on nDCG@10, then lists the three largest losses and the largest gain with its judged@10. The prose's point rests on the mean and median difference lines and on the largest-gain lines."""2from lib.fusion import reciprocal_rank_fusion3from lib.lexical import LexicalSearch4from lib.metrics import evaluate5from lib.semantic import SemanticSearch6from lib.store import load_products, load_queries, qrels78queries, judgments = load_queries("test"), qrels("test")9bm25 = LexicalSearch(load_products()).run(queries, k=100)10hybrid = reciprocal_rank_fusion([bm25, SemanticSearch().run(queries, k=100)])11before, after = evaluate(bm25, judgments), evaluate(hybrid, judgments)1213# rounded to three decimals first, so a difference that rounds to 0 counts as a tie; sorted from the largest loss up14difference = (after["ndcg@10"] - before["ndcg@10"]).round(3).sort_values() # one value per test query15print(f"mean difference: {difference.mean():.3f}")16print(f"median difference: {difference.median():.3f}")17print("hybrid better:", (difference > 0).sum())18print("bm25 better:", (difference < 0).sum())19print("tied:", (difference == 0).sum())2021print("\nhybrid's three largest losses:")22print(queries.loc[difference.index[:3], "query"].to_frame().assign(difference=difference[:3]).to_string())23# the last one after sorting: the largest gain24largest = difference.index[-1]25print("\nlargest gain:", queries.at[largest, "query"])26print(f"its gain: {difference[largest]:.3f}")27print(f"its bm25 judged@10: {before.at[largest, 'judged@10']:.3f}")28print(f"share of the mean gain from this query: {difference[largest] / difference.sum():.3f}")2930relative = after["ndcg@10"] / before["ndcg@10"] - 1 # per-query relative gains31print(f"\nrelative gain of the means: {after['ndcg@10'].mean() / before['ndcg@10'].mean() - 1:.3f}")32print(f"mean of per-query relative gains: {relative.mean():.3f}")33print(f"its bm25 ndcg@10: {before.at[largest, 'ndcg@10']:.3f}")34print(f"its relative gain: {relative[largest]:.3f}")35print(f"mean of per-query relative gains without it: {relative.drop(largest).mean():.3f}")
python per-query-hybrid-versus-bm25.pymean difference: 0.087
median difference: 0.057
hybrid better: 17
bm25 better: 7
tied: 6
hybrid's three largest losses:
query difference
query_id
293 wood floor protector mat -0.072
292 decorative white pillow -0.069
249 bistro table and chairs -0.066
largest gain: outdoor light fixtures
its gain: 0.714
its bm25 judged@10: 0.200
share of the mean gain from this query: 0.274
relative gain of the means: 0.120
mean of per-query relative gains: 0.218
its bm25 ndcg@10: 0.286
its relative gain: 2.492
mean of per-query relative gains without it: 0.140
平均增益为 0.087,而中位数为 0.057。混合检索在更多查询上胜出,胜负比为 17 比 7,另有 6 个查询持平。
有一个查询贡献了很大一部分增益。outdoor light fixtures 的增益为 0.714,占总增益的 0.274,而 BM25 在该查询上的 judged@10 为 0.200。对这个查询而言,BM25 的前 10 个结果大多是无人标注的灯泡,第 1 章论证了将它们记为 0 分是合理的,因此这一增益成立。对于另一个查询,排序器所舍弃的未标注商品也许恰恰是好商品。混合检索的三个最大损失,是购物者会感到搜索变差的查询,所以应当先阅读这些查询。
差异应以绝对数值报告。均值的相对增益为 0.120,但逐查询相对增益的均值为 0.218,因为逐查询比值的均值会对 BM25 得分较低的查询赋予最大的权重。
outdoor light fixtures 上 BM25 的得分为 0.286,仅它一个查询就贡献了 2.492 的相对增益。去掉这一个查询后,相对增益的均值降至 0.140。Fuhr 指出,均值的相对变化无法可靠地说明各查询上的变化,并且“对于算术平均,只有绝对改进才有意义”(Fuhr, 2017, section 2.4)。
**工作准则:**每次比较都要打印均值差和中位数差、胜、负、持平的查询数,以及损失最大的查询,然后阅读这些查询。在认可排序器之前,先检查变化最大的那些查询的 judged@10。报告均值的绝对差,而不是百分比。
用置换检验检验差距
在几十个查询上得到的两个均值之间的差距,可能完全来自偶然:换一批查询,就会得到不同的差距。浅层指标会使这个问题更严重:Buckley 和 Voorhees 发现,对于 precision at 10 这类指标,研究者需要远多于 50 个查询,才能可靠地区分系统(Buckley and Voorhees, 2000)。显著性检验要问的是:在零假设(即假定两个排序器同样好)之下,两个排序器之间出现这样的差距有多令人意外。它给出的答案是 p 值:在零假设下,出现至少与你测得的差距一样大的差距的概率。按惯例,p 值低于 0.05 即视为存在真实差异的证据,而当一章中进行多次比较时,本书的规则会把这一标准收得更严。
本书中的检验都是配对的:两个排序器在相同的查询上打分,因此每个检验都作用于逐查询差值,困难查询的低分会相互抵消。置换检验直接构造零假设下会出现的差距。如果两个排序器可以互换,那么每个查询的差值取相反符号的可能性是相同的。因此,随机翻转每个符号,重新计算均值,重复数千次,再统计翻转后的均值与 0 的距离不小于真实均值的次数。对于 n 个查询,共有 2^n 种符号模式,测试查询的符号模式数为 1,073,741,824,所以该检验抽取其中的 9,999 种,其 p 值是一个估计值。
在测试查询上,混合排序落后于语义检索,而第 1 章没有回答这个差距是否真实。相对于语义检索,混合排序在测试查询上的胜、负和平局数为 9 / 14 / 7。检验会发现混合排序的劣势大于偶然吗?先定下你的答案,再运行它。
1"""Estimates the p-value of the hybrid's deficit against semantic retrieval by flipping each query's sign at random, once by hand and once with SciPy's permutation_test. The prose's point rests on the observed mean, the two p-value lines, and the line that compares the p-value with 0.05."""2import numpy as np3from scipy.stats import permutation_test45from lib.fusion import reciprocal_rank_fusion6from lib.lexical import LexicalSearch7from lib.metrics import evaluate8from lib.semantic import SemanticSearch9from lib.store import load_products, load_queries, qrels1011queries, judgments = load_queries("test"), qrels("test")12semantic = SemanticSearch().run(queries, k=100)13hybrid = reciprocal_rank_fusion([LexicalSearch(load_products()).run(queries, k=100), semantic])14after = evaluate(hybrid, judgments)["ndcg@10"].to_numpy()15before = evaluate(semantic, judgments)["ndcg@10"].to_numpy()16difference = after - before # hybrid minus semantic, per test query17print("test queries:", len(difference))18print(f"sign patterns for the test queries: {2 ** len(difference):,}")19print(f"observed mean difference: {difference.mean():.3f}")2021# if the two rankers were interchangeable, each query's difference would be as likely to have the other sign22# a fixed seed, so every run draws the same sign patterns23rng = np.random.default_rng(2026)24# 9,999 random sign patterns, one sign for each query in each25signs = rng.choice([-1, 1], size=(9999, len(difference)))26flipped = (signs * difference).mean(axis=1) # 9,999 mean differences under that assumption27# the 95th percentile of the flipped means' sizes28print(f"95% of flipped means are smaller than: {np.quantile(np.abs(flipped), 0.95):.3f}")29print(f"p-value by hand: {np.mean(np.abs(flipped) >= abs(difference.mean())):.3f}")3031result = permutation_test((after, before), lambda after_scores, before_scores, axis: np.mean(after_scores - before_scores, axis=axis),32 # samples: flip each query's pair of scores, which is the paired setting33 permutation_type="samples", rng=np.random.default_rng(2026))34print(f"p-value from scipy: {result.pvalue:.3f}")35print("difference at the 0.05 level:", "larger than chance" if result.pvalue < 0.05 else "within chance")
python permutation-test-by-hand.pytest queries: 30
sign patterns for the test queries: 1,073,741,824
observed mean difference: -0.018
95% of flipped means are smaller than: 0.022
p-value by hand: 0.117
p-value from scipy: 0.125
difference at the 0.05 level: within chance
在 0.05 的水平上,混合排序相对于语义检索的劣势within chance。观察到的平均差值为 -0.018,95% 的翻转均值在绝对值上小于 0.022。p 值,即翻转均值至少与观察值一样大的比例,为 0.117。
SciPy 的 permutation_test 在设置为 permutation_type="samples"(其文档为配对检验给出的设置)时运行,返回 0.125(SciPy 文档)。两个估计值略有不同,因为各自抽取了自己的随机符号模式。要说明劣势真实存在,均值必须压过离散程度,而少数几个朝相反方向波动的查询就会弥合这个差距。
SciPy 的重采样函数通过 rng 接收随机数生成器,它取代了 random_state 参数。在 1.18.1 中两者都可使用,但文档指出“新代码中应只使用 rng 关键字”,因为使用 random_state 的调用在过渡期之后将发出警告(SciPy 文档)。本书的每个示例都以固定的种子传入 rng。
工作准则: 对于每一个你打算据以行动的差距,都要对逐查询差值做配对置换检验。本书依据这一检验作出决定,如其规则所述。p 值高于阈值时,应读作“没有证据表明存在差异”,而不是“没有差异”:查询如此之少,检验可能漏掉真实的差距,McKechnie 等人主张在衡量假阳性的同时也衡量这类漏检(McKechnie et al., 2025)。对于检验无法与偶然区分的差距,切勿加以解释。
用配对 t 检验得到 p 值
配对 t 检验用公式而不是重采样来回答同一个问题。它把逐查询差值的均值除以标准误。标准误是差值的标准差除以查询数的平方根,用来估计在不同的查询样本之间,均值会移动多远。如果两个排序器同样好,并且差值大致服从正态分布,那么这个比值 t 服从 t 分布。t 分布呈钟形,尾部比正态分布更重,其形状由自由度决定,这里的自由度比查询数少 1。p 值是 t 在任一方向上至少偏离 0 这么远的概率(Sakai, 2014, section 3.1.1):
t = mean(d) / (sd(d) / sqrt(n)) d: the per-query differences, n: the number of queries
逐查询的 nDCG 差值被限制在 −1 到 1 之间,并且有离群值,例如outdoor light fixtures 上的增益,所以它们并不服从正态分布。实际上,t 检验仍然站得住。Smucker 等人在 50 个主题上,用平均精度均值(MAP,另一种排序指标)对 TREC 运行结果评分,发现随机化检验、自助法检验和 t 检验之间几乎没有实际差别,其中随机化检验就是置换检验(Smucker et al., 2007)。Urbano 等人在由 TREC 数据构建的模拟中,使用了 25、50 和 100 个主题的集合,发现“t 检验和置换检验的表现几乎完全相同”(Urbano et al., 2019)。两项研究都不建议在这项任务中使用 Wilcoxon 符号秩检验和符号检验。SciPy 的 wilcoxon 默认还会丢弃差值恰好为 0 的所有查询,因为默认设置为 zero_method='wilcox'(SciPy documentation),而当两次运行差别很小时,许多查询会出现并列。
公式和重采样在第 1 章的差距上是否一致?取其中两个,即语义检索对比 BM25和混合检索对比语义检索。手工算出的公式 t 值应与 SciPy 的 ttest_rel 一致:
1"""Computes the paired t statistic and its p-value by hand for two gaps, and checks each p-value against SciPy's ttest_rel. The prose's point rests on the t and p lines printed for each gap."""2import math34from scipy.stats import t, ttest_rel56from lib.fusion import reciprocal_rank_fusion7from lib.lexical import LexicalSearch8from lib.metrics import evaluate9from lib.semantic import SemanticSearch10from lib.store import load_products, load_queries, qrels1112queries, judgments = load_queries("test"), qrels("test")13bm25 = LexicalSearch(load_products()).run(queries, k=100)14semantic = SemanticSearch().run(queries, k=100)15runs = {"bm25": bm25, "semantic": semantic, "hybrid": reciprocal_rank_fusion([bm25, semantic])}16ndcg = {name: evaluate(run, judgments)["ndcg@10"] for name, run in runs.items()} # per test query1718for after, before in [("semantic", "bm25"), ("hybrid", "semantic")]:19 difference = ndcg[after] - ndcg[before]20 n = len(difference)21 # ddof=1 divides by n - 1, the sample standard deviation the t-test uses22 standard_error = difference.std(ddof=1) / math.sqrt(n) # how much the mean would move between samples23 statistic = difference.mean() / standard_error24 p_value = 2 * t.sf(abs(statistic), df=n - 1) # both tails of the t distribution25 print(f"{after} minus {before}: {difference.mean():.3f}")26 print(f"{after} minus {before} t by hand: {statistic:.3f}")27 print(f"{after} minus {before} p by hand: {p_value:.3f}")28 print(f"{after} minus {before} p from ttest_rel: {ttest_rel(ndcg[after], ndcg[before]).pvalue:.3f}")
python paired-t-test-by-hand.pysemantic minus bm25: 0.105
semantic minus bm25 t by hand: 3.294
semantic minus bm25 p by hand: 0.003
semantic minus bm25 p from ttest_rel: 0.003
hybrid minus semantic: -0.018
hybrid minus semantic t by hand: -1.590
hybrid minus semantic p by hand: 0.123
hybrid minus semantic p from ttest_rel: 0.123
对于语义检索对比 BM25,均值差 0.105 给出 t = 3.294,p 值为 0.003,与 ttest_rel 给出的相同。这是偶然很少会产生的差距。
对于混合检索对比语义检索,t = -1.590,p 值为 0.123。这与置换检验对同一差距给出的 0.125 接近。
工作准则: 请与置换检验一并报告配对 t 检验 scipy.stats.ttest_rel,把它作为无需重采样的核对。当两者落在阈值的两侧时,要明确说明,并以置换检验为准。不要用 Wilcoxon 检验或符号检验来比较平均分数。
这部分尚未翻译,因此以英文显示。
Report how big the gap is
A p-value says how surprising a gap is, not how big it is. On a large sample, nearly any modification produces a significant but tiny difference (Fuhr, 2017, section 2.8). Sakai recommends reporting an effect size and a confidence interval with every p-value (Sakai, 2014). A confidence interval is a range of plausible values for the true mean difference: the mean over all queries like these, scored with these judgments, not over the store's search traffic. A 95% interval comes from a procedure meant to cover that value 95% of the time.
The bootstrap builds a confidence interval by resampling. Draw as many queries as you have from the queries you have, with replacement, so that some appear twice and others not at all. Compute the mean difference, and repeat thousands of times. The percentile interval is the middle 95% of the resampled means. The effect size of a paired comparison is the mean difference divided by the standard deviation of the differences (Sakai, 2014, section 3.1.2). It is how large the gap is next to how much it varies from query to query. It isn't Cohen's d, the effect size for unpaired samples, and Sakai warns against comparing the two directly.
Can a hand-built percentile interval match SciPy's bootstrap with method="percentile"? Build one by hand for semantic retrieval against BM25 and for the hybrid against semantic retrieval, check each against SciPy, then run SciPy's default method:
1"""Builds a 95% percentile interval by hand from 9,999 resampled means, checks it against SciPy's percentile method, then reads SciPy's default BCa interval and the effect size for each gap. The prose's point rests on the BCa low and high lines and on the line that says whether each interval includes 0."""2import numpy as np3from scipy.stats import bootstrap45from lib.fusion import reciprocal_rank_fusion6from lib.lexical import LexicalSearch7from lib.metrics import evaluate8from lib.semantic import SemanticSearch9from lib.store import load_products, load_queries, qrels1011queries, judgments = load_queries("test"), qrels("test")12bm25 = LexicalSearch(load_products()).run(queries, k=100)13semantic = SemanticSearch().run(queries, k=100)14runs = {"bm25": bm25, "semantic": semantic, "hybrid": reciprocal_rank_fusion([bm25, semantic])}15ndcg = {name: evaluate(run, judgments)["ndcg@10"] for name, run in runs.items()} # per test query1617for after, before in [("semantic", "bm25"), ("hybrid", "semantic")]:18 difference, name = (ndcg[after] - ndcg[before]).to_numpy(), f"{after} minus {before}"19 # choice draws with replacement, so a query can appear twice in one set and not at all in another20 resampled = np.random.default_rng(2026).choice(difference, size=(9999, len(difference))) # 9,999 sets of queries21 # 2.5% in each tail leaves the middle 95%22 low, high = np.percentile(resampled.mean(axis=1), [2.5, 97.5]) # the middle 95% of their means23 print(f"{name}: {difference.mean():.3f}")24 print(f"{name} interval by hand: {low:.3f} to {high:.3f}")25 percentile = bootstrap((difference,), np.mean, method="percentile", rng=np.random.default_rng(2026))26 print(f"{name} percentile interval from scipy: "27 f"{percentile.confidence_interval.low:.3f} to {percentile.confidence_interval.high:.3f}")28 # BCa is SciPy's default method, so no method argument is given29 bca = bootstrap((difference,), np.mean, rng=np.random.default_rng(2026)).confidence_interval30 print(f"{name} bca low: {bca.low:.3f}\n{name} bca high: {bca.high:.3f}")31 print(f"{name} bca interval and 0: {'includes 0' if bca.low <= 0 <= bca.high else 'excludes 0'}")32 # ddof=1: the standard deviation of the differences, with n - 1 in the denominator33 print(f"{name} effect size: {difference.mean() / difference.std(ddof=1):.3f}")
python bootstrap-interval-by-hand.pysemantic minus bm25: 0.105
semantic minus bm25 interval by hand: 0.049 to 0.172
semantic minus bm25 percentile interval from scipy: 0.049 to 0.172
semantic minus bm25 bca low: 0.055
semantic minus bm25 bca high: 0.185
semantic minus bm25 bca interval and 0: excludes 0
semantic minus bm25 effect size: 0.601
hybrid minus semantic: -0.018
hybrid minus semantic interval by hand: -0.041 to 0.003
hybrid minus semantic percentile interval from scipy: -0.041 to 0.003
hybrid minus semantic bca low: -0.042
hybrid minus semantic bca high: 0.002
hybrid minus semantic bca interval and 0: includes 0
hybrid minus semantic effect size: -0.290
SciPy's percentile interval matches the one built by hand: 0.049 to 0.172 and -0.041 to 0.003. Its default method, BCa (bias-corrected and accelerated), reads different percentiles off the same resampled means. It shifts them by how far the resampled means sit off-centre from the observed mean, and stretches them by the skew it estimates by leaving out one query at a time (SciPy source). The book's harness reports BCa, designed for skewed resampled means like these.
For semantic retrieval against BM25, BCa runs from 0.055 to 0.185, so it excludes 0, and the effect size is 0.601.
For the hybrid against semantic retrieval, the interval runs from -0.042 to 0.002. It includes 0, and the effect size is -0.290. That interval allows anything from a loss of a few hundredths to a gain of at most 0.002: no gain worth having.
A bootstrap interval only approximates its level, and on a few dozen queries it falls short. To check, treat the hybrid's 30 per-query differences against semantic retrieval as if they were every query there is, so the true gap is known. Then draw 1000 new sets of the same size from them, build an interval on each, and count how often it covers the true gap:
1"""Draws new samples from the test queries' differences and counts how often a BCa or a t interval covers the known gap, at two levels. The prose's point rests on the four coverage lines and the standard-error line."""2import numpy as np3from scipy.stats import bootstrap, ttest_1samp45from lib.fusion import reciprocal_rank_fusion6from lib.lexical import LexicalSearch7from lib.metrics import evaluate8from lib.semantic import SemanticSearch9from lib.store import load_products, load_queries, qrels1011queries, judgments = load_queries("test"), qrels("test")12semantic = SemanticSearch().run(queries, k=100)13hybrid = reciprocal_rank_fusion([LexicalSearch(load_products()).run(queries, k=100), semantic])14population = (evaluate(hybrid, judgments)["ndcg@10"] - evaluate(semantic, judgments)["ndcg@10"]).to_numpy()15true_gap = population.mean() # known, because these differences stand in for every query there is1617# a fixed seed, so the coverage counts repeat on every run18rng, trials = np.random.default_rng(2026), 100019print("trials:", trials)20# binomial standard error of a proportion near 0.95, over the trials21print(f"standard error of a coverage near 0.95: {(0.95 * 0.05 / trials) ** 0.5:.3f}")22for level in [0.95, round(1 - 0.05 / 3, 4)]: # the usual level, and the level for three comparisons23 covered = {"bca": 0, "t": 0}24 for _ in range(trials):25 sample = rng.choice(population, size=len(population)) # another set of 30 queries like these26 # 1,999 resamples per interval, fewer than SciPy's default, to keep 1,000 trials quick27 bca = bootstrap((sample,), np.mean, confidence_level=level, n_resamples=1999, rng=rng).confidence_interval28 t = ttest_1samp(sample, 0).confidence_interval(level) # the interval that matches the t-test29 covered["bca"] += bca.low <= true_gap <= bca.high30 covered["t"] += t.low <= true_gap <= t.high31 for method, count in covered.items():32 print(f"{method} coverage at {level}: {count / trials:.3f}")
python interval-coverage-check.pytrials: 1000
standard error of a coverage near 0.95: 0.007
bca coverage at 0.95: 0.927
t coverage at 0.95: 0.945
bca coverage at 0.9833: 0.955
t coverage at 0.9833: 0.975
At a nominal 95%, BCa covers the true gap 0.927 of the time and the t-test's interval 0.945. At 0.9833, the level the harness uses for its planned comparisons, BCa covers 0.955 and the t interval 0.975.
Each figure is uncertain by about 0.007 either way. BCa falls short of its level at both, and the t interval comes closer to its own. So a bootstrap interval that barely excludes 0 deserves care. In a simulation, a test built on the same resampling, the bootstrap-shift test, "shows a systematic bias towards small p-values" (Urbano et al., 2019).
Whether a gap matters is a judgment no test makes for you. For precision@10, Fuhr notes that "a difference of 0.02 translates into one more relevant document per five queries" (Fuhr, 2017, section 2.8). For nDCG@10, decide before you test what gap would justify the cost of a change.
Working rule: Give every comparison as its mean difference, with its bootstrap interval, its effect size and its p-value. If the interval includes 0, the comparison shows no improvement. If it excludes 0 but lies below the smallest gap worth the change's cost, the difference may be real and still not worth shipping. Use the interval to describe a gap's size, not to decide whether it exists.
事先确定什么算作已确立
当差异只是噪声时,每次显著性水平为 0.05 的检验都有 5% 的概率出现假阳性,而一章中若进行多次检验,这种风险就会累积多次。如果每个差异都是噪声,且各检验相互独立,那么对 10 个系统两两比较,至少出现一次错误的“显著”结果的概率约为 0.90(Sakai, 2014, section 3)。Bonferroni 校正是一种简单而保守的补救办法:检验 m 个假设时,每个假设都以 0.05 除以 m 作为检验水平(Fuhr, 2017, section 2.7)。无论各检验是否独立,它都能使出现任何假阳性的概率不超过 0.05。但这只有在这 m 个比较于看到结果之前就已确定时才成立。Fuhr 指出了破坏这一前提的常见做法:作者先在数据上找出哪种竞争方法表现最好,然后只拿自己的方法与这一种比较(Fuhr, 2017, section 2.6)。
因此,本书在每一章中都采用同一条规则:
- 在报告任何比较结果之前,先列出在测试查询上计划进行的比较。
- 当某个差异的配对置换检验给出的 p 值低于本章的 Bonferroni 阈值,即 0.05 除以计划比较的数量时,该差异即视为已确立。
- 同时报告配对 t 检验和相应的自助法区间,区间的水平与阈值对应,即 1 减去阈值。若其中任何一个与置换检验的结论不一致,须予以说明。
- 在查看测试结果之后才选定的比较属于探索性比较。 应当如此标注,绝不能称之为已确立。
- **在训练查询上的比较用于调优,**不能作为证据。
我们让置换检验来做决定,是因为它对差异的分布形状不做任何假设,只假设在零假设下每个查询的符号相当于抛一次硬币。Urbano 等人发现两者的表现几乎一致(Urbano et al., 2019),因此 t 检验可以用作很好的核对。在只有几十个查询的情况下,自助法区间的覆盖率低于其名义水平(覆盖率检查),所以区间只能描述差异,而不能对差异做出判定。
第 1 章逐个阶段地构建商店的流水线,仅这一顺序就确定了其计划比较:每个阶段与前一个阶段比较。由此得到语义检索对比 BM25、混合检索对比语义检索,以及重排序后的混合检索对比混合检索,共 3。阈值为 0.0167,区间的水平为 0.9833。将重排序后的混合检索与语义检索比较属于探索性比较,因为语义检索之所以成为需要超越的排序器,是由于它在这些相同的测试查询上得分最高。
工作准则: 在对测试查询打任何分之前,先写出计划比较的清单,并保持清单简短,因为每增加一项比较,其他比较的门槛就会随之提高。当需要同时比较许多排序器时,Bonferroni 校正会变得过于严格。Sakai 建议先做方差分析,检验这些排序器之间是否存在任何差异,然后再做随机化 Tukey HSD 检验,该检验对所有排序器两两比较,同时把出现任何假阳性的概率控制在给定水平(Sakai, 2014)。
在评估框架中比较各次运行
本书在评估框架中应用该规则。评估框架保持除排序器之外的一切不变:查询、判定、指标和检验都相同,因此两次运行的差别只来自它们的排序结果。lib.metrics.evaluate 为每个查询返回一行,这正是配对检验所需的形式。compare 位于 lib.significance 中,这是一个收录本章代码的简短模块,它对两次运行中某一指标的那一列执行各项检验,并返回平均差、自助法区间、置换检验和 t 检验的 p 值、效应量,以及胜、负和平局的数目。其中的 verdict 函数把本书的规则应用于一个结果。请保留每个查询一行的数据框,只在最后求平均。
下面用它来运行第 1 章的四个排序器,然后进行 3 项计划内比较和那项探索性比较:
1"""Runs chapter 1's four rankers through the harness and applies the Bonferroni threshold to the three planned comparisons and one exploratory comparison. The prose's point rests on the judged@10 lines, the established lines, and the exploratory comparison's p-value and agreement lines."""2from lib.fusion import reciprocal_rank_fusion3from lib.lexical import LexicalSearch4from lib.metrics import evaluate5from lib.recorded import cross_encoder_scores6from lib.semantic import SemanticSearch7from lib.significance import compare, verdict8from lib.store import load_products, load_queries, qrels910queries, judgments = load_queries("test"), qrels("test")11bm25 = LexicalSearch(load_products()).run(queries, k=100)12semantic = SemanticSearch().run(queries, k=100)13hybrid = reciprocal_rank_fusion([bm25, semantic])14score = {(query_id, product_id): value for query_id, product_id, value in cross_encoder_scores().itertuples(index=False)}15# chapter 1's rerank: only the top 50 are reordered; the rest keep their hybrid order16reranked = {query_id: sorted(ranking[:50], key=lambda product_id: score[(query_id, product_id)], reverse=True)17 + ranking[50:] for query_id, ranking in hybrid.items()}1819runs = {"bm25": bm25, "semantic": semantic, "hybrid": hybrid, "reranked": reranked}20scores = {name: evaluate(run, judgments) for name, run in runs.items()} # one row per test query21for name, frame in scores.items():22 print(f"{name} ndcg@10: {frame['ndcg@10'].mean():.3f}\n{name} judged@10: {frame['judged@10'].mean():.3f}")2324planned = [("semantic", "bm25"), ("hybrid", "semantic"), ("reranked", "hybrid")] # each stage against the last25exploratory = [("reranked", "semantic")] # chosen after chapter 1 saw the test means26threshold = 0.05 / len(planned) # Bonferroni, over the planned comparisons only27print(f"\nplanned comparisons: {len(planned)}\np-value threshold: {threshold:.4f}\ninterval level: {1 - threshold:.4f}")28# each planned test has a 0.95 chance of no false positive, if its gap is noise29print(f"chance of a false positive, all gaps noise, tests independent: {1 - 0.95 ** len(planned):.3f}")30established, disagreements = 0, 031for after, before in planned + exploratory:32 result, name = compare(scores[after]["ndcg@10"], scores[before]["ndcg@10"], confidence=1 - threshold), f"{after} minus {before}"33 # verdict runs on the exploratory one too, only for its agreement checks; it is never counted as established34 checks, kind = verdict(result, threshold), "planned" if (after, before) in planned else "exploratory"35 print(f"\n{name}: {result['mean difference']:.3f}\n{name} kind: {kind}")36 print(f"{name} interval: {result['ci low']:.3f} to {result['ci high']:.3f}")37 print(f"{name} permutation p: {result['permutation p']:.3f}\n{name} t-test p: {result['t-test p']:.3f}")38 print(f"{name} effect size: {result['effect size']:.3f}\n{name} wins, losses, ties: {result['wins']} / {result['losses']} / {result['ties']}")39 # only a planned comparison can be established; the exploratory one is labelled and never called so40 print(f"{name} established: {'yes' if checks['established'] and kind == 'planned' else 'no'}")41 for method in ["t-test", "interval"]:42 print(f"{name} {method} agrees: {'yes' if checks[method + ' agrees'] else 'no'}")43 if kind == "planned":44 established += checks["established"]45 # a planned comparison where the t-test or the interval disagrees with the permutation test46 disagreements += not (checks["t-test agrees"] and checks["interval agrees"])47print(f"\nplanned comparisons established: {established}\nplanned comparisons with a disagreement: {disagreements}")
python compare-chapter-one-rankers.pybm25 ndcg@10: 0.725
bm25 judged@10: 0.927
semantic ndcg@10: 0.830
semantic judged@10: 0.990
hybrid ndcg@10: 0.812
hybrid judged@10: 0.980
reranked ndcg@10: 0.798
reranked judged@10: 0.980
planned comparisons: 3
p-value threshold: 0.0167
interval level: 0.9833
chance of a false positive, all gaps noise, tests independent: 0.143
semantic minus bm25: 0.105
semantic minus bm25 kind: planned
semantic minus bm25 interval: 0.047 to 0.205
semantic minus bm25 permutation p: 0.001
semantic minus bm25 t-test p: 0.003
semantic minus bm25 effect size: 0.601
semantic minus bm25 wins, losses, ties: 21 / 7 / 2
semantic minus bm25 established: yes
semantic minus bm25 t-test agrees: yes
semantic minus bm25 interval agrees: yes
hybrid minus semantic: -0.018
hybrid minus semantic kind: planned
hybrid minus semantic interval: -0.048 to 0.006
hybrid minus semantic permutation p: 0.125
hybrid minus semantic t-test p: 0.123
hybrid minus semantic effect size: -0.290
hybrid minus semantic wins, losses, ties: 9 / 14 / 7
hybrid minus semantic established: no
hybrid minus semantic t-test agrees: yes
hybrid minus semantic interval agrees: yes
reranked minus hybrid: -0.014
reranked minus hybrid kind: planned
reranked minus hybrid interval: -0.053 to 0.020
reranked minus hybrid permutation p: 0.382
reranked minus hybrid t-test p: 0.375
reranked minus hybrid effect size: -0.165
reranked minus hybrid wins, losses, ties: 11 / 12 / 7
reranked minus hybrid established: no
reranked minus hybrid t-test agrees: yes
reranked minus hybrid interval agrees: yes
reranked minus semantic: -0.032
reranked minus semantic kind: exploratory
reranked minus semantic interval: -0.078 to -0.003
reranked minus semantic permutation p: 0.048
reranked minus semantic t-test p: 0.048
reranked minus semantic effect size: -0.377
reranked minus semantic wins, losses, ties: 9 / 16 / 5
reranked minus semantic established: no
reranked minus semantic t-test agrees: yes
reranked minus semantic interval agrees: no
planned comparisons established: 1
planned comparisons with a disagreement: 0
未经判定的商品对 BM25 的影响最重:它的 judged@10 为 0.927,而语义检索为 0.990,混合排序为 0.980,重排序后的混合排序为 0.980。
进行 3 项检验、每项取 0.05 时,若每个差距都只是噪声且各检验相互独立,至少出现一次假阳性的概率将为 0.143。Bonferroni 校正无论哪种情形都把它限制在 0.05。下面是在测试查询上以 nDCG@10 得到的结果,区间的水平为 0.9833:
| 度量 | 语义检索减 BM25 | 混合排序减语义检索 | 重排序减混合排序 | 重排序减语义检索 |
|---|---|---|---|---|
| 类别 | planned | planned | planned | exploratory |
| 平均差 | 0.105 | -0.018 | -0.014 | -0.032 |
| 区间 | 0.047 to 0.205 | -0.048 to 0.006 | -0.053 to 0.020 | -0.078 to -0.003 |
| 置换检验 p 值 | 0.001 | 0.125 | 0.382 | 0.048 |
| t 检验 p 值 | 0.003 | 0.123 | 0.375 | 0.048 |
| 效应量 | 0.601 | -0.290 | -0.165 | -0.377 |
| 胜 / 负 / 平局 | 21 / 7 / 2 | 9 / 14 / 7 | 11 / 12 / 7 | 9 / 16 / 5 |
| 是否确立 | yes | no | no | no |
在 3 项计划内比较中,有 1 项得以确立:语义检索相对 BM25 的领先,其置换检验 p 值为 0.001。混合排序相对语义检索的差异,以及重排序器相对混合排序的差异,均未得到确立:在这些查询上,两项改动都没有显示出改进,也没有被证明更差。评估框架会用置换检验来核对 t 检验和区间:有 0 项计划内比较存在分歧。
探索性比较正是各项核对开始出现分歧之处。它的置换检验 p 值为 0.048,需要与未校正时的 0.05 以及校正后的 0.0167 相比较,其区间为 -0.078 to -0.003。
t 检验与置换检验的结论一致,但排除了 0 的区间却不一致。当询问每项核对是否与置换检验一致时,评估框架对 t 检验的回答是 yes,对区间的回答是 no:区间排除 0,而 p 值却高于阈值,这正是覆盖率检查所警示的那类分歧。无论数值如何,这项比较都无法确立,因为它是在看到测试均值之后才选定的。它至多为新的查询提出一个问题:这个重排序器是否落后于语义检索?
第 1 章的决定依然成立,依据的规则并不取决于哪个排序器得分最高:要超越的排序器,是没有任何计划内比较证明其可被超越的最简单的流水线。语义检索胜过了 BM25,而后面的两个阶段都没有胜过其前一阶段。
**工作准则:**对每一项改动,报告每次运行的平均 nDCG@10 和 judged@10,然后给出一张与此类似的表,每项比较各占一列,并标明哪些是探索性比较。只有当一项改动相对于要超越的排序器得以确立时,它才能上线。
生产环境中会出什么问题
单一的查询样本可能产生误导。 测试查询上的结果取决于哪些查询恰好落入测试集。Fuhr 认为简单的留出法并不可靠,因为其结果可能在很大程度上取决于划分方式,他转而主张使用交叉验证(Fuhr, 2017, section 2.5)。Sakai 也认为“不同的训练/验证/测试划分可能产生不同的结果”(Sakai, 2020)。本书采用一种固定的划分,使每一章都在相同的测试查询上报告结果。要了解差距可能会变动多大,可以对测试框架中的探索性比较,在训练查询上也运行一遍。请将其视为说明,而不是证据,因为训练查询是用于调优的,该规则对此有明确规定:
1"""Runs the reranked-minus-semantic comparison on the test queries and on the train queries, so the verdict can be compared across the two samples. The prose's point rests on the two gap lines and the higher-scoring-run lines."""2from lib.fusion import reciprocal_rank_fusion3from lib.lexical import LexicalSearch4from lib.metrics import evaluate5from lib.recorded import cross_encoder_scores6from lib.semantic import SemanticSearch7from lib.significance import compare8from lib.store import load_products, load_queries, qrels910lexical, embeddings = LexicalSearch(load_products()), SemanticSearch()11score = {(query_id, product_id): value for query_id, product_id, value in cross_encoder_scores().itertuples(index=False)}1213for split in ["test", "train"]: # the same comparison on another sample of the store's queries14 queries, judgments = load_queries(split), qrels(split)15 semantic = embeddings.run(queries, k=100)16 hybrid = reciprocal_rank_fusion([lexical.run(queries, k=100), semantic])17 # as in chapter 1: rerank the top 50 and keep the rest in hybrid order18 reranked = {query_id: sorted(ranking[:50], key=lambda product_id: score[(query_id, product_id)], reverse=True)19 + ranking[50:] for query_id, ranking in hybrid.items()}20 result = compare(evaluate(reranked, judgments)["ndcg@10"], evaluate(semantic, judgments)["ndcg@10"])21 gap = result["mean difference"]22 print(f"{split} queries:", len(judgments))23 print(f"{split} reranked minus semantic: {gap:.3f}")24 print(f"{split} higher-scoring run:", "reranked" if gap > 0 else "semantic" if gap < 0 else "neither")25 if split == "test":26 print(f"test permutation p: {result['permutation p']:.3f}")
python split-changes-the-verdict.pytest queries: 30
test reranked minus semantic: -0.032
test higher-scoring run: semantic
test permutation p: 0.048
train queries: 50
train reranked minus semantic: 0.014
train higher-scoring run: reranked
得分更高的运行在测试查询上是 semantic,在训练查询上是 reranked,因此哪个运行得分更高会随划分而翻转。
在 30 条测试查询上,重排序的混合检索减去语义检索的差值为 -0.032(置换检验 p = 0.048)。在 50 条训练查询上,该差值为 0.014。在单一样本上,p 值如此接近 0.05,是一个薄弱的依据。第 11 章讨论如何让测试查询不参与任何选择。
重复使用测试查询就是在检验多个假设。 你在同一批查询上测试的每一项改动,都会增加一次比较,即便相隔数月也是如此。Fuhr 指出,“集合的重复使用同样属于多重假设检验”(Fuhr, 2017, section 2.7)。请记录在测试查询上进行过的比较,并在设定阈值时将它们计入。
得分相同会使运行结果无法重复。 两个得分相同的商品可能以任意一种顺序返回,而当得分相同时,多线程索引可能使顺序变得不确定(Lin and Yang, 2019)。即使顺序可以重复,它也可能是任意的:并列的次序取决于某个库的 top-k 选择,或稳定排序的输入恰好留下的顺序。因此,本书的各模块按 product_id 来打破并列。LexicalSearch 和 lib.fusion 按它对相同得分排序,而 SemanticSearch 则对按 product_id 顺序存储的向量做稳定排序。在所有版本都已固定的情况下,re-check examples 会对照记录的输出重新运行每个示例,因此其中任何一种顺序发生变化都会被发现。在你自己的系统中,请在评分之前按固定且不携带任何信息的顺序来打破并列,例如按 product_id。
未经评判的商品会拉低排序器的得分。 如果排序器给出了无人评判过的商品,即使这些商品很好,也会因此损失 nDCG。测试之前请检查 judged@10,当它下降时,请让人对新商品做出评判。
工作准则: 记录你在测试查询上运行的每一次比较;将临界结果或探索性结果视为需要用新查询来检验的问题;按固定顺序打破得分并列;并在每个 nDCG@10 旁报告 judged@10。
本章做出的决定
下列每项选择,都是本书此后的默认做法。后续章节只有在说明理由时,才会改动其中一项。
- 主要指标:采用线性增益的 nDCG@10(等级 2、1 和 0),折扣为 log2(i + 1),理想排序由所有已标注的商品构成。同时报告 precision@10 和 MRR。“相关”指等级为 exact(完全匹配)。
- 我们自己实现的指标是基准。 本书报告
lib.metrics的结果,并用 scikit-learn 的ndcg_score加以核对,每一行都包含所有已标注的商品。 - 每次比较都给出逐查询的差异: 均值和中位数、胜、负与平的数量、损失最大的查询,以及变化最大的那些查询的 judged@10。
- 认定差异成立的规则: 先列出在测试查询上计划进行的比较。当某个差异的置换检验(9,999 次随机符号翻转)给出的 p 值低于 0.05 除以计划比较的数量时,该差异成立。同时报告配对 t 检验和相应水平的 BCa 自助法区间,两者不一致时需要指明。在看到测试结果之后才选定的比较,标注为探索性比较,绝不称之为成立。训练查询只用于调参。
- 对于未能成立的差距,报告为“未显示出改进”, 绝不报告为退步,也不对其作任何解释。
- 所有重采样都使用带固定种子的
rng, 并通过lib.significance进行。 - 计划比较给我们的启示: 在第 1 章的三项比较中,语义检索相对 BM25 的领先成立,另外两项不成立,因此语义检索仍是需要超越的排序器。此后的每项改动,都在测试框架中与它所改动的排序器比较,只有胜过这个需要超越的排序器,才会被采用。
本章,在本版次中
版次 2026.11.10示例于 2026年11月19日 通过本章 12个示例 个示例中有 12 个通过
译自英文版图书,版次 2026.11.10。