衡量搜尋
閱讀約需 75 分鐘 分鐘,13 節 中共有 12 個範例。
讀完本章後,你將能從頭計算標準的排序指標,並以 scikit-learn 驗證結果;能在比較排序器之前選定一項指標;也能判斷兩個排序器在你的測試查詢上的差距,是已經確立的差異,還是可能出於偶然。
第 1 章最後得出了一個排序,並以語意檢索作為要超越的排序器。在商店的測試查詢上(這些搜尋沒有任何排序器針對它們調校過),BM25 的 nDCG@10 為 0.725,語意檢索為 0.830,混合式為 0.812,重新排序後的混合式為 0.798。在相信這個排序之前,必須先回答兩個問題。這些指標衡量的是否是對購物者重要的事,而且計算是否正確?另外,這些差距是真實的,還是在 30 筆查詢上單憑偶然就可能出現?本章回答這兩個問題,並以本書對「已確立的差異」所訂的規則,以及一套套用該規則的測試框架作結。
此部分尚未翻譯,因此以英文呈現。
Count the relevant products in the top k
Most ranking metrics start from one question: how many of the results a shopper sees are relevant? Precision at k, written precision@k, is the share of the top k results that are relevant. Recall at k, recall@k, is the share of all relevant products that appear in the top k. Manning and colleagues describe precision as the share of retrieved documents that are relevant and recall as the share of relevant documents that are retrieved, and they note that raising one usually costs the other (Manning et al., 2008, section 8.3). Both need a yes-or-no idea of relevant. In this book a product is relevant to a query when its judgment is exact, grade 2. A product nobody judged for the query counts as not relevant.
BM25's top 10 for the store's test query clear shower curtain mixes clear curtains, curtains that aren't clear, and shower curtain hooks. How many of those ten are exact, and what is the most any test query could score? The example answers both by hand and checks the count against lib.metrics:
1"""Counts the exact products in BM25's top 10 for one query, computes precision and recall by hand, and checks them against lib.metrics, then finds the best score each test query could reach. The prose's point rests on the "exact in the top 10" and "exact products judged" lines, and on the ceiling lines after the blank line."""2import pandas as pd34from lib.lexical import LexicalSearch5from lib.metrics import precision_at_k, recall_at_k6from lib.store import load_products, load_queries, qrels78queries, judgments = load_queries("test"), qrels("test")9ranking = list(LexicalSearch(load_products()).search(queries.at[246, "query"], k=10).index) # clear shower curtain10# grade 2 is exact; a product the judges never graded is absent from this dict, so it is not exact11exact = {product_id for product_id, grade in judgments[246].items() if grade == 2}1213found = sum(product_id in exact for product_id in ranking[:10])14print("exact in the top 10:", found)15print("exact products judged:", len(exact))16print(f"precision@10 by hand: {found / 10:.3f}")17print(f"recall@10 by hand: {found / len(exact):.3f}")18print(f"precision@10 from lib.metrics: {precision_at_k(ranking, judgments[246], 10):.3f}")19print(f"recall@10 from lib.metrics: {recall_at_k(ranking, judgments[246], 10):.3f}")2021# the best any ranking can do: a top 10 that holds as many exact products as the query has22exact_counts = pd.Series({query_id: sum(grade == 2 for grade in per_query.values())23 for query_id, per_query in judgments.items()})24# a top 10 can hold at most 10 exact products, and no more than the query has25best_precision = exact_counts.clip(upper=10) / 1026# only 10 results are shown, so at most 10 of a query's exact products can be retrieved27best_recall = exact_counts.clip(upper=10) / exact_counts28print()29print("queries where precision@10 can't reach 1:", (best_precision < 1).sum())30print("fewest exact products for a query:", exact_counts.min())31print("queries with that few:", (exact_counts == exact_counts.min()).sum())32print(f"best precision@10 for those queries: {best_precision.min():.3f}")33print("queries where recall@10 can't reach 1:", (best_recall < 1).sum())34most = exact_counts.idxmax()35print("query with the most exact products:", queries.at[most, "query"])36print("its exact products:", exact_counts[most])37print(f"its best recall@10: {best_recall[most]:.3f}")
python precision-recall-by-hand.pyexact in the top 10: 4
exact products judged: 12
precision@10 by hand: 0.400
recall@10 by hand: 0.333
precision@10 from lib.metrics: 0.400
recall@10 from lib.metrics: 0.333
queries where precision@10 can't reach 1: 7
fewest exact products for a query: 4
queries with that few: 4
best precision@10 for those queries: 0.400
queries where recall@10 can't reach 1: 22
query with the most exact products: outdoor light fixtures
its exact products: 170
its best recall@10: 0.059
lib.metrics agrees with the hand count. Of the top 10, 4 are exact, so precision@10 is 0.400. Against the 12 exact products the judges marked for this query, recall@10 is 0.333. precision_at_k and recall_at_k give the same two values.
Neither metric has one ceiling for every query. A query with fewer than 10 exact products can't fill its top 10 with them, so precision@10 can't reach 1. 7 test queries are like that, and the 4 with only 4 exact products can reach a precision@10 of 0.400 at best.
A query with more than 10 exact products can't reach full recall in 10 results. That holds for 22 of the test queries. The query with the most, outdoor light fixtures, has 170 exact products, so its recall@10 tops out at 0.059. A mean over the test queries mixes queries that can score 1 with queries that can't.
Recall has a second problem, because its denominator counts only the exact products the judges found. WANDS judged candidates gathered for each query, not the whole catalog (chapter 1), so some exact products were never judged. Zobel found that judgments gathered this way overestimate recall (Know where the judgments came from).
Working rule: Use precision@10 when you care how much of the first page is right, and read it next to each query's ceiling. Don't make recall@10 a headline in product search, where a query can have more than a hundred relevant products, as outdoor light fixtures has. We recommend recall at a deep cutoff, such as the top 100, for one job only: checking that retrieval passes enough relevant products to a reranker. Even then, read it as an overestimate.
此部分尚未翻譯,因此以英文呈現。
Score the first relevant result with reciprocal rank
Precision counts a whole page, but some shoppers want one product: a replacement part, or a model they already know. For them, what matters is how far down the first good result sits. Reciprocal rank (RR) scores a query as 1 divided by the position of the first relevant result, or 0 if there is none, and the mean of RR over queries is the mean reciprocal rank (MRR). TREC, the Text REtrieval Conference, runs shared evaluations of search systems. MRR was the official measure of its question-answering track at TREC-8, where each question scored the reciprocal of the rank of the first correct answer, or 0 when none of the five answers was correct. The track chose it partly because it stays between 0 and 1 and averages well (Voorhees, 1999).
Averaging RR is disputed. Fuhr points out that under RR "the difference between ranks 1 and 2 is the same as that between ranks 2 and ∞", so RR isn't an interval scale, one on which equal steps mean equal differences, and he argues that its mean shouldn't be computed (Fuhr, 2017, section 2.1). Sakai sees no reason to forbid it (Sakai, 2020). Moffat argues that such metrics are more meaningful in their current form than in the intervalized versions proposed for them (Moffat, 2022). We use MRR with that caveat: it is a coarse summary.
RR depends on where you draw the line for relevant, so score BM25's ranking for clear shower curtain twice: once counting only exact products, once counting partial ones too. Then compute MRR for BM25 and semantic retrieval over the test queries:
1"""Writes reciprocal rank by hand for one query under two grade thresholds, then computes MRR for BM25 and semantic retrieval and checks it against lib.metrics. The prose's point rests on the two rr lines for clear shower curtain and the two MRR lines."""2import pandas as pd34from lib.lexical import LexicalSearch5from lib.metrics import evaluate6from lib.semantic import SemanticSearch7from lib.store import load_products, load_queries, qrels8910# relevant=2 counts only exact products; relevant=1 counts partial ones too11def reciprocal_rank(ranking, judgments, relevant=2):12 """1 / the position of the first relevant product, or 0 if the ranking has none."""13 # positions count from 1, as a shopper counts results14 for position, product_id in enumerate(ranking, start=1):15 # a product nobody judged has grade 0, so it never counts as relevant16 if judgments.get(product_id, 0) >= relevant:17 return 1 / position18 return 0.0192021queries, judgments = load_queries("test"), qrels("test")22# each run goes 100 products deep, so RR can look past the top 10 for the first exact product23runs = {"bm25": LexicalSearch(load_products()).run(queries, k=100), "semantic": SemanticSearch().run(queries, k=100)}2425ranking = runs["bm25"][246] # clear shower curtain26print(f"rr, exact counts as relevant: {reciprocal_rank(ranking, judgments[246]):.3f}")27print(f"rr, partial or better counts: {reciprocal_rank(ranking, judgments[246], relevant=1):.3f}")2829for name, run in runs.items():30 rr = pd.Series({query_id: reciprocal_rank(run[query_id], judgments[query_id]) for query_id in judgments})31 print(f"{name} mrr: {rr.mean():.3f}")32 print(f"{name} mrr from lib.metrics: {evaluate(run, judgments)['rr'].mean():.3f}")33 print(f"{name} queries with rr of 1:", (rr == 1).sum())
python reciprocal-rank-by-hand.pyrr, exact counts as relevant: 0.500
rr, partial or better counts: 1.000
bm25 mrr: 0.650
bm25 mrr from lib.metrics: 0.650
bm25 queries with rr of 1: 16
semantic mrr: 0.787
semantic mrr from lib.metrics: 0.787
semantic queries with rr of 1: 22
The grade threshold changes RR for this one query. BM25's first result for clear shower curtain is a partial match and its second is exact, so RR is 0.500 when only exact products count, and 1.000 when partial ones count too. Whenever you quote RR, say which grades count. The mean over the test queries matches the rr column of lib.metrics.evaluate: BM25 scores 0.650 and semantic retrieval 0.787. That column looks for the first exact product anywhere in the run, not only in the top 10.
RR has a blind spot. It is 1 for 22 test queries under semantic retrieval and for 16 under BM25, and on those queries it sees nothing below the first result. A ranker that puts one exact product first and nine irrelevant ones after it scores the same as a perfect top 10.
Working rule: Report MRR as the headline only when shoppers need one right product, as for a model number or a named product such as kohler whitehaven farmhouse kitchen sink. For queries where shoppers browse a page of options, such as wooden chair outdoor, use nDCG@10 as the headline and report MRR beside it.
從折損增益建立 nDCG
與精確率和 RR 不同,第 1 章用來評分單一排名的 nDCG@10 會讀取等級。每個結果都以其等級作為增益,再除以一個隨位置增大的折損,其總和即為折損累積增益(discounted cumulative gain,DCG)。正規化 DCG(normalized DCG,nDCG)則是將某個排名的 DCG 除以理想排名的 DCG。其中有兩個部分存在多種版本,而這些版本會改變數值。第一個是理想排名。提出 DCG 的 Järvelin 與 Kekäläinen,是以「搜尋主題的召回基礎,而非某種 IR 技術的結果」為依據(Järvelin and Kekäläinen, 2002)。實務上,這表示要採用針對該查詢所有已評判商品的最佳順序,而不只是排序器所回傳商品的最佳順序。第二個是折損:他們提出將文件的分數除以其名次的對數,而各版本的差異在於採用哪一種對數,以及從哪個名次開始折損。
lib.metrics 中的公式如下,其中 grade(i) 是位置 i 上商品的等級:
DCG@k = sum over positions i from 1 to k of grade(i) / log2(i + 1)
nDCG@k = DCG@k / DCG@k of the ideal ranking
這並不是唯一通行的版本。Järvelin 與 Kekäläinen 對名次小於對數底數的結果不做折損,因此以 2 為底時,前兩個位置完全計入。Burges 等人使用指數增益 2^grade − 1,並搭配 log(1 + rank) 折損(Burges et al., 2005)。評估工具之間也有差異:ir-measures 文件為 pytrec_eval 與 gdeval 列出了不同的 nDCG 選項(ir-measures)。因此每當引用 nDCG 時,都應註明所用的變體。
請從頭撰寫 DCG 與 nDCG,並以 lib.metrics 在第 1 章的 BM25、語意與混合檢索結果上加以核對。其中有一個分數好得令人難以置信:
1"""Writes DCG and nDCG from scratch and checks them against lib.metrics on every test query of three runs, then prints one query's grades beside its best possible grades. The prose's point rests on the largest-difference line and on the ndcg@10 line for luau string lights, read with the grade lines above it."""2import math34from lib.fusion import reciprocal_rank_fusion5from lib.lexical import LexicalSearch6from lib.metrics import ndcg_at_k7from lib.semantic import SemanticSearch8from lib.store import load_products, load_queries, qrels91011def dcg(grades, k):12 """Each grade divided by log2 of its position plus one, added up over the top k."""13 # positions count from 1, so the first result is divided by log2(2)14 return sum(grade / math.log2(position + 1) for position, grade in enumerate(grades[:k], start=1))151617def ndcg(ranking, judgments, k=10):18 """DCG of the ranking over DCG of the best order of every judged product (unjudged counts as 0)."""19 # the ideal ranking: every judged product, best grade first20 ideal = dcg(sorted(judgments.values(), reverse=True), k)21 # a product nobody judged counts as grade 0 in the ranking22 return dcg([judgments.get(product_id, 0) for product_id in ranking], k) / ideal232425queries, judgments = load_queries("test"), qrels("test")26bm25 = LexicalSearch(load_products()).run(queries, k=100)27semantic = SemanticSearch().run(queries, k=100)28runs = {"bm25": bm25, "semantic": semantic, "hybrid": reciprocal_rank_fusion([bm25, semantic])}2930gaps = [abs(ndcg(run[query_id], judgments[query_id]) - ndcg_at_k(run[query_id], judgments[query_id], 10))31 for run in runs.values() for query_id in judgments]32print("rankings checked:", len(gaps))33print(f"largest difference from lib.metrics: {max(gaps):.3f}")3435ranking, per_query = bm25[461], judgments[461] # luau string lights36print()37print("query:", queries.at[461, "query"])38print("grades in the top 10:", [per_query.get(product_id, 0) for product_id in ranking[:10]])39print("best possible grades:", sorted(per_query.values(), reverse=True)[:10])40print("exact products judged:", sum(grade == 2 for grade in per_query.values()))41print("exact in the top 10:", sum(per_query.get(product_id, 0) == 2 for product_id in ranking[:10]))42print(f"ndcg@10: {ndcg(ranking, per_query):.3f}")
python ndcg-from-scratch.pyrankings checked: 90
largest difference from lib.metrics: 0.000
query: luau string lights
grades in the top 10: [2, 2, 2, 2, 1, 1, 1, 1, 1, 1]
best possible grades: [2, 2, 2, 2, 1, 1, 1, 1, 1, 1]
exact products judged: 4
exact in the top 10: 4
ndcg@10: 1.000
在全部 90 個排名中,手寫版本與 lib.metrics 的差異從未超過 0.000。
對於 luau string lights,BM25 的前 10 名包含 4 個完全符合的商品,得分為 1.000。這並非錯誤。該查詢只有 4 個已評判的完全符合商品,所以最佳的前 10 名會先放這些商品,再放部分符合的商品,而 BM25 找到了這個順序。nDCG@10 為 1 的意思是「與已評判商品的最佳順序一樣好」,而不是「十個完全符合的商品」。
**實用原則:**使用線性增益、log2(i + 1) 的形式,截斷點取 10,理想排名則由所有已評判商品建立,並且每當引用數值時都要說明這個變體。nDCG 只應在以相同評判評分的結果之間比較:新的評判會改變理想排名,進而改變每一個分數。
與 scikit-learn 的 ndcg_score 相符
將指標與獨立計算該指標的函式庫比對,是一種檢查方式。在 scikit-learn 1.9.1 中,ndcg_score 使用的變體與 lib.metrics 相同:等級即為增益,每個增益除以 log2(名次 + 1),而正規化用的 DCG,則是把相同等級依最佳順序排序後,在 k 處截斷所得的 DCG(scikit-learn 原始碼)。其文件引用了 Järvelin 與 Kekäläinen 的論文,但程式碼實作的是後來的名次 + 1 形式(scikit-learn 文件)。有三個細節,會讓直接呼叫的結果與你自己的計算不一致:
- 它只對你傳入的產品評分,因此僅以這些產品建立理想排名。
- 預設情況下,它會對分數相同的產品,取其增益的平均值(scikit-learn 文件)。這等同於對這些同分產品的所有排列順序取平均(McSherry and Najork, 2008)。
- 它每個查詢需要一列資料,且每列長度必須相同,並回傳各列的平均值(scikit-learn 原始碼)。
先做直接呼叫:ndcg_score 取得 BM25 對 clear shower curtain 前 10 名的等級,並給予能使它們維持 BM25 順序的分數。接著,依 lib.metrics 看待查詢的方式建立每個查詢的資料列,再比較這三次執行的結果:
1"""Compares scikit-learn's ndcg_score with lib.metrics for one query, first with only the top 10 passed and then with every judged product in the row, and then on every test query of three runs. The prose's point rests on the ndcg@10 lines for clear shower curtain and on the 5-product lines."""2from sklearn.metrics import ndcg_score34from lib.fusion import reciprocal_rank_fusion5from lib.lexical import LexicalSearch6from lib.metrics import ndcg_at_k7from lib.semantic import SemanticSearch8from lib.store import load_products, load_queries, qrels91011def sklearn_ndcg(ranking, judgments, k=10):12 """One row for one query: every judged product, plus any retrieved product nobody judged."""13 # every judged product once, then the retrieved products nobody judged14 products = list(dict.fromkeys([*judgments, *ranking]))15 position = {product_id: i for i, product_id in enumerate(ranking)}16 grades = [[judgments.get(product_id, 0) for product_id in products]]17 # ndcg_score ranks by score, so the negated position puts rank 1 highest18 scores = [[-position.get(product_id, len(ranking)) for product_id in products]] # not retrieved: below all19 return ndcg_score(grades, scores, k=k)202122queries, judgments = load_queries("test"), qrels("test")23bm25 = LexicalSearch(load_products()).run(queries, k=100)24semantic = SemanticSearch().run(queries, k=100)25runs = {"bm25": bm25, "semantic": semantic, "hybrid": reciprocal_rank_fusion([bm25, semantic])}2627top = bm25[246][:10] # clear shower curtain28# scores 10 down to 1 put the top 10 in BM25's order, so the first product scores highest29naive = ndcg_score([[judgments[246].get(product_id, 0) for product_id in top]], [list(range(10, 0, -1))], k=10)30print(f"ndcg@10, only the top 10 passed: {naive:.3f}")31print(f"ndcg@10 from lib.metrics: {ndcg_at_k(top, judgments[246], 10):.3f}")32print(f"ndcg@10, every judged product passed: {sklearn_ndcg(top, judgments[246]):.3f}")3334gaps = [abs(sklearn_ndcg(run[query_id], judgments[query_id]) - ndcg_at_k(run[query_id], judgments[query_id], 10))35 for run in runs.values() for query_id in judgments]36print("rankings compared:", len(gaps))37print(f"largest difference: {max(gaps):.3f}")3839short = bm25[246][:5] # a ranking shorter than k40print(f"5-product ranking, lib.metrics: {ndcg_at_k(short, judgments[246], 10):.3f}")41print(f"5-product ranking, scikit-learn: {sklearn_ndcg(short, judgments[246]):.3f}")
python ndcg-matches-scikit-learn.pyndcg@10, only the top 10 passed: 0.901
ndcg@10 from lib.metrics: 0.615
ndcg@10, every judged product passed: 0.615
rankings compared: 90
largest difference: 0.000
5-product ranking, lib.metrics: 0.539
5-product ranking, scikit-learn: 0.629
若只傳入前 10 名,ndcg_score 回傳 0.901,而 lib.metrics 的 0.615 與之不同,原因在於它只除以這十個產品的最佳順序,因此看不到 BM25 漏掉的那些確切產品。lib.metrics 則除以該查詢所有已評判產品(253 個)的最佳順序。範例中的 sklearn_ndcg 以一套做法修正了這個問題。每一列包含所有已評判的產品,以及任何檢索到但無人評判的產品(等級為 0)。這套做法依名次為檢索到的產品評分,並給其他每個產品同一個低於所有檢索產品的分數。採用這套做法後,兩者在全部 90 個排名上都一致(最大差異為 0.000)。請對每個查詢各呼叫一次 ndcg_score,再自行平均結果,因為每個查詢的資料列長度不同。
這套做法有一個限制,較短的執行結果會顯現出來。ndcg_score 把執行中未檢索到的產品,視為在前 k 名內的空位上同分,並以其平均增益計入這些空位。lib.metrics 則不給空位任何分數,因為購物者在那裡看不到任何東西。把 BM25 的排名截短為 5 個產品,兩者就會不一致:lib.metrics 為 0.539,而 ndcg_score 為 0.629。過短的執行結果確實會發生:LexicalSearch 只回傳分數大於 0 的產品。
實用原則: 報告你自己計算的 nDCG,並以 ndcg_score 作為獨立檢查,讓每個查詢的資料列包含所有已評判的產品,且分數取自名次。在比較兩者之前,先確認每次執行都至少包含 k 個產品。若你改為傳入排序器的原始分數,請先以固定順序打破同分,例如依 product_id 排序,否則兩者在分數相同之處會不一致。
在比較之前先選定指標
每一種指標都是對購物者的一種模型。Precision@10 假設購物者會看完整個第一頁,並計算頁面上好商品的數量。RR 假設購物者看到第一個好商品就會停止。nDCG@10 假設注意力會隨著頁面往下而衰減,而且完全符合的商品比部分符合的商品更有價值:依本書的等級,價值是兩倍。指數增益 2^grade − 1(如 Burges et al. (2005))會把等級轉換為 0、1 和 3 的增益,因此一個完全符合的商品相當於三個部分符合的商品。就經驗法則而言,當兩個排序器差距很大時,這些模型的結論會一致。當排序器的表現接近時,指標本身就可能決定誰勝出,這正是要在查看結果之前先選定指標的原因。Fuhr 主張事先陳述假設,形式如「我們想使用指標 X 比較方法 A 與 B」(Fuhr, 2017, 第 2.6 節)。
以語意檢索(要擊敗的排序器)和混合式檢索為例,第 1 章無法將兩者區分開來。選擇指標屬於調整評估的一部分,所以要在 50 筆訓練查詢上做選擇,並把測試查詢留到報告時使用。用四種方式為兩個排序器評分:
1"""Scores semantic retrieval and the hybrid four ways on the train queries and names the higher scorer for each metric. The prose's point rests on the "higher" lines, the count of metrics each ranker wins, and the largest gap."""2from lib.fusion import reciprocal_rank_fusion3from lib.lexical import LexicalSearch4from lib.metrics import evaluate5from lib.semantic import SemanticSearch6from lib.store import load_products, load_queries, qrels78queries, judgments = load_queries("train"), qrels("train") # choosing a metric is tuning: use train9bm25 = LexicalSearch(load_products()).run(queries, k=100)10semantic = SemanticSearch().run(queries, k=100)11runs = {"semantic": semantic, "hybrid": reciprocal_rank_fusion([bm25, semantic])}1213exponential = {query_id: {product_id: 2 ** grade - 1 for product_id, grade in per_query.items()}14 for query_id, per_query in judgments.items()} # gains 0, 1, 3 instead of 0, 1, 215means = {}16for name, run in runs.items():17 scores = evaluate(run, judgments)18 means[name] = {"precision@10": scores["precision@10"].mean(), "mrr": scores["rr"].mean(),19 "ndcg@10": scores["ndcg@10"].mean(),20 "ndcg@10 with exponential gains": evaluate(run, exponential)["ndcg@10"].mean()}2122print("train queries:", len(judgments))23winners = []24for metric in means["semantic"]:25 for name in runs:26 print(f"{name} {metric}: {means[name][metric]:.3f}")27 # max keeps the first name it meets, so on an exact tie the order of runs decides28 winners.append(max(runs, key=lambda name: means[name][metric]))29 print(f"higher {metric}: {winners[-1]}")30for name in runs:31 print(f"metrics won by {name}:", winners.count(name))32print(f"largest gap: {max(abs(means['semantic'][m] - means['hybrid'][m]) for m in means['semantic']):.3f}")
python metric-choice-on-train.pytrain queries: 50
semantic precision@10: 0.502
hybrid precision@10: 0.488
higher precision@10: semantic
semantic mrr: 0.673
hybrid mrr: 0.710
higher mrr: hybrid
semantic ndcg@10: 0.772
hybrid ndcg@10: 0.774
higher ndcg@10: hybrid
semantic ndcg@10 with exponential gains: 0.709
hybrid ndcg@10 with exponential gains: 0.708
higher ndcg@10 with exponential gains: semantic
metrics won by semantic: 2
metrics won by hybrid: 2
largest gap: 0.037
四種指標對勝者的看法並不一致。語意檢索在其中 2 項領先,混合式檢索在 2 項領先,而且兩者之間沒有任何差距大於 0.037。當兩個排序器如此接近時,差距往哪一邊倒是由指標決定。第 1 章選定的主要指標 nDCG@10 偏向 hybrid。這些是訓練查詢,因此以上內容都無法說明哪個排序器比較好。
所以選擇應該依據購物者,而不是依據結果。商店呈現的是一頁商品,而且其判斷是分級的,因此如第 1 章所選,主要指標仍然是採用線性增益的 nDCG@10,並以 precision@10 和 MRR 作為輔助。
實用原則: 在比較排序器之前,先選定一個主要指標、其增益方式與截斷位置,並把它們寫下來。當判斷為分級、且購物者會瀏覽一整頁時,我們建議使用採線性增益的 nDCG@10。當完全符合的價值遠高於接近符合時(例如必須能裝得上的備用零件),改用指數增益;當多數查詢只有一個正確答案時,改用 MRR。在主要指標旁一併報告其他指標,而且在看到主要指標偏向哪個排序器之後,絕不更改它。
要看逐查詢的差異,而不只是平均值
以查詢為單位對主要指標取平均,是對整個商店的摘要,但購物者看到的是單一查詢的結果。TREC Robust 評測項目正是建立在這一點上。其主辦者發現,針對平均值進行調校,可能會改善原本就表現良好的查詢,有時還會犧牲表現不佳的查詢(Voorhees, 2004)。因此,在相信平均值的變化之前,請先查看逐查詢差異:每個查詢一個數字,即新排序器的分數減去舊排序器的分數。這些差異也是顯著性檢定所使用的證據。
平均而言,第 1 章中商店的混合檢索結果得分明顯高於 BM25。請以 nDCG@10 逐一查詢比較兩者:
1"""Compares the hybrid and BM25 query by query on nDCG@10, then lists the three largest losses and the largest gain with its judged@10. The prose's point rests on the mean and median difference lines and on the largest-gain lines."""2from lib.fusion import reciprocal_rank_fusion3from lib.lexical import LexicalSearch4from lib.metrics import evaluate5from lib.semantic import SemanticSearch6from lib.store import load_products, load_queries, qrels78queries, judgments = load_queries("test"), qrels("test")9bm25 = LexicalSearch(load_products()).run(queries, k=100)10hybrid = reciprocal_rank_fusion([bm25, SemanticSearch().run(queries, k=100)])11before, after = evaluate(bm25, judgments), evaluate(hybrid, judgments)1213# rounded to three decimals first, so a difference that rounds to 0 counts as a tie; sorted from the largest loss up14difference = (after["ndcg@10"] - before["ndcg@10"]).round(3).sort_values() # one value per test query15print(f"mean difference: {difference.mean():.3f}")16print(f"median difference: {difference.median():.3f}")17print("hybrid better:", (difference > 0).sum())18print("bm25 better:", (difference < 0).sum())19print("tied:", (difference == 0).sum())2021print("\nhybrid's three largest losses:")22print(queries.loc[difference.index[:3], "query"].to_frame().assign(difference=difference[:3]).to_string())23# the last one after sorting: the largest gain24largest = difference.index[-1]25print("\nlargest gain:", queries.at[largest, "query"])26print(f"its gain: {difference[largest]:.3f}")27print(f"its bm25 judged@10: {before.at[largest, 'judged@10']:.3f}")28print(f"share of the mean gain from this query: {difference[largest] / difference.sum():.3f}")2930relative = after["ndcg@10"] / before["ndcg@10"] - 1 # per-query relative gains31print(f"\nrelative gain of the means: {after['ndcg@10'].mean() / before['ndcg@10'].mean() - 1:.3f}")32print(f"mean of per-query relative gains: {relative.mean():.3f}")33print(f"its bm25 ndcg@10: {before.at[largest, 'ndcg@10']:.3f}")34print(f"its relative gain: {relative[largest]:.3f}")35print(f"mean of per-query relative gains without it: {relative.drop(largest).mean():.3f}")
python per-query-hybrid-versus-bm25.pymean difference: 0.087
median difference: 0.057
hybrid better: 17
bm25 better: 7
tied: 6
hybrid's three largest losses:
query difference
query_id
293 wood floor protector mat -0.072
292 decorative white pillow -0.069
249 bistro table and chairs -0.066
largest gain: outdoor light fixtures
its gain: 0.714
its bm25 judged@10: 0.200
share of the mean gain from this query: 0.274
relative gain of the means: 0.120
mean of per-query relative gains: 0.218
its bm25 ndcg@10: 0.286
its relative gain: 2.492
mean of per-query relative gains without it: 0.140
平均增益為 0.087,但中位數為 0.057。混合檢索在較多查詢上勝出,勝負比為 17 比 7,平手的有 6。
有一個查詢貢獻了相當大的一部分成果。outdoor light fixtures 的增益為 0.714,占總增益的 0.274,而 BM25 在該查詢上的 judged@10 為 0.200。對這個查詢而言,BM25 的前 10 名多半是無人標註過的燈泡,而第 1 章主張把它們評為 0 分是合理的,因此這項增益成立。對另一個查詢而言,排序器捨棄的未標註商品也許是好商品。混合檢索最大的三項損失,是購物者會看到搜尋變差的查詢,所以請先閱讀這些查詢。
請以絕對數值報告差異。平均值的相對增益為 0.120,但逐查詢相對增益的平均值為 0.218,這是因為逐查詢比值的平均會給 BM25 得分低的查詢最大的權重。
outdoor light fixtures 的 BM25 得分為 0.286,光是它就貢獻了 2.492 的相對增益。去掉這一個查詢後,相對增益的平均值降至 0.140。Fuhr 指出,平均值的相對變化無法可靠地說明各查詢的變化,而且「對算術平均而言,只有絕對的改善才有意義」(Fuhr, 2017, section 2.4)。
實用原則: 每次比較時,都要印出差異的平均值與中位數、勝、負、平手的計數,以及損失最大的查詢,然後閱讀這些查詢。在肯定排序器之前,先檢查變動最大的查詢的 judged@10。請報告平均值的絕對差異,而不是百分比。
以排列檢定檢驗差距
在幾十個查詢上,兩個平均值之間的差距可能純粹來自偶然:換另一批查詢取樣,就會得到不同的差距。淺層指標使這個問題更嚴重:Buckley 與 Voorhees 發現,若使用 precision at 10 這類指標,研究者需要遠多於 50 個查詢,才能可靠地區分系統(Buckley and Voorhees, 2000)。顯著性檢定(significance test)探問的是:在虛無假設(null hypothesis,即假設兩個排序器同樣好)之下,兩個排序器之間的差距會有多令人意外。它的答案是 p 值:在虛無假設下,出現至少與你所測得差距一樣大的差距之機率。依慣例,p 值低於 0.05 視為存在真實差異的證據,而當一章進行多項比較時,本書的規則會將此標準收緊。
本書中的檢定皆為成對檢定:兩個排序器在相同的查詢上評分,因此每個檢定都處理各查詢的差值,困難查詢的低分也會相互抵消。排列檢定(permutation test)直接建構出虛無假設下會看到的差距。若兩個排序器可以互換,則每個查詢的差值出現相反符號的可能性也相同。因此,隨機翻轉每個符號、重新計算平均值,重複數千次,再計算翻轉後的平均值離 0 至少與真實平均值一樣遠的次數。對 n 個查詢而言,共有 2^n 種符號模式,測試查詢的情形為 1,073,741,824,所以檢定從中抽取 9,999 種,其 p 值是一個估計值。
在測試查詢上,混合式排序落後於語意檢索,而第 1 章並未釐清這個差距是否真實。相對於語意檢索,混合式排序在測試查詢上的勝、敗與平手次數為 9 / 14 / 7。檢定是否會發現混合式排序的劣勢超出偶然的程度?先決定你的答案,再執行。
1"""Estimates the p-value of the hybrid's deficit against semantic retrieval by flipping each query's sign at random, once by hand and once with SciPy's permutation_test. The prose's point rests on the observed mean, the two p-value lines, and the line that compares the p-value with 0.05."""2import numpy as np3from scipy.stats import permutation_test45from lib.fusion import reciprocal_rank_fusion6from lib.lexical import LexicalSearch7from lib.metrics import evaluate8from lib.semantic import SemanticSearch9from lib.store import load_products, load_queries, qrels1011queries, judgments = load_queries("test"), qrels("test")12semantic = SemanticSearch().run(queries, k=100)13hybrid = reciprocal_rank_fusion([LexicalSearch(load_products()).run(queries, k=100), semantic])14after = evaluate(hybrid, judgments)["ndcg@10"].to_numpy()15before = evaluate(semantic, judgments)["ndcg@10"].to_numpy()16difference = after - before # hybrid minus semantic, per test query17print("test queries:", len(difference))18print(f"sign patterns for the test queries: {2 ** len(difference):,}")19print(f"observed mean difference: {difference.mean():.3f}")2021# if the two rankers were interchangeable, each query's difference would be as likely to have the other sign22# a fixed seed, so every run draws the same sign patterns23rng = np.random.default_rng(2026)24# 9,999 random sign patterns, one sign for each query in each25signs = rng.choice([-1, 1], size=(9999, len(difference)))26flipped = (signs * difference).mean(axis=1) # 9,999 mean differences under that assumption27# the 95th percentile of the flipped means' sizes28print(f"95% of flipped means are smaller than: {np.quantile(np.abs(flipped), 0.95):.3f}")29print(f"p-value by hand: {np.mean(np.abs(flipped) >= abs(difference.mean())):.3f}")3031result = permutation_test((after, before), lambda after_scores, before_scores, axis: np.mean(after_scores - before_scores, axis=axis),32 # samples: flip each query's pair of scores, which is the paired setting33 permutation_type="samples", rng=np.random.default_rng(2026))34print(f"p-value from scipy: {result.pvalue:.3f}")35print("difference at the 0.05 level:", "larger than chance" if result.pvalue < 0.05 else "within chance")
python permutation-test-by-hand.pytest queries: 30
sign patterns for the test queries: 1,073,741,824
observed mean difference: -0.018
95% of flipped means are smaller than: 0.022
p-value by hand: 0.117
p-value from scipy: 0.125
difference at the 0.05 level: within chance
在 0.05 的水準下,混合式排序相對於語意檢索的劣勢為within chance。觀察到的平均差值為 -0.018,而 95% 的翻轉平均值,其絕對值小於 0.022。p 值是翻轉平均值至少與觀察值一樣大的比例,為 0.117。
SciPy 的 permutation_test 搭配 permutation_type="samples"(其文件為成對檢定所指定的設定)執行後,回傳 0.125(SciPy documentation)。兩個估計值略有差異,因為各自抽取了自己的隨機符號模式。要判定為真實的劣勢,平均值必須壓過離散程度,而少數幾個查詢朝反方向擺動,就足以抹平這個差距。
SciPy 的重抽樣函式透過 rng 接收隨機產生器,它取代了 random_state 引數。在 1.18.1 中兩者皆可使用,但文件指出「新程式碼應只使用 rng 關鍵字」,因為使用 random_state 的呼叫在過渡期之後將會發出警告(SciPy documentation)。本書的每個範例都以固定種子傳入 rng。
實用原則: 對每一個你打算據以行動的差距,都以對各查詢差值進行的成對排列檢定來檢驗。本書即依據此檢定做決定,如其規則所述。若 p 值高於門檻,應解讀為「沒有證據顯示存在差異」,而非「沒有差異」:查詢數量這麼少時,檢定可能漏掉真實的差距,McKechnie 等人主張應在假陽性之外一併衡量這類漏判(McKechnie et al., 2025)。切勿解釋檢定無法與偶然區分的差距。
以成對 t 檢定取得 p 值
成對 t 檢定以公式回答同一個問題,不需要重抽樣。它將各查詢的差異之平均值除以標準誤(standard error):標準誤是差異的標準差除以查詢數的平方根,用來估計平均值在不同查詢樣本之間會移動多遠。若兩個排序器同樣好(即無真實差異),且差異大致呈常態分布,這個比值 t 會服從 t 分布:呈鐘形,尾部比常態分布更厚,形狀由自由度決定,此處的自由度為查詢數減 1。p 值是 t 在任一方向上與 0 至少相距這麼遠的機率(Sakai, 2014, section 3.1.1):
t = mean(d) / (sd(d) / sqrt(n)) d: the per-query differences, n: the number of queries
各查詢的 nDCG 差異介於 −1 與 1 之間,而且有離群值,例如outdoor light fixtures 上的增益,因此並不呈常態分布。實務上 t 檢定仍然可靠。Smucker 等人針對以平均精確率均值(MAP,另一種排序指標)評分的 TREC 執行結果,在 50 個主題上發現,隨機化檢定、自助法(bootstrap)檢定與 t 檢定在實務上差異不大,其中隨機化檢定即排列檢定(Smucker et al., 2007)。Urbano 等人在以 TREC 資料建立的模擬中,使用 25、50 與 100 個主題的集合,發現「t 檢定與排列檢定的表現幾乎相同」(Urbano et al., 2019)。兩項研究都不建議在這項工作上使用 Wilcoxon 符號秩檢定與符號檢定。此外,SciPy 的 wilcoxon 預設會以 zero_method='wilcox' 剔除所有差異恰為 0 的查詢(SciPy documentation),而兩次執行只有些微差異時,許多查詢的分數會相同(差異為 0)。
公式與重抽樣在第 1 章的差距上是否一致?取其中兩項:語意檢索對 BM25,以及混合式檢索對語意檢索。以手算得出的公式 t 值,應與 SciPy 的 ttest_rel 相符:
1"""Computes the paired t statistic and its p-value by hand for two gaps, and checks each p-value against SciPy's ttest_rel. The prose's point rests on the t and p lines printed for each gap."""2import math34from scipy.stats import t, ttest_rel56from lib.fusion import reciprocal_rank_fusion7from lib.lexical import LexicalSearch8from lib.metrics import evaluate9from lib.semantic import SemanticSearch10from lib.store import load_products, load_queries, qrels1112queries, judgments = load_queries("test"), qrels("test")13bm25 = LexicalSearch(load_products()).run(queries, k=100)14semantic = SemanticSearch().run(queries, k=100)15runs = {"bm25": bm25, "semantic": semantic, "hybrid": reciprocal_rank_fusion([bm25, semantic])}16ndcg = {name: evaluate(run, judgments)["ndcg@10"] for name, run in runs.items()} # per test query1718for after, before in [("semantic", "bm25"), ("hybrid", "semantic")]:19 difference = ndcg[after] - ndcg[before]20 n = len(difference)21 # ddof=1 divides by n - 1, the sample standard deviation the t-test uses22 standard_error = difference.std(ddof=1) / math.sqrt(n) # how much the mean would move between samples23 statistic = difference.mean() / standard_error24 p_value = 2 * t.sf(abs(statistic), df=n - 1) # both tails of the t distribution25 print(f"{after} minus {before}: {difference.mean():.3f}")26 print(f"{after} minus {before} t by hand: {statistic:.3f}")27 print(f"{after} minus {before} p by hand: {p_value:.3f}")28 print(f"{after} minus {before} p from ttest_rel: {ttest_rel(ndcg[after], ndcg[before]).pvalue:.3f}")
python paired-t-test-by-hand.pysemantic minus bm25: 0.105
semantic minus bm25 t by hand: 3.294
semantic minus bm25 p by hand: 0.003
semantic minus bm25 p from ttest_rel: 0.003
hybrid minus semantic: -0.018
hybrid minus semantic t by hand: -1.590
hybrid minus semantic p by hand: 0.123
hybrid minus semantic p from ttest_rel: 0.123
在語意檢索對 BM25 的比較中,平均差異 0.105 得出 t = 3.294,p 值為 0.003,與 ttest_rel 的結果相同。這是單憑機率很少會產生的差距。
在混合式檢索對語意檢索的比較中,t = -1.590,p 值為 0.123。這與排列檢定對同一差距得出的 0.125 很接近。
實用原則: 將成對 t 檢定 scipy.stats.ttest_rel 與排列檢定並列報告,作為不需要重抽樣的檢驗。當兩者落在門檻的不同側時,應明確說明,並以排列檢定為準。不要使用 Wilcoxon 檢定或符號檢定來比較平均分數。
回報差距有多大
p 值說明的是差距有多令人意外,而不是差距有多大。在大樣本上,幾乎任何修改都會產生顯著但微小的差異(Fuhr, 2017, section 2.8)。Sakai 建議每報告一個 p 值,就一併報告效果量與信賴區間(Sakai, 2014)。信賴區間(confidence interval)是真實平均差異的一段合理數值範圍:所謂真實平均差異,指的是以這些判斷標準為這類查詢評分、對所有查詢取得的平均,而不是針對商店搜尋流量的平均。95% 區間來自一種程序,這種程序設計成有 95% 的機會涵蓋該值。
自助法(bootstrap)以重抽樣來建立信賴區間。從手上的查詢中,以取後放回的方式抽出與原本數量相同的查詢,因此有些查詢會出現兩次,有些則完全不出現。計算平均差異,然後重複數千次。百分位區間(percentile interval)是重抽樣平均值中間的 95%。成對比較的效果量(effect size)是平均差異除以差異的標準差(Sakai, 2014, section 3.1.2)。它表示差距相對於各查詢之間變動幅度的大小。它不是 Cohen's d;Cohen's d 是用於非成對樣本的效果量,Sakai 提醒不要直接比較兩者。
手工建立的百分位區間,能否與 SciPy 的 bootstrap 搭配 method="percentile" 的結果相符?我們分別為語意檢索對 BM25、以及混合檢索對語意檢索,以手工建立區間,再逐一與 SciPy 核對,最後執行 SciPy 的預設方法:
1"""Builds a 95% percentile interval by hand from 9,999 resampled means, checks it against SciPy's percentile method, then reads SciPy's default BCa interval and the effect size for each gap. The prose's point rests on the BCa low and high lines and on the line that says whether each interval includes 0."""2import numpy as np3from scipy.stats import bootstrap45from lib.fusion import reciprocal_rank_fusion6from lib.lexical import LexicalSearch7from lib.metrics import evaluate8from lib.semantic import SemanticSearch9from lib.store import load_products, load_queries, qrels1011queries, judgments = load_queries("test"), qrels("test")12bm25 = LexicalSearch(load_products()).run(queries, k=100)13semantic = SemanticSearch().run(queries, k=100)14runs = {"bm25": bm25, "semantic": semantic, "hybrid": reciprocal_rank_fusion([bm25, semantic])}15ndcg = {name: evaluate(run, judgments)["ndcg@10"] for name, run in runs.items()} # per test query1617for after, before in [("semantic", "bm25"), ("hybrid", "semantic")]:18 difference, name = (ndcg[after] - ndcg[before]).to_numpy(), f"{after} minus {before}"19 # choice draws with replacement, so a query can appear twice in one set and not at all in another20 resampled = np.random.default_rng(2026).choice(difference, size=(9999, len(difference))) # 9,999 sets of queries21 # 2.5% in each tail leaves the middle 95%22 low, high = np.percentile(resampled.mean(axis=1), [2.5, 97.5]) # the middle 95% of their means23 print(f"{name}: {difference.mean():.3f}")24 print(f"{name} interval by hand: {low:.3f} to {high:.3f}")25 percentile = bootstrap((difference,), np.mean, method="percentile", rng=np.random.default_rng(2026))26 print(f"{name} percentile interval from scipy: "27 f"{percentile.confidence_interval.low:.3f} to {percentile.confidence_interval.high:.3f}")28 # BCa is SciPy's default method, so no method argument is given29 bca = bootstrap((difference,), np.mean, rng=np.random.default_rng(2026)).confidence_interval30 print(f"{name} bca low: {bca.low:.3f}\n{name} bca high: {bca.high:.3f}")31 print(f"{name} bca interval and 0: {'includes 0' if bca.low <= 0 <= bca.high else 'excludes 0'}")32 # ddof=1: the standard deviation of the differences, with n - 1 in the denominator33 print(f"{name} effect size: {difference.mean() / difference.std(ddof=1):.3f}")
python bootstrap-interval-by-hand.pysemantic minus bm25: 0.105
semantic minus bm25 interval by hand: 0.049 to 0.172
semantic minus bm25 percentile interval from scipy: 0.049 to 0.172
semantic minus bm25 bca low: 0.055
semantic minus bm25 bca high: 0.185
semantic minus bm25 bca interval and 0: excludes 0
semantic minus bm25 effect size: 0.601
hybrid minus semantic: -0.018
hybrid minus semantic interval by hand: -0.041 to 0.003
hybrid minus semantic percentile interval from scipy: -0.041 to 0.003
hybrid minus semantic bca low: -0.042
hybrid minus semantic bca high: 0.002
hybrid minus semantic bca interval and 0: includes 0
hybrid minus semantic effect size: -0.290
SciPy 的百分位區間與手工建立的相符:0.049 to 0.172 與 -0.041 to 0.003。它的預設方法是 BCa(bias-corrected and accelerated,偏差校正與加速),會從同一批重抽樣平均值中讀取不同的百分位數。它依重抽樣平均值偏離觀測平均值中心的程度來平移這些百分位數,並依逐一剔除單一查詢所估計的偏態來拉伸它們(SciPy source)。本書的測試框架回報 BCa,這種方法是為這類偏態的重抽樣平均值而設計。
對於語意檢索對 BM25,BCa 區間從 0.055 到 0.185,因此它excludes 0,效果量為 0.601。
對於混合檢索對語意檢索,區間從 -0.042 到 0.002。它includes 0,效果量為 -0.290。這個區間容許的情形,從損失零點零幾,到最多增益 0.002:沒有值得採用的增益。
自助法區間只是近似其信賴水準,而在幾十個查詢上,實際涵蓋率會不足。為了檢驗,我們把混合檢索相對於語意檢索的 30 筆逐查詢差異,當作全部的查詢,如此真實差距便是已知的。接著從中抽出 1000 組相同大小的新樣本,為每組建立區間,並計算區間涵蓋真實差距的頻率:
1"""Draws new samples from the test queries' differences and counts how often a BCa or a t interval covers the known gap, at two levels. The prose's point rests on the four coverage lines and the standard-error line."""2import numpy as np3from scipy.stats import bootstrap, ttest_1samp45from lib.fusion import reciprocal_rank_fusion6from lib.lexical import LexicalSearch7from lib.metrics import evaluate8from lib.semantic import SemanticSearch9from lib.store import load_products, load_queries, qrels1011queries, judgments = load_queries("test"), qrels("test")12semantic = SemanticSearch().run(queries, k=100)13hybrid = reciprocal_rank_fusion([LexicalSearch(load_products()).run(queries, k=100), semantic])14population = (evaluate(hybrid, judgments)["ndcg@10"] - evaluate(semantic, judgments)["ndcg@10"]).to_numpy()15true_gap = population.mean() # known, because these differences stand in for every query there is1617# a fixed seed, so the coverage counts repeat on every run18rng, trials = np.random.default_rng(2026), 100019print("trials:", trials)20# binomial standard error of a proportion near 0.95, over the trials21print(f"standard error of a coverage near 0.95: {(0.95 * 0.05 / trials) ** 0.5:.3f}")22for level in [0.95, round(1 - 0.05 / 3, 4)]: # the usual level, and the level for three comparisons23 covered = {"bca": 0, "t": 0}24 for _ in range(trials):25 sample = rng.choice(population, size=len(population)) # another set of 30 queries like these26 # 1,999 resamples per interval, fewer than SciPy's default, to keep 1,000 trials quick27 bca = bootstrap((sample,), np.mean, confidence_level=level, n_resamples=1999, rng=rng).confidence_interval28 t = ttest_1samp(sample, 0).confidence_interval(level) # the interval that matches the t-test29 covered["bca"] += bca.low <= true_gap <= bca.high30 covered["t"] += t.low <= true_gap <= t.high31 for method, count in covered.items():32 print(f"{method} coverage at {level}: {count / trials:.3f}")
python interval-coverage-check.pytrials: 1000
standard error of a coverage near 0.95: 0.007
bca coverage at 0.95: 0.927
t coverage at 0.95: 0.945
bca coverage at 0.9833: 0.955
t coverage at 0.9833: 0.975
在名目 95% 下,BCa 涵蓋真實差距的比例為 0.927,t 檢定的區間則為 0.945。在 0.9833(測試框架用於其規劃比較的水準)下,BCa 涵蓋 0.955,t 區間涵蓋 0.975。
每個數字上下各有約 0.007 的不確定性。BCa 在兩種水準下都未達其名目水準,而 t 區間較接近自己的名目水準。因此,對於僅僅勉強排除 0 的自助法區間,應當審慎看待。在一項模擬中,以同樣重抽樣為基礎的檢定,也就是自助位移檢定(bootstrap-shift test),「系統性地偏向較小的 p 值」(Urbano et al., 2019)。
差距是否重要,是任何檢定都無法代為做出的判斷。以 precision@10 而言,Fuhr 指出「0.02 的差異相當於每五個查詢多出一份相關文件」(Fuhr, 2017, section 2.8)。以 nDCG@10 而言,應在檢定之前先決定,多大的差距才足以抵銷一項變更的成本。
實用原則: 每一項比較都要給出平均差異,並附上其自助法區間、效果量與 p 值。如果區間包含 0,該比較就沒有顯示出改進。如果區間排除 0,但低於值得承擔該變更成本的最小差距,則差異可能是真的,卻仍不值得上線。使用區間來描述差距的大小,而不是用來判斷差距是否存在。
事先決定什麼才算確立
當差距只是雜訊時,每一次顯著水準為 0.05 的檢定都有 5% 的機率出現偽陽性,而一章若執行多次檢定,就會多次承擔這項風險。若每個差距都只是雜訊,且各檢定相互獨立,以這種方式對 10 個系統做兩兩比較,至少出現一次偽「顯著」結果的機率約為 0.90(Sakai, 2014, section 3)。Bonferroni 校正是一種簡單而保守的補救方式:檢定 m 個假設時,每個假設都以 0.05 除以 m 作為門檻(Fuhr, 2017, section 2.7)。無論各檢定是否獨立,它都能使出現任何偽陽性的機率維持在 0.05 以下或等於 0.05。但它只有在這 m 個比較於看到結果之前就已固定時才有效。Fuhr 指出了破壞這項前提的常見做法:作者先找出哪一個競爭方法在資料上表現最好,再只拿自己的方法與那一個方法比較(Fuhr, 2017, section 2.6)。
因此,本書在每一章都採用同一條規則:
- 在報告任何比較之前,先列出在測試查詢上的規劃比較。
- 當成對的排列檢定所得的 p 值低於該章的 Bonferroni 門檻時,該差異即為確立;門檻為 0.05 除以規劃比較的數量。
- 在旁邊一併報告成對 t 檢定與 bootstrap 區間,區間採用對應的水準,即 1 減去門檻。當其中任何一項與排列檢定的結論不一致時,須明確說明。
- 在看過測試結果之後才選定的比較屬於探索性比較。 須如此標示,且絕不稱之為已確立。
- 在訓練查詢上的比較用於調校, 不作為證據。
我們讓排列檢定來裁決,是因為它不對差異的分布形狀做任何假設,只假設在虛無假設下,每個查詢的正負號就像擲硬幣一樣。Urbano 等人發現兩者的表現幾乎相同(Urbano et al., 2019),這使 t 檢定成為良好的檢核。在只有幾十個查詢的情況下,bootstrap 區間的涵蓋率低於其標稱水準(涵蓋率檢查),因此區間用來描述差距,而不用來裁決差距。
第 1 章一次建構一個階段的商店管線,光是這個順序就決定了它的規劃比較:每個階段都與前一個階段比較。這包括語意檢索對 BM25、混合式檢索對語意檢索,以及重新排序的混合式檢索對混合式檢索,共 3 項。門檻為 0.0167,區間採用的水準為 0.9833。將重新排序的混合式檢索與語意檢索相比屬於探索性比較,因為語意檢索之所以成為要超越的排序器,是由於它在這些相同的測試查詢上得分最高。
實用原則: 在對測試查詢評分之前,先寫下規劃比較的清單,並保持簡短,因為每多一項比較,其他比較的門檻就會更高。當需要一次比較許多排序器時,Bonferroni 會變得過於嚴格。Sakai 建議先做變異數分析,檢驗這些排序器之間是否存在任何差異,再做隨機化 Tukey HSD 檢定,該檢定會比較每一對排序器,同時使出現任何偽陽性的機率維持在該顯著水準(Sakai, 2014)。
在評估框架中比較執行結果
本書在評估框架(evaluation harness)中套用那項規則。評估框架讓除了排序器以外的一切都維持固定:相同的查詢、判斷、指標與檢定,因此兩次執行之間只有排序結果不同。lib.metrics.evaluate 每個查詢回傳一列資料,這正是成對檢定所需的形式。compare 位於 lib.significance,這是一個彙整本章程式碼的簡短模組,會對兩次執行中某一項指標的欄位進行檢定。它會回傳平均差異、自助法(bootstrap)區間、排列檢定與 t 檢定的 p 值、效果量,以及勝、負與平手的次數。它的 verdict 函式會將本書的規則套用到結果上。請保留逐查詢的資料框,只在最後才取平均。
接著用它執行第 1 章的四個排序器,然後進行 3 項規劃比較與一項探索性比較:
1"""Runs chapter 1's four rankers through the harness and applies the Bonferroni threshold to the three planned comparisons and one exploratory comparison. The prose's point rests on the judged@10 lines, the established lines, and the exploratory comparison's p-value and agreement lines."""2from lib.fusion import reciprocal_rank_fusion3from lib.lexical import LexicalSearch4from lib.metrics import evaluate5from lib.recorded import cross_encoder_scores6from lib.semantic import SemanticSearch7from lib.significance import compare, verdict8from lib.store import load_products, load_queries, qrels910queries, judgments = load_queries("test"), qrels("test")11bm25 = LexicalSearch(load_products()).run(queries, k=100)12semantic = SemanticSearch().run(queries, k=100)13hybrid = reciprocal_rank_fusion([bm25, semantic])14score = {(query_id, product_id): value for query_id, product_id, value in cross_encoder_scores().itertuples(index=False)}15# chapter 1's rerank: only the top 50 are reordered; the rest keep their hybrid order16reranked = {query_id: sorted(ranking[:50], key=lambda product_id: score[(query_id, product_id)], reverse=True)17 + ranking[50:] for query_id, ranking in hybrid.items()}1819runs = {"bm25": bm25, "semantic": semantic, "hybrid": hybrid, "reranked": reranked}20scores = {name: evaluate(run, judgments) for name, run in runs.items()} # one row per test query21for name, frame in scores.items():22 print(f"{name} ndcg@10: {frame['ndcg@10'].mean():.3f}\n{name} judged@10: {frame['judged@10'].mean():.3f}")2324planned = [("semantic", "bm25"), ("hybrid", "semantic"), ("reranked", "hybrid")] # each stage against the last25exploratory = [("reranked", "semantic")] # chosen after chapter 1 saw the test means26threshold = 0.05 / len(planned) # Bonferroni, over the planned comparisons only27print(f"\nplanned comparisons: {len(planned)}\np-value threshold: {threshold:.4f}\ninterval level: {1 - threshold:.4f}")28# each planned test has a 0.95 chance of no false positive, if its gap is noise29print(f"chance of a false positive, all gaps noise, tests independent: {1 - 0.95 ** len(planned):.3f}")30established, disagreements = 0, 031for after, before in planned + exploratory:32 result, name = compare(scores[after]["ndcg@10"], scores[before]["ndcg@10"], confidence=1 - threshold), f"{after} minus {before}"33 # verdict runs on the exploratory one too, only for its agreement checks; it is never counted as established34 checks, kind = verdict(result, threshold), "planned" if (after, before) in planned else "exploratory"35 print(f"\n{name}: {result['mean difference']:.3f}\n{name} kind: {kind}")36 print(f"{name} interval: {result['ci low']:.3f} to {result['ci high']:.3f}")37 print(f"{name} permutation p: {result['permutation p']:.3f}\n{name} t-test p: {result['t-test p']:.3f}")38 print(f"{name} effect size: {result['effect size']:.3f}\n{name} wins, losses, ties: {result['wins']} / {result['losses']} / {result['ties']}")39 # only a planned comparison can be established; the exploratory one is labelled and never called so40 print(f"{name} established: {'yes' if checks['established'] and kind == 'planned' else 'no'}")41 for method in ["t-test", "interval"]:42 print(f"{name} {method} agrees: {'yes' if checks[method + ' agrees'] else 'no'}")43 if kind == "planned":44 established += checks["established"]45 # a planned comparison where the t-test or the interval disagrees with the permutation test46 disagreements += not (checks["t-test agrees"] and checks["interval agrees"])47print(f"\nplanned comparisons established: {established}\nplanned comparisons with a disagreement: {disagreements}")
python compare-chapter-one-rankers.pybm25 ndcg@10: 0.725
bm25 judged@10: 0.927
semantic ndcg@10: 0.830
semantic judged@10: 0.990
hybrid ndcg@10: 0.812
hybrid judged@10: 0.980
reranked ndcg@10: 0.798
reranked judged@10: 0.980
planned comparisons: 3
p-value threshold: 0.0167
interval level: 0.9833
chance of a false positive, all gaps noise, tests independent: 0.143
semantic minus bm25: 0.105
semantic minus bm25 kind: planned
semantic minus bm25 interval: 0.047 to 0.205
semantic minus bm25 permutation p: 0.001
semantic minus bm25 t-test p: 0.003
semantic minus bm25 effect size: 0.601
semantic minus bm25 wins, losses, ties: 21 / 7 / 2
semantic minus bm25 established: yes
semantic minus bm25 t-test agrees: yes
semantic minus bm25 interval agrees: yes
hybrid minus semantic: -0.018
hybrid minus semantic kind: planned
hybrid minus semantic interval: -0.048 to 0.006
hybrid minus semantic permutation p: 0.125
hybrid minus semantic t-test p: 0.123
hybrid minus semantic effect size: -0.290
hybrid minus semantic wins, losses, ties: 9 / 14 / 7
hybrid minus semantic established: no
hybrid minus semantic t-test agrees: yes
hybrid minus semantic interval agrees: yes
reranked minus hybrid: -0.014
reranked minus hybrid kind: planned
reranked minus hybrid interval: -0.053 to 0.020
reranked minus hybrid permutation p: 0.382
reranked minus hybrid t-test p: 0.375
reranked minus hybrid effect size: -0.165
reranked minus hybrid wins, losses, ties: 11 / 12 / 7
reranked minus hybrid established: no
reranked minus hybrid t-test agrees: yes
reranked minus hybrid interval agrees: yes
reranked minus semantic: -0.032
reranked minus semantic kind: exploratory
reranked minus semantic interval: -0.078 to -0.003
reranked minus semantic permutation p: 0.048
reranked minus semantic t-test p: 0.048
reranked minus semantic effect size: -0.377
reranked minus semantic wins, losses, ties: 9 / 16 / 5
reranked minus semantic established: no
reranked minus semantic t-test agrees: yes
reranked minus semantic interval agrees: no
planned comparisons established: 1
planned comparisons with a disagreement: 0
未經判斷的商品對 BM25 的影響最大:其 judged@10 為 0.927,語意檢索為 0.990,混合式為 0.980,重新排序後的混合式為 0.980。
若進行 3 項檢定,每項採 0.05,則在所有差距都只是雜訊、且各檢定相互獨立的情況下,至少出現一次偽陽性的機率為 0.143。Bonferroni 校正無論如何都能將它限制在 0.05。以下是測試查詢上的 nDCG@10,區間採用 0.9833 的水準:
| 項目 | 語意檢索減 BM25 | 混合式減語意檢索 | 重新排序減混合式 | 重新排序減語意檢索 |
|---|---|---|---|---|
| 類型 | planned | planned | planned | exploratory |
| 平均差異 | 0.105 | -0.018 | -0.014 | -0.032 |
| 區間 | 0.047 to 0.205 | -0.048 to 0.006 | -0.053 to 0.020 | -0.078 to -0.003 |
| 排列檢定 p 值 | 0.001 | 0.125 | 0.382 | 0.048 |
| t 檢定 p 值 | 0.003 | 0.123 | 0.375 | 0.048 |
| 效果量 | 0.601 | -0.290 | -0.165 | -0.377 |
| 勝/負/平手 | 21 / 7 / 2 | 9 / 14 / 7 | 11 / 12 / 7 | 9 / 16 / 5 |
| 是否成立 | yes | no | no | no |
在 3 項規劃比較中,有 1 項成立:語意檢索領先 BM25,排列檢定的 p 值為 0.001。混合式相對於語意檢索的落後,以及重新排序相對於混合式的落後,都不成立:在這些查詢上,兩項變更都沒有顯示出改善,也沒有被證明較差。評估框架會以排列檢定為基準,檢查 t 檢定與區間:有 0 項規劃比較出現不一致。
探索性比較正是各項檢查開始出現分歧之處。它的排列檢定 p 值為 0.048,要對照的是未校正時的 0.05,以及校正後的 0.0167,而它的區間範圍為 -0.078 to -0.003。
t 檢定與排列檢定的結論一致,但不含 0 的區間則不然。當詢問各項檢查是否與排列檢定一致時,評估框架對 t 檢定的回答是 yes,對區間的回答是 no:區間排除 0,而 p 值卻高於門檻,這正是涵蓋率檢查所警告的那類不一致。無論數字為何,這項比較都無法成立,因為它是在看過測試平均值之後才選定的。它至多只能為新的查詢提出一個問題:這個重新排序器是否落後於語意檢索?
第 1 章的決定依然成立,其依據的規則並不取決於哪個排序器得分最高:要超越的排序器,是沒有任何規劃比較顯示其被超越的最簡單管線。語意檢索勝過了 BM25,而後續兩個階段都沒有勝過前一個階段。
實用原則: 對每一項變更,都要報告各次執行的平均 nDCG@10 與 judged@10,再附上類似這樣的表格,每項比較各佔一欄,並標明哪些是探索性比較。只有當一項變更相對於要超越的排序器已經成立時,才可上線。
正式環境中會出什麼問題
單一查詢樣本可能造成誤導。 測試查詢上的結果,取決於哪些查詢剛好被分到測試集。Fuhr 主張,簡單的保留集(holdout)方法並不可靠,因為其結果可能高度依賴資料切分方式,因此他改而要求使用交叉驗證(Fuhr, 2017, 第 2.5 節)。Sakai 也同意「不同的訓練/驗證/測試切分可能產生不同的結果」(Sakai, 2020)。本書採用單一固定的切分,使每一章都在相同的測試查詢上回報結果。若要看差距可能變動多少,可以把測試框架中的探索性比較,也在訓練查詢上執行一次。請把這視為說明而非證據,因為訓練查詢是用來調校的,正如該規則所述:
1"""Runs the reranked-minus-semantic comparison on the test queries and on the train queries, so the verdict can be compared across the two samples. The prose's point rests on the two gap lines and the higher-scoring-run lines."""2from lib.fusion import reciprocal_rank_fusion3from lib.lexical import LexicalSearch4from lib.metrics import evaluate5from lib.recorded import cross_encoder_scores6from lib.semantic import SemanticSearch7from lib.significance import compare8from lib.store import load_products, load_queries, qrels910lexical, embeddings = LexicalSearch(load_products()), SemanticSearch()11score = {(query_id, product_id): value for query_id, product_id, value in cross_encoder_scores().itertuples(index=False)}1213for split in ["test", "train"]: # the same comparison on another sample of the store's queries14 queries, judgments = load_queries(split), qrels(split)15 semantic = embeddings.run(queries, k=100)16 hybrid = reciprocal_rank_fusion([lexical.run(queries, k=100), semantic])17 # as in chapter 1: rerank the top 50 and keep the rest in hybrid order18 reranked = {query_id: sorted(ranking[:50], key=lambda product_id: score[(query_id, product_id)], reverse=True)19 + ranking[50:] for query_id, ranking in hybrid.items()}20 result = compare(evaluate(reranked, judgments)["ndcg@10"], evaluate(semantic, judgments)["ndcg@10"])21 gap = result["mean difference"]22 print(f"{split} queries:", len(judgments))23 print(f"{split} reranked minus semantic: {gap:.3f}")24 print(f"{split} higher-scoring run:", "reranked" if gap > 0 else "semantic" if gap < 0 else "neither")25 if split == "test":26 print(f"test permutation p: {result['permutation p']:.3f}")
python split-changes-the-verdict.pytest queries: 30
test reranked minus semantic: -0.032
test higher-scoring run: semantic
test permutation p: 0.048
train queries: 50
train reranked minus semantic: 0.014
train higher-scoring run: reranked
得分較高的執行結果,在測試查詢上是 semantic,在訓練查詢上則是 reranked,因此哪一個執行結果得分較高,會隨切分而翻轉。
在 30 筆測試查詢上,重排序的混合檢索減去語意檢索的差值為 -0.032(排列檢定 p = 0.048)。在 50 筆訓練查詢上,差值為 0.014。在單一樣本上,p 值如此接近 0.05,並不是穩固的依據。第 11 章說明如何讓測試查詢不介入任何選擇。
重複使用測試查詢,等於在檢定許多假設。 你在同一批查詢上測試的每一項變更,都會增加一次比較,即使相隔數月也一樣。Fuhr 指出「重複使用測試集合也屬於多重假設檢定的情形」(Fuhr, 2017, 第 2.7 節)。請記錄在測試查詢上執行過的比較,並在設定門檻時把它們計入。
分數相同會使執行結果無法重現。 兩項分數相同的商品,可能以任一種順序回傳,而且當分數相同時,多執行緒索引可能使順序不具確定性(Lin and Yang, 2019)。即使順序可重現,也可能是任意的:平手時的順序,取決於函式庫的 top-k 選取,或穩定排序的輸入恰好留下什麼順序。因此本書的模組以 product_id 打破平手。LexicalSearch 與 lib.fusion 以它來排序相同的分數,而 SemanticSearch 則對依 product_id 順序儲存的向量做穩定排序。在所有版本都已固定的情況下,re-check examples 會把每個範例重新執行,並與其記錄的輸出比對,因此其中任何一種順序有變動都會顯現出來。在你自己的系統中,計分之前,請以固定且不帶資訊的順序打破平手,例如依 product_id。
未經判定的商品會拉低排序器的分數。 若排序器呈現了沒有人判定過的商品,即使它們很好,nDCG 也會因此下降。測試前請先檢查 judged@10,當它下降時,請讓新商品接受判定。
實用原則: 記錄你在測試查詢上執行的每一次比較;把臨界或探索性的結果視為需要用新查詢檢驗的問題;以固定順序打破分數平手;並在每個 nDCG@10 旁一併回報 judged@10。
本章做出的決定
下列每項選擇,從此處起都是本書的預設值。後續章節若要更動其中一項,必須說明理由。
- 主要指標:採用線性增益的 nDCG@10(等級 2、1 與 0),折減使用 log2(i + 1),理想排序則取自每一項已判定的商品。同時報告 precision@10 與 MRR。相關指的是完全符合。
- 我們自己的指標是基準。 本書回報
lib.metrics的結果,並以 scikit-learn 的ndcg_score進行核對,每一列都納入所有已判定的商品。 - 每一次比較都呈現各查詢的差異: 平均數與中位數、勝、負與平手的次數、損失最大的查詢,以及變動最大的查詢其 judged@10。
- 認定差異成立的規則: 先列出在測試查詢上計畫進行的比較。當某項差異的排列檢定(以 9,999 次隨機正負號翻轉進行)得出的 p 值低於 0.05 除以計畫比較的數目時,該差異才算成立。並列報告成對 t 檢定,以及對應水準的 BCa 自助法區間,兩者不一致時須加以說明。看過測試結果之後才挑選的比較,一律標示為探索性,絕不稱為已成立。訓練查詢僅用於調校。
- 對於未成立的差距,報告為「未顯示改善」, 絕不說成退步,也不加以解釋。
- 所有重抽樣都使用
rng並固定種子, 經由lib.significance執行。 - 計畫中的比較帶來的啟示: 在第 1 章的比較中,語意檢索領先 BM25 的差異成立,另外兩項則未成立,因此語意檢索仍是待超越的排序器。之後的每項改動,都在測試框架中與其所改動的排序器比較,只有勝過待超越的排序器才會採用。
本章,於本版次中
版次 2026.11.10範例於 2026年11 月 19 通過測試本章 12 個範例 個範例中有 12 個通過
譯自英文書,版次 2026.11.10。