混合檢索
閱讀約需 78 分鐘 分鐘,12 節 中共有 12 個範例。
讀完本章後,你可以把詞彙式排名與語意排名合併為一個排名,做法可以是加總正規化後的分數,也可以使用倒數排名融合(reciprocal rank fusion);你也能在訓練查詢上調整每個排名的權重,並在測試查詢上以成對檢定判斷這個組合是否勝過兩個輸入中較好的那一個。
商店以兩個檢索器搜尋其家具與家居用品目錄,而第 1 章把它們融合成一個排名。這個融合並未勝過單獨使用語意檢索:在測試查詢上,它的 nDCG@10 為 0.812,語意檢索則為 0.830,而第 3 章的檢定對這個差距也無法判定孰優孰劣。因此,商店需要知道,更好的混合方式能否勝過語意檢索,也就是目前最好的排序器,因而是必須超越的對象。
所以本章先探討 BM25(本書以 bm25s 預設值建立、未做詞幹還原的基準)與語意檢索(以開放模型 Snowflake/snowflake-arctic-embed-m-v1.5 產生嵌入)是否會在不同的查詢上失敗。只有在兩者失敗之處不同時,混合才可能有幫助,因此本章接著調整兩種混合方式。
業界的搜尋系統經常把兩者搭配使用。有一個語意商品搜尋模型的設計目的,是「在臨時(ad hoc)商品檢索中補足詞彙式搜尋引擎」(Nigam et al., 2019),而一個社群網路的搜尋引擎則同時使用嵌入檢索與關鍵字比對來為文件評分(Huang et al., 2020)。
商店有 50 個用於調整的訓練查詢,以及 30 個用於回報結果的測試查詢。本章找出各個檢索器勝出的查詢,並分別檢查各檢索器單獨的表現。接著建立兩種混合(blend),也就是以一個權重決定詞彙式排名與語意排名各自占多少份量的融合方式,並調整該權重。每一項分析與每一個調整步驟,都在訓練查詢上進行。本章規劃在測試查詢上做兩項比較,且僅在最後做一次:將各個調整後的混合,即加權總和與加權倒數排名融合,分別與語意檢索比較。
此部分尚未翻譯,因此以英文呈現。
Find the queries each retriever wins
Hybrid retrieval combines the rankings of a lexical and a semantic retriever into one ranking. Combining rankings is called fusion. Fusion can only help if the two retrievers fail on different queries: if both put poor products first for a query, no mix of their rankings fixes it. Published studies find that they do. Embeddings miss rare entities and selective keywords that BM25 matches (chapter 6), and on a dataset built to test the limits of one embedding per text, BM25 finds what embeddings miss but loses most of its score when words are replaced with synonyms (chapter 1). So the first check is per query, not on average.
Score both of the store's retrievers by nDCG@10 on every train query, keeping each one's top 100.
1"""Scores BM25 and semantic retrieval on each train query and counts the queries each one wins, ties included.2The prose's point rests on the win counts and on the mean of the better score per query."""3import numpy as np45from lib.lexical import LexicalSearch6from lib.metrics import evaluate7from lib.semantic import SemanticSearch8from lib.store import load_products, load_queries, qrels910products, queries, judgments = load_products(), load_queries("train"), qrels("train")11bm25 = LexicalSearch(products).run(queries, k=100)12semantic = SemanticSearch().run(queries, k=100)13bm25_scores, semantic_scores = evaluate(bm25, judgments)["ndcg@10"], evaluate(semantic, judgments)["ndcg@10"]1415print("train queries:", len(queries))16print(f"bm25 ndcg@10: {bm25_scores.mean():.3f}")17print(f"semantic ndcg@10: {semantic_scores.mean():.3f}")18# rounded to 3 places, so differences that are float noise count as ties19lead = (bm25_scores - semantic_scores).round(3) # per train query; above 0 means BM25 wins20print("bm25 better:", (lead > 0).sum())21print("semantic better:", (lead < 0).sum())22print("tied:", (lead == 0).sum())23print(f"better of the two, chosen per query: {np.maximum(bm25_scores, semantic_scores).mean():.3f}")24has_digit = queries["query"].str.contains(r"\d")25print("queries with a digit:", has_digit.sum())26print("queries with a digit, bm25 better:", (lead[has_digit] > 0).sum())27print()28print("queries where bm25 leads by 0.1 or more:", (lead >= 0.1).sum())29print(queries.loc[lead[lead >= 0.1].index, "query"].to_string())
python bm25-semantic-wins-by-query.pytrain queries: 50
bm25 ndcg@10: 0.695
semantic ndcg@10: 0.772
bm25 better: 15
semantic better: 31
tied: 4
better of the two, chosen per query: 0.802
queries with a digit: 5
queries with a digit, bm25 better: 1
queries where bm25 leads by 0.1 or more: 6
query_id
44 chrome shower head and valve
146 laundry vanity cabinet
277 barstool patio sets
313 malachi sled
359 french molding
432 butcher block island
Neither retriever wins every query. Semantic retrieval averages 0.772 nDCG@10 against BM25's 0.695, but BM25 still wins 15 train queries to semantic retrieval's 31, and 4 tie.
If you could choose the better of the two for each query, the mean would be 0.802. That isn't a ceiling for fusion, which mixes the lists rather than choosing one, but it shows what the differences could be worth.
Next, see whether the two lists hold different products, and what share of each query's exact products (those the judges graded as fully matching) each list finds:
1"""Measures how much BM25's and semantic retrieval's top 10s overlap, and the share of each query's exact products2that each list, and their union, holds at depths 100 and 20. The prose's point rests on the union's shares at both depths."""3import numpy as np45from lib.lexical import LexicalSearch6from lib.metrics import recall_at_k7from lib.semantic import SemanticSearch8from lib.store import load_products, load_queries, qrels910products, queries, judgments = load_products(), load_queries("train"), qrels("train")11bm25 = LexicalSearch(products).run(queries, k=100)12semantic = SemanticSearch().run(queries, k=100)1314shared = [len(set(bm25[query_id][:10]) & set(semantic[query_id][:10])) for query_id in queries.index]15print(f"products in both top 10s, mean: {np.mean(shared):.1f}")161718def share_retrieved(run):19 """The share of each query's exact products that the run's list holds, averaged over queries."""20 # k is the whole list, so this is the share of exact products (grade 2) anywhere in it21 shares = [recall_at_k(ranking, judgments[query_id], k=len(ranking)) for query_id, ranking in run.items()]22 return np.mean(shares)232425for depth in [100, 20]:26 lists = {"bm25": {query_id: ranking[:depth] for query_id, ranking in bm25.items()},27 "semantic": {query_id: ranking[:depth] for query_id, ranking in semantic.items()}}28 # the union in first-seen order, without repeats29 lists["union"] = {query_id: list(dict.fromkeys(lists["bm25"][query_id] + lists["semantic"][query_id]))30 for query_id in queries.index}31 for name, run in lists.items():32 print(f"{name} top {depth}, share of exact products retrieved: {share_retrieved(run):.3f}")
python overlap-and-union-recall.pyproducts in both top 10s, mean: 3.7
bm25 top 100, share of exact products retrieved: 0.672
semantic top 100, share of exact products retrieved: 0.759
union top 100, share of exact products retrieved: 0.810
bm25 top 20, share of exact products retrieved: 0.297
semantic top 20, share of exact products retrieved: 0.353
union top 20, share of exact products retrieved: 0.445
The two top 10s share 3.7 products on average, so most of each is missing from the other. Fusion can't rank a product neither list holds, so each list's depth caps what fusion can reach. The union beats either list at both depths:
| Share of exact products retrieved | Top 20 | Top 100 |
|---|---|---|
| BM25 | 0.297 | 0.672 |
| Semantic retrieval | 0.353 | 0.759 |
| Union of both | 0.445 | 0.810 |
Two train queries show the kinds of failure:
1"""Prints the top 5 products of BM25 and semantic retrieval for two train queries, with their judgments, and counts2the exact products in each top 10. The prose's point rests on those exact counts and on the two spellings of 7qt."""3from lib.lexical import LexicalSearch4from lib.semantic import SemanticSearch5from lib.store import load_products, load_queries, product_text, qrels67LABELS = {2: "exact", 1: "partial", 0: "irrelevant"}8products, queries, judgments = load_products(), load_queries("train"), qrels("train")9bm25 = LexicalSearch(products).run(queries, k=10)10semantic = SemanticSearch().run(queries, k=10)11text = product_text(products)12# the size written as one token, `7qt`13print("products containing 7qt:", text.str.contains(r"\b7qt\b", case=False).sum())14# the size written apart, `7 qt`15print("products containing 7 qt:", text.str.contains(r"\b7 qt\b", case=False).sum())1617for query_id in [432, 62]: # butcher block island, 7qt slow cooker18 query = queries.at[query_id, "query"]19 for name, run in [("bm25", bm25), ("semantic", semantic)]:20 # grade 2 is exact, 1 partial, 0 irrelevant; None for a product nobody judged21 grades = [judgments[query_id].get(product_id) for product_id in run[query_id]]22 print()23 print(f"exact in {name} top 10 for {query}: {grades.count(2)}")24 for product_id, grade in list(zip(run[query_id], grades))[:5]:25 print(f" {LABELS.get(grade, 'unjudged'):<10} {products.at[product_id, 'name'][:55]}")
python two-queries-two-winners.pyproducts containing 7qt: 0
products containing 7 qt: 9
exact in bm25 top 10 for butcher block island: 9
exact carol kitchen island with butcher block top
exact brunton kitchen island with butcher block top
partial brunton kitchen cart butcher block top
exact janigian kitchen island with butcher block
exact avelar kitchen island with butcher block
exact in semantic top 10 for butcher block island: 3
partial boosblock® prep table with butcher block top
partial boosblock® butcher block
partial prep table
partial pro chef kitchen cart with butcher block top
exact brunton kitchen island with butcher block top
exact in bm25 top 10 for 7qt slow cooker: 1
partial cuisinart 6.5 quart programmable slow cooker
partial midea 6.5 qt . slow cooker
partial delonghi cks1660d livenza programmable 6 qt . slow cook
partial nesco 1.5 qt . analog slow cooker
partial 360 cookware gourmet slow cooker
exact in semantic top 10 for 7qt slow cooker: 7
exact 7-qt . stay or go slow cooker
exact crock-pot 7 qt . slow cooker
exact zing 7 qt . digital slow cooker
exact black + decker 7 qt . digital slow cooker
exact crock-pot 7 qt . slow cooker
For butcher block island, BM25's top 10 holds 9 exact products to semantic retrieval's 3. BM25 rewards products whose text contains all three words: kitchen islands with butcher-block tops. Semantic retrieval ranks prep tables and carts with butcher-block tops near the top too, and the judges graded those partial: the right material, but not the island the query asks for.
For 7qt slow cooker it's the other way round: BM25's top 10 holds 1 exact and semantic retrieval's 7. The catalog has 0 products whose text contains 7qt and 9 that contain 7 qt, so BM25 matches only slow cooker and ranks cookers of every size, while semantic retrieval puts 7-quart cookers first.
Those are single queries: they show a kind of failure, not its frequency. The per-query count lists the 6 train queries where BM25 leads by 0.1 nDCG@10 or more. Most name a specific kind of product in two or more words that all have to match, and one, malachi sled, names a product line.
Chapter 1 found BM25 ahead on 2 of the 2 test queries that give a size in inches, yet of the 5 train queries that contain a digit, BM25 wins 1.
Working rule: Before you fuse, count per query where each retriever wins on your train queries, and check how many relevant products the union of their lists holds that the better one misses. Consider fusing when each wins a real share of queries and the union holds more relevant products than either list. That is necessary, not sufficient: next, check each retriever alone, and test the tuned blend on held-out queries before you adopt it.
融合之前,先檢查每個檢索器
知道各個檢索器在哪些查詢上表現較佳還不夠:融合會讓每個輸入都對最終排序有發言權,弱的輸入也不例外。一項涵蓋 11 個資料集的混合搜尋研究發現了「最弱環節」效應:一個弱檢索器就可能大幅拉低融合結果的準確度,因此融合之前應先逐一檢查(Wang et al., PVLDB 2026)。所以要比較各檢索器的平均 nDCG@10,並在旁邊對照 judged@10。接著逐一查詢比較,使用成對檢定:如果兩者的品質相同,出現這麼大的平均差異的可能性有多高?本書採用的檢定是排列檢定,並以成對 t 檢定作為對照。
如同第 1 章,以倒數排名融合(reciprocal rank fusion,RRF)來融合 BM25 與語意檢索:每項商品在各個排序中的得分為 1 / (60 + 其排名),再把兩個排序的得分加總,因此每個排序的票數相等。接著依勝出者劃分訓練查詢,並比較融合與單獨使用語意檢索的結果:
1"""Compares BM25, semantic retrieval and their RRF fusion on the train queries, then splits the queries by which2retriever wins. The prose's point rests on the hybrid-minus-semantic lines, which give the net change, and on the mean3change within each winning group."""4from lib.fusion import reciprocal_rank_fusion5from lib.lexical import LexicalSearch6from lib.metrics import evaluate7from lib.semantic import SemanticSearch8from lib.significance import compare9from lib.store import load_products, load_queries, qrels1011products, queries, judgments = load_products(), load_queries("train"), qrels("train")12bm25 = LexicalSearch(products).run(queries, k=100)13semantic = SemanticSearch().run(queries, k=100)14hybrid = reciprocal_rank_fusion([bm25, semantic]) # k = 60 and equal votes, as in chapter 115bm25_scores, semantic_scores, hybrid_scores = (evaluate(run, judgments) for run in (bm25, semantic, hybrid))1617print(f"bm25 judged@10: {bm25_scores['judged@10'].mean():.3f}")18print(f"semantic judged@10: {semantic_scores['judged@10'].mean():.3f}")19for name, after, before in [("semantic minus bm25", semantic_scores, bm25_scores),20 ("hybrid minus semantic", hybrid_scores, semantic_scores)]:21 result = compare(after["ndcg@10"], before["ndcg@10"]) # paired by query22 print(f"{name}, mean ndcg@10: {result['mean difference']:.3f}")23 print(f"{name}, permutation p: {result['permutation p']:.3f}")24 print(f"{name}, t-test p: {result['t-test p']:.3f}")25print()2627# rounded to 3 places, so float noise can't make a query a win or a loss28change = (hybrid_scores["ndcg@10"] - semantic_scores["ndcg@10"]).round(3) # fusion against semantic alone29lead = (bm25_scores["ndcg@10"] - semantic_scores["ndcg@10"]).round(3) # above 0 means BM25 wins30for name, group in [("bm25 wins", lead > 0), ("semantic wins", lead < 0)]:31 print(f"{name}, queries: {group.sum()}")32 print(f"{name}, fusion better than semantic: {(change[group] > 0).sum()}")33 print(f"{name}, fusion worse than semantic: {(change[group] < 0).sum()}")34 print(f"{name}, mean change in ndcg@10: {change[group].mean():.3f}")
python fusion-gain-by-winning-retriever.pybm25 judged@10: 0.930
semantic judged@10: 0.964
semantic minus bm25, mean ndcg@10: 0.077
semantic minus bm25, permutation p: 0.003
semantic minus bm25, t-test p: 0.004
hybrid minus semantic, mean ndcg@10: 0.002
hybrid minus semantic, permutation p: 0.863
hybrid minus semantic, t-test p: 0.872
bm25 wins, queries: 15
bm25 wins, fusion better than semantic: 12
bm25 wins, fusion worse than semantic: 2
bm25 wins, mean change in ndcg@10: 0.101
semantic wins, queries: 31
semantic wins, fusion better than semantic: 5
semantic wins, fusion worse than semantic: 25
semantic wins, mean change in ndcg@10: -0.045
BM25 的 judged@10 為 0.930,語意檢索則為 0.964,因此 BM25 的前 10 名裡有稍多無人評判的商品,而第 2 章說明了這些商品如何拉低分數。語意檢索的 nDCG@10 領先 0.077,排列檢定的 p 值為 0.003,t 檢定的 p 值為 0.004。在測試查詢上,第 3 章確認了同樣的領先。因此這家商店有一個強的檢索器和一個較弱的檢索器。
依勝出者劃分後,在 BM25 勝出的查詢上,融合平均而言有幫助;在語意檢索勝出的查詢上,融合則有損害:
| 查詢類型 | 查詢數 | 融合優於語意檢索 | 融合劣於語意檢索 | nDCG@10 平均變化 |
|---|---|---|---|---|
| BM25 勝出 | 15 | 12 | 2 | 0.101 |
| 語意檢索勝出 | 31 | 5 | 25 | -0.045 |
依勝出者將查詢分組,本來就會產生這種模式,因為幾乎任何混合都會在 BM25 勝出時有幫助,在它落敗時造成損害。重要的是所有訓練查詢上的淨變化。融合與語意檢索平均相差 0.002 nDCG@10,排列檢定的 p 值為 0.863。在這些查詢上,等票數的融合沒有帶來可測量的增益。
這種劃分利用評判來決定哪個檢索器勝出,所以它只能用來診斷這種模式,無法為即時查詢挑選檢索器,因為即時查詢沒有評判。
實用原則: 融合之前,先在訓練查詢上單獨檢查每個檢索器,使用 judged@10 與成對檢定。當其中一個較弱、勝出的查詢也較少時,不要給它同等的票數:可以為它加權並調整權重,或者不採用它。等票數適合品質大致相同的兩個檢索器。
先正規化分數,再相加
要讓較弱的檢索器(在這家商店中是 BM25)發言權小一點,最直接的做法是將兩個檢索器的分數加權相加:α 乘以語意分數,加上 (1 − α) 乘以 BM25 分數。權重總和為 1 的加總稱為凸組合(convex combination),也就是加權和。這種做法只有在分數的尺度可以比較時才有效,而這裡並非如此:BM25 分數沒有固定範圍,會隨查詢中字詞的數量與稀有程度而增加;語意檢索的分數是餘弦相似度,介於 −1 與 1 之間。分數正規化(score normalization)會在相加之前,針對每個查詢將各檢索器的分數重新縮放。逐查詢的 min-max 縮放會把該查詢已排名商品中的最低分對應為 0,最高分對應為 1。一項針對融合函數的研究稱它為「文獻中事實上的標準方法」,並測試了兩種替代方案(Bruch et al., 2023)。理論 min-max(theoretical min-max)改用各分數可能的最低值,而不是實際觀察到的最低值:BM25 為 0,餘弦為 −1。z 分數(z-score)則減去查詢分數的平均值,再除以其標準差。這些範例共用的模組 lib/blend.py 以幾行程式碼寫出了上述每一種方法。
min-max(s) = (s − lowest score among the query's candidates) / (highest − lowest)
blend = α · min-max(similarity) + (1 − α) · min-max(BM25)
加總需要每個被排名商品的兩種分數。查詢的候選集(candidate set)是 BM25 前 100 名與語意檢索前 100 名的聯集,lib.blend.candidates 會為其中每個商品查出兩種分數:BM25 分數(商品與查詢沒有共同字詞時為 0)以及餘弦相似度。
先從商店的原始分數開始,在訓練查詢上以 α = 0.5 相加,再在 min-max 縮放後做同樣的事,範例中以手算寫出,並以 lib.blend.weighted_sum 核對:
1"""Adds BM25 and cosine scores with equal weights, raw and after min-max scaling, and checks the scaled sum against2lib.blend.weighted_sum. The prose's point rests on the two sums' top 10 overlaps and nDCG@10, and on the count of3train queries where the by-hand ranking matches the library's."""4from lib.blend import candidates, weighted_sum5from lib.lexical import LexicalSearch6from lib.metrics import evaluate7from lib.semantic import SemanticSearch8from lib.significance import compare9from lib.store import load_products, load_queries, qrels1011products, queries, judgments = load_products(), load_queries("train"), qrels("train")12lexical, semantic = LexicalSearch(products), SemanticSearch()13tables = candidates(lexical, semantic, queries) # both scores for each query's candidates1415# the candidate tables are keyed by query_id16table = tables[36] # laundry basket with wheels17print("candidates for laundry basket with wheels:", len(table))18print(f"bm25 range: {table['bm25'].min():.3f} to {table['bm25'].max():.3f}")19print(f"similarity range: {table['similarity'].min():.3f} to {table['similarity'].max():.3f}")202122def min_max(scores):23 return (scores - scores.min()) / (scores.max() - scores.min())242526def blend(normalize, alpha=0.5):27 run = {}28 for query_id, table in tables.items():29 score = alpha * normalize(table["similarity"]) + (1 - alpha) * normalize(table["bm25"])30 # the top 100; the stable sort keeps the candidates' order on equal scores31 run[query_id] = list(score.sort_values(ascending=False, kind="stable").index[:100])32 return run333435top_10s = {"bm25": lexical.run(queries, k=10), "semantic": semantic.run(queries, k=10)}36per_query = {}37for name, normalize in [("raw", lambda scores: scores), ("min-max", min_max)]:38 run = blend(normalize)39 per_query[name] = evaluate(run, judgments)["ndcg@10"]40 print(f"{name} sum ndcg@10: {per_query[name].mean():.3f}")41 for other, top in top_10s.items():42 shared = [len(set(run[query_id][:10]) & set(top[query_id])) for query_id in queries.index]43 print(f"{name} sum, products shared with the {other} top 10: {sum(shared) / len(shared):.1f}")44# the library sorts equal scores by product_id; this blend keeps the candidates' order45by_hand, library = blend(min_max), weighted_sum(tables, 0.5)46matches = sum(by_hand[query_id] == library[query_id] for query_id in tables)47print(f"train queries ranked as lib.blend.weighted_sum ranks them: {matches} of {len(tables)}")48p_value = compare(per_query["min-max"], per_query["raw"])["permutation p"] # a tuning check on train49print("min-max minus raw, permutation p:", "below 0.001" if p_value < 0.001 else f"{p_value:.3f}")
python raw-scores-versus-min-max.pycandidates for laundry basket with wheels: 125
bm25 range: 2.059 to 8.229
similarity range: 0.336 to 0.597
raw sum ndcg@10: 0.706
raw sum, products shared with the bm25 top 10: 9.6
raw sum, products shared with the semantic top 10: 3.9
min-max sum ndcg@10: 0.771
min-max sum, products shared with the bm25 top 10: 6.5
min-max sum, products shared with the semantic top 10: 6.2
train queries ranked as lib.blend.weighted_sum ranks them: 50 of 50
min-max minus raw, permutation p: below 0.001
對於 laundry basket with wheels,125 個候選商品的 BM25 分數範圍為 2.059 to 8.229,相似度範圍為 0.336 to 0.597。BM25 的分布範圍大上許多倍,因此原始分數總和的排序由它決定。在所有訓練查詢上,原始分數總和的前 10 名平均與 BM25 的前 10 名共有 9.6 個商品,與語意檢索的前 10 名則共有 3.9 個。
權重上說是「各占一半」,但排名多半取決於 BM25:原始分數總和的 nDCG@10 為 0.706,與 BM25 的 0.695 相近。
經過 min-max 縮放後,兩個檢索器都有發言權。縮放後總和的前 10 名與 BM25 共有 6.5 個商品,與語意檢索共有 6.2 個,nDCG@10 升至 0.771。這項與原始分數總和之差異的排列檢定 p 值為 below 0.001。手算版本對 50 of 50 個訓練查詢的排名,與 lib.blend.weighted_sum 完全相同。
原始分數在權重經過調校後也可以運作。以嵌入為基礎的檢索器 DPR 的作者,以 BM25 加上 1.1 倍的 DPR 相似度來排序候選項目,其中的 1.1 是在開發集上選定的(Karpukhin et al., 2020)。正規化讓 α 對每個查詢都有相同的意義;若不正規化,BM25 所占的發言比重會隨其範圍在不同查詢間變動。不過,min-max 並不是絕對尺度:它是相對於該查詢的候選項目而言,因此無論最佳候選項目有多差,它都會得到 1。
實用原則: 切勿以猜測的權重將原始的 BM25 與餘弦分數直接相加。我們建議以逐查詢 min-max 作為預設做法,因為它不需要事先知道任一評分函數的特性。若你知道各分數可能的最低值,並且希望有一個不隨候選項目移動的下限,請使用理論 min-max。無論採用哪一種,都要在訓練查詢上調校 α。
以訓練查詢測試倒數排名融合的設定
與分數不同,排名共用同一個尺度,因此以排名進行融合不需要正規化。倒數排名融合(reciprocal rank fusion,RRF)為每項商品計分的方式,是在包含該商品的各個排名上,加總 1 除以常數 k 與其名次之和的值(Cormack et al., 2009),如第 1 章的公式所寫。代價是資訊的損失:「由於 RRF 是排名的函數,它忽略了分數的分布,因此捨棄了有用的資訊」(Bruch et al., 2023)。一項僅略居領先者之後的商品,與一項遠遠落後的商品,其分數只由名次決定。
常數 k 決定前段名次的主導程度。第 1 章採用 RRF 作者提出的 k = 60;他們在前導研究中發現 60 接近最佳值,且確切的選擇影響不大。他們的前導研究融合了 Wumpus Search(單一搜尋引擎)的 30 種設定,使用的是來自 TREC(一系列共同的檢索評測)的文件集(Cormack et al., 2009)。Bruch et al. 融合了一個詞彙檢索器與一個語意檢索器,為各自設定獨立的常數 η,並發現結果對兩個常數之間的平衡很敏感,這個平衡的作用相當於權重:「在語意模型本身已表現強勁的情況下(範例之一是提高 η_Lex 以降低詞彙系統的比重),會帶來更好的效能」。商店中兩個排名共用同一個 k,並未設定兩者之間的平衡。
每份清單的深度同樣重要。某項商品若在一個檢索器的前 20 名內,卻不在另一個檢索器的前 20 名內,即使它在後者的名次很低,也無法從第二個排名得到任何分數。如 Bruch et al. 所言:「除非針對每個查詢,由檢索系統 o 對整個文件集 D 排名,否則無法精確計算名次。」在商店的訓練查詢上,兩份前 20 名清單的聯集所含的完全相符商品,遠少於兩份前 100 名清單的聯集,如各清單中完全相符商品的占比所示。
在訓練查詢上試驗六個 k 值,融合的清單深度為 100。執行前先規劃一項檢驗:最佳的 k 是否優於預設值 k = 60?
1"""Fuses BM25 and semantic retrieval with six values of k on the train queries, and tests the best k against the2default k = 60. The prose's point rests on the rank ratios at k = 1 and k = 60, on the best k, and on its permutation3p-value."""4from lib.fusion import reciprocal_rank_fusion5from lib.lexical import LexicalSearch6from lib.metrics import evaluate7from lib.semantic import SemanticSearch8from lib.significance import compare9from lib.store import load_products, load_queries, qrels1011products, queries, judgments = load_products(), load_queries("train"), qrels("train")12bm25 = LexicalSearch(products).run(queries, k=100)13semantic = SemanticSearch().run(queries, k=100)1415# RRF gives rank r the score 1 / (k + r), so rank 1 over rank 10 is (k + 10) / (k + 1)16for k in [1, 60]:17 print(f"rank 1 over rank 10 at k={k}: {(1 / (k + 1)) / (1 / (k + 10)):.3f}")18scores = {k: evaluate(reciprocal_rank_fusion([bm25, semantic], k=k), judgments)["ndcg@10"]19 for k in [1, 10, 30, 60, 100, 300]}20for k, per_query in scores.items():21 print(f"k={k} ndcg@10: {per_query.mean():.3f}")2223best = max(scores, key=lambda k: scores[k].mean()) # ties go to the smaller k24print("best k on train:", best)25result = compare(scores[best], scores[60]) # the planned check: best k against the default26print(f"best k minus k=60: {result['mean difference']:.3f}")27print(f"best k minus k=60, permutation p: {result['permutation p']:.3f}")28print(f"best k minus k=60, t-test p: {result['t-test p']:.3f}")29print(f"best k minus k=60, 95% interval: {result['ci low']:.3f} to {result['ci high']:.3f}")30print(f"best k minus k=60, wins, losses, ties: {result['wins']}, {result['losses']}, {result['ties']}")
python rrf-k-on-train.pyrank 1 over rank 10 at k=1: 5.500
rank 1 over rank 10 at k=60: 1.148
k=1 ndcg@10: 0.753
k=10 ndcg@10: 0.770
k=30 ndcg@10: 0.771
k=60 ndcg@10: 0.774
k=100 ndcg@10: 0.778
k=300 ndcg@10: 0.777
best k on train: 100
best k minus k=60: 0.004
best k minus k=60, permutation p: 0.093
best k minus k=60, t-test p: 0.103
best k minus k=60, 95% interval: 0.001 to 0.011
best k minus k=60, wins, losses, ties: 6, 3, 41
在 k = 1 時,第 1 名的得分是第 10 名的 5.500 倍。在 k = 60 時,則是 1.148 倍,因此在兩份清單中都接近頂端的商品,會勝過只在一份清單中位居頂端的商品。從 k = 1 到 k = 300,六種設定的得分依序為 0.753、0.770、0.771、0.774、0.778 與 0.777。
訓練集上最佳的設定是 k = 100,領先 k = 60 0.004,排列檢定的 p 值為 0.093,t 檢定的 p 值為 0.103。本書的判定依據排列檢定,該檢定在 0.05 的水準下無法將此增益與機遇區分開來。此增益的 95% 拔靴法信賴區間 0.001 to 0.011 並未包含 0,因此該區間與兩項檢定的結論不一致。
兩種設定的排名只在少數查詢上有差異(勝、負與平手的數目:6, 3, 41)。哪個 k 最佳也相當脆弱:若將 BM25 並列的商品以相反順序列出,勝出的會變成 k = 300。這與 Cormack et al. 的發現相符,即 k 的選擇影響不大。
實用原則: 預設保持 k = 60,並融合至少 100 名深的清單,因為較深的清單只會增加候選商品。只有在訓練集上的最佳 k 以排列檢定能與機遇區分的幅度勝過 k = 60,也就是 p 值低於 0.05 時,才改用其他的 k。當其中一個檢索器較強時,如商店的情形,要調整的設定不是 k,而是 RRF 的平等投票。應為各個排名賦予權重,並加以調校。
以訓練查詢調校混合權重
本章的兩種混合方式各有一個權重 α,代表分配給語意檢索的比例。在加權總和中,α 乘上正規化後的相似度,1 − α 則乘上正規化後的 BM25 分數。在加權 RRF 中,商品的分數為 (1 − α) / (k + 其 BM25 排名) 加上 α / (k + 其語意排名)。在兩種方式中,α = 0 表示只用 BM25,α = 1 表示只用語意檢索。對 RRF 而言,α = 0.5 就是第 1 章的未加權融合,只是每個分數都減半。以下以一個訓練查詢的第一名商品為例,列出算式,其中 α 取 0.75 作為說明:
1"""Scores every candidate for one train query by weighted RRF, by hand, and checks the ranking against2lib.blend.weighted_rrf. The prose's point rests on the top product's two ranks and its score, and on the count of3places that match the library."""4from lib.blend import weighted_rrf5from lib.lexical import LexicalSearch6from lib.semantic import SemanticSearch7from lib.store import load_products, load_queries89alpha, k = 0.75, 60 # an illustrative weight; tuning picks one on train10products, queries = load_products(), load_queries("train")11bm25 = LexicalSearch(products).run(queries, k=100)12semantic = SemanticSearch().run(queries, k=100)13query_id = 36 # laundry basket with wheels141516def part(ranking, weight, product_id):17 """weight / (k + rank) from one ranking, or 0 when the product isn't in it."""18 # index counts from 0, so one is added to get the rank19 return weight / (k + ranking.index(product_id) + 1) if product_id in ranking else 0.0202122candidates = dict.fromkeys(bm25[query_id] + semantic[query_id]) # the union, in the order first seen23score = {product_id: part(bm25[query_id], 1 - alpha, product_id) + part(semantic[query_id], alpha, product_id)24 for product_id in candidates}25ranking = sorted(score, key=lambda product_id: (-score[product_id], product_id)) # ties by product_id26first = ranking[0]27print("first product:", products.at[first, "name"])28print("its bm25 rank:", bm25[query_id].index(first) + 1)29print("its semantic rank:", semantic[query_id].index(first) + 1)30print(f"its weighted rrf score: {score[first]:.5f}")31library = weighted_rrf(bm25, semantic, alpha)[query_id] # the top 10032matches = sum(mine == theirs for mine, theirs in zip(ranking, library))33print(f"places that match lib.blend.weighted_rrf: {matches} of {len(library)}")
python weighted-rrf-one-product.pyfirst product: laundry sorter basket on wheels
its bm25 rank: 1
its semantic rank: 2
its weighted rrf score: 0.01620
places that match lib.blend.weighted_rrf: 100 of 100
對於 laundry basket with wheels,第一名商品 laundry sorter basket on wheels 在 BM25 排名中位於第 1 名,在語意檢索排名中位於第 2 名,因此分數為 0.25 / (60 + 1) + 0.75 / (60 + 2) = 0.01620。
此範例以同樣方式替每個候選商品評分;其排名與 lib.blend.weighted_rrf 在 100 of 100 處相符。
調校的做法是在訓練查詢上嘗試一組 α 值,並保留平均 nDCG@10 最佳的那一個。Bruch 等人(2023)針對凸組合(CC)的調校提出三項發現:無論採用哪種分數正規化,調校後的 CC 表現都差不多;它在領域內與領域外都勝過 RRF;而且具有樣本效率。所謂樣本效率,是指 α 在「使用不到 5% 的訓練資料」時就已收斂,而其訓練集遠大於商店的資料量。
對加權總和,在三種正規化下分別將 α 在 0、0.1、… 1 之間調校;對 k = 60 的加權 RRF 也同樣調校。接著作為調校的檢查,將每種正規化與 min-max 比較,並將為測試查詢預定採用的兩種混合方式與語意檢索比較:
1"""Tunes α on the train queries for the weighted sum under three normalizations and for weighted RRF, then compares2the two blends that go to test with semantic retrieval. The prose's point rests on each setting's best α and nDCG@10,3on the p-values against tuned min-max, and on the two gains over semantic retrieval."""4from lib.blend import ALPHAS, NORMALIZATIONS, best_alpha, candidates, scores_by_alpha, weighted_rrf, weighted_sum5from lib.lexical import LexicalSearch6from lib.semantic import SemanticSearch7from lib.significance import compare8from lib.store import load_products, load_queries, qrels910products, queries, judgments = load_products(), load_queries("train"), qrels("train")11lexical, semantic = LexicalSearch(products), SemanticSearch()12tables = candidates(lexical, semantic, queries)13bm25, semantic_run = lexical.run(queries, k=100), semantic.run(queries, k=100)1415per_query = {} # nDCG@10 per query (rows) and alpha (columns)16for name in NORMALIZATIONS:17 per_query[name] = scores_by_alpha(lambda alpha: weighted_sum(tables, alpha, name), judgments)18per_query["weighted rrf"] = scores_by_alpha(lambda alpha: weighted_rrf(bm25, semantic_run, alpha), judgments)19semantic_alone = per_query["min-max"][1.0] # alpha = 1 is semantic retrieval in either blend20print("alphas tried:", len(ALPHAS))21print("min-max ndcg@10 for alpha 0 to 1:", " ".join(f"{mean:.3f}" for mean in per_query["min-max"].mean()))22print(f"semantic alone ndcg@10: {semantic_alone.mean():.3f}")23# alpha 0.5 is unweighted RRF, chapter 1's fusion24print(f"unweighted rrf (alpha 0.5) ndcg@10: {per_query['weighted rrf'][0.5].mean():.3f}")2526best = {name: frame[best_alpha(frame)] for name, frame in per_query.items()}27for name, frame in per_query.items():28 print(f"{name} best alpha: {best_alpha(frame)}")29 print(f"{name} best ndcg@10: {best[name].mean():.3f}")30for name in ["theoretical min-max", "z-score", "weighted rrf"]: # each tuned, against tuned min-max31 print(f"{name} against min-max, permutation p: {compare(best[name], best['min-max'])['permutation p']:.3f}")32for name in ["min-max", "weighted rrf"]: # the two blends that go to test33 gain = compare(best[name], semantic_alone)34 print(f"{name} gain over semantic: {gain['mean difference']:.3f}")35 print(f"{name} gain over semantic, permutation p: {gain['permutation p']:.3f}")36 print(f"{name} gain over semantic, t-test p: {gain['t-test p']:.3f}")
python tune-blend-weights-on-train.pyalphas tried: 11
min-max ndcg@10 for alpha 0 to 1: 0.695 0.709 0.736 0.746 0.758 0.771 0.781 0.784 0.777 0.778 0.772
semantic alone ndcg@10: 0.772
unweighted rrf (alpha 0.5) ndcg@10: 0.774
min-max best alpha: 0.7
min-max best ndcg@10: 0.784
theoretical min-max best alpha: 0.9
theoretical min-max best ndcg@10: 0.789
z-score best alpha: 0.7
z-score best ndcg@10: 0.787
weighted rrf best alpha: 0.6
weighted rrf best ndcg@10: 0.787
theoretical min-max against min-max, permutation p: 0.164
z-score against min-max, permutation p: 0.349
weighted rrf against min-max, permutation p: 0.776
min-max gain over semantic: 0.013
min-max gain over semantic, permutation p: 0.177
min-max gain over semantic, t-test p: 0.161
weighted rrf gain over semantic: 0.015
weighted rrf gain over semantic, permutation p: 0.215
weighted rrf gain over semantic, t-test p: 0.205
在相同查詢上,單用語意檢索的分數為 0.772。各設定的最佳 α 與分數如下:
| 設定 | 最佳 α | 平均 nDCG@10 | 與調校後 min-max 比較的排列檢定 p 值 |
|---|---|---|---|
| 加權總和,min-max | 0.7 | 0.784 | — |
| 加權總和,理論 min-max | 0.9 | 0.789 | 0.164 |
| 加權總和,z-score | 0.7 | 0.787 | 0.349 |
| 加權 RRF,k = 60 | 0.6 | 0.787 | 0.776 |
在 0.05 的水準下,訓練查詢上的檢定無法分辨另兩種正規化與 min-max 的差異,這與 Bruch 等人所述正規化影響甚微的發現相符。檢定同樣無法分辨加權 RRF 與調校後的 min-max。Bruch 等人的第二項發現,即加權總和勝過 RRF,在此並未顯現,而這些查詢也無法對此下定論。
由於 α = 1 在網格中,每個調校後的混合方式在訓練查詢上的分數至少與語意檢索相當。然而,即使在選出它們的那些查詢上,排列檢定在 0.05 的水準下也無法將任一混合方式的增益與隨機波動區分開來。加權總和的增益為 0.013,排列檢定 p 值為 0.177;加權 RRF 的增益為 0.015,排列檢定 p 值為 0.215。每項增益也都是在同一批查詢上 11 次嘗試中的最佳值,這使增益看起來更好。
實用原則: 在訓練查詢上以粗略的網格調校一個權重,並讓正規化維持為 min-max,除非另一種正規化勝出的幅度,大到訓練查詢上的檢定能將它與隨機波動區分開來,亦即 p 值低於 0.05。請將相對於最佳單一檢索器的訓練增益視為樂觀的估計,並預期它在測試查詢上會縮小;對於任何從數個選項中挑出最佳者的設定,這都是一項經驗法則。在信任該設定之前,先檢查它是否穩定,再以測試查詢報告結果。
確認調校後的權重是否穩定
調校後的權重來自幾十個查詢,因此可能取決於你手上有哪些查詢。自助法(bootstrap)可以顯示這種依賴有多大:對訓練查詢重複抽樣,在每個樣本上調校 α,並重複許多次。如果選出的 α 變動很大,表示調校出的值反映的是樣本,而不是商店本身。第 3 章使用同樣的重抽樣方法,為分數建立區間。
以固定的隨機種子對訓練查詢重抽樣 1000 次,並在同樣的重抽樣樣本上,針對兩種調校後的混合方式(min-max 分數的加權總和,以及加權 RRF),計算各個 α 獲勝的次數:
1"""Resamples the train queries with a fixed seed and counts which α wins each resample, for the weighted sum and for2weighted RRF on the same resamples. The prose's point rests on the α tuned on all train queries and on the counts of3resamples choosing each α."""4import numpy as np5import pandas as pd67from lib.blend import best_alpha, candidates, scores_by_alpha, weighted_rrf, weighted_sum8from lib.lexical import LexicalSearch9from lib.semantic import SemanticSearch10from lib.store import load_products, load_queries, qrels1112products, queries, judgments = load_products(), load_queries("train"), qrels("train")13lexical, semantic = LexicalSearch(products), SemanticSearch()14tables = candidates(lexical, semantic, queries)15bm25, semantic_run = lexical.run(queries, k=100), semantic.run(queries, k=100)16blends = {"weighted sum": scores_by_alpha(lambda alpha: weighted_sum(tables, alpha), judgments),17 "weighted rrf": scores_by_alpha(lambda alpha: weighted_rrf(bm25, semantic_run, alpha), judgments)}1819resamples = 100020print("resamples:", resamples)21# a fixed seed, so every run draws the same resamples22rng = np.random.default_rng(2026)23draws = [rng.integers(0, len(queries), len(queries)) for _ in range(resamples)] # the same draws for both24for name, per_query in blends.items(): # per_query: train queries x alphas25 print(f"{name} alpha tuned on all train queries: {best_alpha(per_query)}")26 # best alpha per resample; idxmax takes the first, smaller alpha when two tie27 chosen = pd.Series([per_query.iloc[draw].mean().idxmax() for draw in draws])28 for alpha, count in chosen.value_counts().sort_index().items():29 print(f"{name} resamples choosing alpha {alpha}: {count}")
python blend-weight-bootstrap.pyresamples: 1000
weighted sum alpha tuned on all train queries: 0.7
weighted sum resamples choosing alpha 0.5: 13
weighted sum resamples choosing alpha 0.6: 171
weighted sum resamples choosing alpha 0.7: 686
weighted sum resamples choosing alpha 0.8: 11
weighted sum resamples choosing alpha 0.9: 74
weighted sum resamples choosing alpha 1.0: 45
weighted rrf alpha tuned on all train queries: 0.6
weighted rrf resamples choosing alpha 0.5: 2
weighted rrf resamples choosing alpha 0.6: 513
weighted rrf resamples choosing alpha 0.7: 76
weighted rrf resamples choosing alpha 0.8: 365
weighted rrf resamples choosing alpha 0.9: 26
weighted rrf resamples choosing alpha 1.0: 18
加權總和的權重相當穩定。在所有訓練查詢上,它的 α 是 0.7,而 α = 0.7 在 686 次重抽樣中獲勝,比其他任何選擇都多。α = 0.6 在 171 次重抽樣中獲勝,單獨使用語意檢索,也就是 α = 1,則獲勝 45 次。
加權 RRF 調校後的 α = 0.6,但較不穩定:它的權重在 0.6 到 0.8 之間移動,其中 α = 0.6 獲勝 513 次,α = 0.8 獲勝 365 次,α = 0.7 獲勝 76 次。
除了等權重的 α = 0.5 之外,兩種權重都落在網格中偏向語意檢索的一側;α = 0.5 在加權總和的重抽樣中獲勝 13 次,在加權 RRF 中獲勝 2 次。
重抽樣並不是唯一的檢查。BM25 分數相同的商品需要某種順序,而 lib.lexical 依 product_id 列出它們,這種順序不帶任何資訊。依 product_id 遞減排列同樣不帶任何資訊。加權總和對平手的商品給予相同的 BM25 項,因此它們的順序對加權總和幾乎沒有影響;但 RRF 會給它們連續的名次,因此分數也不同。請依 product_id 遞減列出 BM25 的平手項目,並在訓練查詢上重新調校最佳 RRF k與兩種權重:
1"""Tunes RRF's k and both blends' α again with BM25's equal scores listed in descending product_id order, to test2whether the order of ties matters. The prose's point rests on each setting's pair of lines, ties by product_id and3ties reversed."""4import numpy as np56from lib.blend import best_alpha, candidates, scores_by_alpha, weighted_rrf, weighted_sum7from lib.fusion import reciprocal_rank_fusion8from lib.lexical import LexicalSearch9from lib.metrics import evaluate10from lib.semantic import SemanticSearch11from lib.store import load_products, load_queries, qrels121314class TiesReversed(LexicalSearch):15 """BM25 with equal scores in descending product_id order, an order as arbitrary as ascending."""1617 def search(self, query, k=100):18 scores = super().search(query, k=len(self.ids)) # every match, so the cut at k can move too19 # highest score first, then descending product_id (np.lexsort sorts by its last key first)20 order = np.lexsort((-scores.index.to_numpy(), -scores.to_numpy()))21 return scores.iloc[order[:k]]222324products, queries, judgments = load_products(), load_queries("train"), qrels("train")25semantic = SemanticSearch()26semantic_run = semantic.run(queries, k=100)27# each order runs in turn, against the same semantic run28for ties, lexical in [("ties by product_id", LexicalSearch(products)), ("ties reversed", TiesReversed(products))]:29 bm25 = lexical.run(queries, k=100)30 by_k = {k: evaluate(reciprocal_rank_fusion([bm25, semantic_run], k=k), judgments)["ndcg@10"].mean()31 for k in [1, 10, 30, 60, 100, 300]}32 print(f"best rrf k, {ties}: {max(by_k, key=by_k.get)}") # ties go to the smaller k33 rrf = scores_by_alpha(lambda alpha: weighted_rrf(bm25, semantic_run, alpha), judgments)34 print(f"weighted rrf alpha, {ties}: {best_alpha(rrf)}")35 tables = candidates(lexical, semantic, queries)36 total = scores_by_alpha(lambda alpha: weighted_sum(tables, alpha), judgments)37 print(f"weighted sum alpha, {ties}: {best_alpha(total)}")
python tuning-with-ties-reversed.pybest rrf k, ties by product_id: 100
weighted rrf alpha, ties by product_id: 0.6
weighted sum alpha, ties by product_id: 0.7
best rrf k, ties reversed: 300
weighted rrf alpha, ties reversed: 0.8
weighted sum alpha, ties reversed: 0.7
反轉平手項目的順序,會改變 RRF 的設定,但不會改變加權總和的設定:
| 設定 | 平手項目依 product_id 排序 | 平手項目反轉 |
|---|---|---|
| 最佳 RRF k | 100 | 300 |
| 加權 RRF α | 0.6 | 0.8 |
| 加權總和 α | 0.7 | 0.7 |
當一個不帶任何資訊的順序就能改變某項設定,表示訓練查詢無法區分這些設定,因此 RRF 的 k 與權重並不穩固,而加權總和的權重則是穩固的。商店沒有更多訓練查詢可用來確定加權 RRF 的權重,而它的測試比較原本就是以平手項目依 product_id 排序所調校的權重來規劃,因此送去測試的是 α = 0.6。
穩定性的意思是訓練查詢對權重的看法一致,並不表示它能改善其他查詢的搜尋結果:重抽樣重複使用同一批查詢,因此會共享這些查詢的共通之處。
實用原則: 對訓練查詢進行自助法重抽樣,並在每個樣本上調校,然後再以另一種不帶任何資訊的順序列出平手項目,重新調校一次。如果選出的權重在網格上分散,就改用較粗的網格或取得更多訓練查詢,或是選擇較簡單的設定。如果光是平手項目的順序就會使它改變,則在報告該權重的每個地方都要說明這一點。如果它在兩項檢查下都保持穩定,就將它帶到測試查詢上。
在測試查詢上報告調校後的混合方法
只有保留不用的查詢,才能顯示調校後的設定在挑選它的那批查詢之外是否仍有幫助。將每個調校後的混合方法在測試查詢上各執行一次,並使用第 3 章的評測框架,逐一查詢與語意檢索(最佳的單一檢索器)比較。本章在檢視結果之前,就已固定這兩項比較。依照第 3 章的做法,Bonferroni 校正將門檻設為排列檢定 p 值低於 0.025,也就是 0.05 除以兩項預先規劃的比較。t 檢定與信賴水準相符(0.975)的 bootstrap 區間則一併列出,供對照。
在商店的訓練查詢上,調校後的混合方法相較於語意檢索分別提升了 0.013 與 0.015,但排列檢定無法將這兩項提升與隨機波動區分開來。請先預測:在測試查詢上,α = 0.7 的加權總和與 α = 0.6 的加權 RRF,得分會高於語意檢索、與其持平,還是低於它?
1"""Chooses each blend's α on the train queries, runs both once on the test queries, and compares each with semantic2retrieval at the corrected threshold. The prose's point rests on the two mean differences and their permutation3p-values, and the assert checks that neither clears the threshold."""4from lib.blend import best_alpha, candidates, scores_by_alpha, weighted_rrf, weighted_sum5from lib.lexical import LexicalSearch6from lib.metrics import evaluate7from lib.semantic import SemanticSearch8from lib.significance import compare9from lib.store import load_products, load_queries, qrels1011lexical, semantic = LexicalSearch(load_products()), SemanticSearch()121314def blends(queries):15 tables, bm25, semantic_run = candidates(lexical, semantic, queries), lexical.run(queries), semantic.run(queries)16 return {"weighted sum": lambda alpha: weighted_sum(tables, alpha), # each blend's run at a given alpha17 "weighted rrf": lambda alpha: weighted_rrf(bm25, semantic_run, alpha)}181920train, test = blends(load_queries("train")), blends(load_queries("test"))21threshold = 0.05 / len(test) # two planned comparisons, Bonferroni-corrected22semantic_scores = evaluate(semantic.run(load_queries("test")), qrels("test"))23print("test queries:", len(semantic_scores))24print(f"semantic ndcg@10: {semantic_scores['ndcg@10'].mean():.3f}")25unweighted = evaluate(test["weighted rrf"](0.5), qrels("test")) # alpha 0.5 is chapter 1's fusion26print(f"unweighted rrf ndcg@10: {unweighted['ndcg@10'].mean():.3f}")27print(f"p-value threshold for each comparison: {threshold:.3f}")28print(f"confidence level of each interval: {1 - threshold:.3f}")29judged = [semantic_scores["judged@10"].mean()]30for name, blend in test.items():31 alpha = best_alpha(scores_by_alpha(train[name], qrels("train"))) # train only32 scores = evaluate(blend(alpha), qrels("test"))33 judged.append(scores["judged@10"].mean())34 # the interval's level matches the corrected threshold35 result = compare(scores["ndcg@10"], semantic_scores["ndcg@10"], confidence=1 - threshold)36 print(f"{name} alpha chosen on train: {alpha}")37 print(f"{name} ndcg@10: {scores['ndcg@10'].mean():.3f}")38 print(f"{name} minus semantic: {result['mean difference']:.3f}")39 print(f"{name} permutation p: {result['permutation p']:.3f}")40 print(f"{name} t-test p: {result['t-test p']:.3f}")41 print(f"{name} interval: {result['ci low']:.3f} to {result['ci high']:.3f}")42 # neither difference is established: each permutation p-value stays at or above the corrected threshold43 assert result['permutation p'] >= threshold44print(f"lowest judged@10: {min(judged):.3f}")45print(f"highest judged@10: {max(judged):.3f}")
python tuned-blends-on-test.pytest queries: 30
semantic ndcg@10: 0.830
unweighted rrf ndcg@10: 0.812
p-value threshold for each comparison: 0.025
confidence level of each interval: 0.975
weighted sum alpha chosen on train: 0.7
weighted sum ndcg@10: 0.816
weighted sum minus semantic: -0.014
weighted sum permutation p: 0.229
weighted sum t-test p: 0.221
weighted sum interval: -0.042 to 0.008
weighted rrf alpha chosen on train: 0.6
weighted rrf ndcg@10: 0.825
weighted rrf minus semantic: -0.005
weighted rrf permutation p: 0.650
weighted rrf t-test p: 0.643
weighted rrf interval: -0.029 to 0.016
lowest judged@10: 0.980
highest judged@10: 0.990
兩種混合方法都沒有勝過語意檢索。語意檢索的得分為 0.830,min-max 分數的加權總和為 0.816,加權 RRF 為 0.825。
這些執行結果的 judged@10 介於 0.980 與 0.990 之間,因此缺漏的判定對三者的影響大致相當。
兩項差異都不能視為成立:每個排列檢定 p 值都仍不低於校正後的門檻 0.025。加權總和與語意檢索的差為 -0.014,排列檢定 p 值為 0.229,t 檢定 p 值為 0.221,區間為 -0.042 to 0.008。加權 RRF 的差為 -0.005,p 值分別為 0.650 與 0.643,區間為 -0.029 to 0.016。兩個區間都包含 0,檢定結果也與之一致,所以結論是「沒有改善」,而不是「更差」。對加權 RRF 而言,這是 α = 0.6 時的結論,而這個權重可能會因為BM25 同分項目的排序而改變。
若將結果說成「高於」,就等於相信了訓練集上的提升;但即使在挑選它的那批查詢上,排列檢定也無法將該提升與隨機波動區分開來,而且它還是多次嘗試中表現最好的一個。
實用原則: 只有當混合方法在保留不用的查詢上勝過最佳的單一檢索器,且排列檢定 p 值低於依預先規劃的比較次數校正後的門檻時,才採用它。若未達成,就保留較簡單的系統。這使得單獨的語意檢索仍是該商店要超越的排序器。請保留調校後的權重與程式碼,並在檢索器或商品目錄變動時,再次測試這些混合方法。
此部分尚未翻譯,因此以英文呈現。
Learn sparse term weights with SPLADE
Instead of fusing two retrievers, you can change one so that it misses less. BM25 only matches words that appear in a product's text. Learned sparse retrieval keeps BM25's machinery, an inverted index of terms, but has a neural model choose each product's terms and their weights. SPLADE is one learned sparse retrieval model (Formal et al., 2021). A language model of the BERT family, a transformer that reads text as subword tokens, reads the product's text and, at each token i, gives every term j in its vocabulary a raw score w_ij. ReLU sets the negative scores to 0, and the original paper sums a logarithm of what's left over the text's tokens:
weight of term j = sum over the text's tokens i of log(1 + ReLU(w_ij))
The logarithm is a saturation, like BM25's: in the authors' words, it "prevents some terms to dominate and naturally ensures sparsity in representations". SPLADE v2 replaced the sum with the maximum over the tokens, and the authors made max pooling their default (Formal et al., 2021, SPLADE v2). A SPLADE vector has one dimension per vocabulary entry, 30,522 for a BERT vocabulary, and typically more than 99% of them are zero (Sentence Transformers). The nonzero terms of a SPLADE vector include expansion terms: words the text doesn't contain but the model weights anyway, so a product can match a query word it never uses.
Expansion terms target one of BM25's failures in the store. Count, on the train queries, how many exact products lack at least one of the query's words, as BM25's tokenizer sees them:
1"""Counts the exact train pairs whose product lacks a word of its query, as BM25's tokenizer sees them, and counts the2exact products of one query by term. The prose's point rests on the share missing a query word and on the per-term3counts for midcentury tv unit."""4from collections import Counter56import bm25s78from lib.store import load_products, load_queries, product_text, qrels910products, queries, judgments = load_products(), load_queries("train"), qrels("train")111213def words(texts):14 """The tokens BM25 matches on: bm25s's default tokenizer with English stopwords, as in lib.lexical."""15 tokenized = bm25s.tokenize(texts, stopwords="en", return_ids=False, show_progress=False)16 return [set(tokens) for tokens in tokenized]171819product_words = dict(zip(products.index, words(product_text(products).tolist())))20query_words = dict(zip(queries.index, words(queries["query"].tolist())))21exact_pairs = [(query_id, product_id) for query_id in queries.index22 for product_id, grade in judgments[query_id].items() if grade == 2]23missed = Counter(word for query_id, product_id in exact_pairs24 for word in query_words[query_id] - product_words[product_id])25# a pair counts as missing when some query word is absent from the product's set of tokens26missing = sum(not query_words[query_id] <= product_words[product_id] for query_id, product_id in exact_pairs)27print("exact train pairs:", len(exact_pairs))28print("exact pairs missing a query word:", missing)29print(f"share missing a query word: {missing / len(exact_pairs):.3f}")30print("most often missing:", ", ".join(f"{word} ({count})" for word, count in missed.most_common(8)))3132exact = [product_id for query_id, product_id in exact_pairs if query_id == 453] # midcentury tv unit33print("exact products for midcentury tv unit:", len(exact))34for word in ["midcentury", "unit", "stand"]:35 print(f"exact products containing {word}:", sum(word in product_words[product_id] for product_id in exact))36mid_century = product_text(products)[exact].str.contains("mid-century")37print("exact products whose text says mid-century:", mid_century.sum())
python exact-products-missing-query-words.pyexact train pairs: 1937
exact pairs missing a query word: 1454
share missing a query word: 0.751
most often missing: ligth (193), decorative (134), modern (102), freestanding (78), floats (58), queen (57), dining (56), infant (43)
exact products for midcentury tv unit: 25
exact products containing midcentury: 0
exact products containing unit: 0
exact products containing stand: 25
exact products whose text says mid-century: 20
Of 1937 exact (query, product) pairs, 1454 miss at least one query word, a share of 0.751. The words missed most often, with their counts, are ligth (193), decorative (134), modern (102), freestanding (78), floats (58), queen (57), dining (56), infant (43). Some are typos, which chapter 8's corrector targets, and plural forms, which chapter 5 stems; others are descriptive words the product text leaves out, or different words for one thing.
For midcentury tv unit, the exact products use other words. Of the 25 exact products, 0 contain midcentury and 0 contain unit. Their text says mid-century instead in 20 and stand in 25.
Expansion terms are aimed at gaps like these, though whether a model learns that a TV stand is a TV unit is something you'd measure.
The SPLADE authors' own technical report gives SPLADE-v3 an MRR@10 above 0.40 on the MS MARCO development set, a passage-ranking benchmark where MRR@10 measures how high the first relevant passage lands, and describes it as "comparing well to cross-encoder re-rankers" (Lassance et al., 2024). The book doesn't run SPLADE. The reference checkpoints, naver/splade-cocondenser-ensembledistil and naver/splade-v3, are licensed CC-BY-NC-SA-4.0, which rules out commercial use. An Apache-2.0 model, opensearch-neural-sparse-encoding-doc-v3-distill, runs its neural model on products only, and encodes queries with a tokenizer and a table of weights.
Working rule: Consider learned sparse retrieval when BM25's misses are mostly vocabulary misses and you want to keep an inverted index. Check the model's licence first, and measure it like any retriever: alone, per query, on train queries, then on test.
以 ColBERT 逐詞元比對
SPLADE 改變的是詞彙檢索,延遲互動(late interaction)改變的則是語意檢索。商店目前的語意檢索是單向量(single-vector):每項產品一個嵌入,每個查詢一個嵌入,只比對一次。延遲互動改為每個詞元保留一個向量,而非每段文字一個向量,因此也稱為多向量(multi-vector)檢索。ColBERT 如同嵌入模型一般,分別對查詢與產品編碼,直到最後才進行比對。對於每個查詢詞元,ColBERT 會找出最相似的產品詞元,再將這些最大值加總,這項運算稱為 MaxSim(Khattab and Zaharia, 2020):
score(query, product) = sum over query tokens q of max over product tokens d of cosine(q, d)
由於產品是在沒有查詢的情況下編碼,其詞元向量可以預先計算;交叉編碼器(cross-encoder)則做不到,因為它把兩段文字當成同一個輸入來讀取。ColBERT 論文指出,其成效「可與現有基於 BERT 的模型相抗衡」,而那些模型是將查詢與段落一併讀取,速度卻「快了兩個數量級」。代價是儲存空間:每個詞元一個向量,而非每項產品一個向量。ColBERTv2 對這些向量加以壓縮,使延遲互動模型所需的空間減少 6 至 10 倍(Santhanam et al., 2022)。
回到 midcentury tv unit,其準確的產品寫的是 stand,從不寫 unit。單一向量把整個查詢收攏成一個方向,把整段產品文字收攏成另一個方向。採用延遲互動時,查詢中的 unit 會與每個產品詞元比對,其中包括 stand,而它的最佳匹配會單獨計入。詞元層級的比對在某些困難案例中有幫助。NevIR 是一項否定句基準測試,隨機選擇的得分為 25%;在該測試中,ColBERTv1 得分為 19.7%,ColBERTv2 為 13.0%,高於所有受測的單向量模型(12% 或更低),但仍低於隨機水準(Weller et al., 2024)。BEIR 是一項以零樣本方式、在模型未受訓練的資料集上進行測試的基準;在其中,重排序器與延遲互動模型「平均而言達到最佳的零樣本表現」,但運算成本很高(Thakur et al., 2021)。目前已有開放的工具,例如 PyLate 函式庫與採用 Apache-2.0 授權的模型 lightonai/GTE-ModernColBERT-v1(Chaffin and Sourty, 2025)。本書同樣沒有執行 ColBERT。
**實用原則:**當單向量檢索遺漏了取決於單一詞彙的細節,而且你負擔得起每個詞元一個向量的儲存時,可以考慮延遲互動。對於與商店規模相當的商品目錄,我們建議先在短名單上測試重排序器(第 9 章):它在查詢時讀取相同的細節,也不需要新的索引,不過第 9 章的重排序器都沒有顯示出相對於商店語意檢索已確立的增益。
判斷混合檢索何時值得採用
融合是否有幫助,取決於被融合的檢索器,已發表的研究結果也並不一致。詞彙檢索與稠密檢索(稠密指以嵌入為基礎)即使在稠密模型較弱時也能互補:在領域外,零樣本混合檢索相較於深度檢索模型,在三個資料集上平均取得 20.4% 的相對提升(T. Chen et al., 2022)。詞彙基準線很重要。稠密檢索器 DPR 的作者發現混合檢索幾乎沒有增益,而以更強的 BM25 進行的複現研究則發現「具統計顯著性的增益」(Ma et al., 2021)。語意檢索器的強度同樣重要。在 BRIGHT 這個以推理密集查詢為主的基準上,以 α = 0.5 對 min-max 分數做加權和來融合 BM25,使中階稠密檢索器的 nDCG@10 提高 3.5 至 4.9 分,卻使最強的兩個檢索器損失 2.0 至 2.9 分。作者警告,融合可能損害本來就很強的檢索器,因為「融合後的清單會提升詞彙相似但語意無關的文件」(Abdallah et al., 2026)。
實際營運的報告通常是將混合檢索與單獨的詞彙搜尋相比。一家零售商的預印本報告指出,在線上 A/B 測試中,混合詞彙與向量搜尋優於單獨的詞彙搜尋,點擊率提高 0.97%,轉換率提高 0.98%(Sonagara et al., 2026)。這是支持在詞彙系統上加入語意檢索的證據,而不是支持在強大的語意檢索上加入詞彙檢索的證據,後者才是這家商店的問題。
這家商店屬於強檢索器的情況:在測試查詢上,第 3 章確立了語意檢索領先 BM25,而且兩種調校過的混合都無法勝過它。商店使用的 BM25 是本書未做詞幹處理的基準線。第 5 章會建立一個更強的版本,而 Ma et al. 的複現研究正是用它重新測試各種混合的理由。
融合分數並不是混合檢索的唯一形式。搜尋引擎也可以把詞彙比對當作語意檢索的過濾器:有一套引擎在其查詢語言中加入嵌入運算子,使「任意組合嵌入與詞項的混合檢索表達式」能以單一查詢執行(Huang et al., 2020)。商店的過濾器可以要求尺寸(例如 48)必須完全相符,再以嵌入對相符結果排序。這是有待測試的假設,而不是結果,而且單純的過濾器會在 7qt slow cooker 上失敗,因為沒有任何商品使用這種拼法。判斷查詢何時帶有這類限制屬於查詢理解(第 8 章),第 8 章會針對購物者提到顏色、形狀、尺寸或表面處理的後續訊息,嘗試這樣的過濾器。為每個查詢選擇權重同樣屬於查詢理解,本書不會實作。若為每個訓練查詢挑選較佳的排序,得分會是 0.802,而逐查詢權重會混合兩份清單,不受該數字的限制。有一種逐查詢的方法是請 LLM(大型語言模型)評判各檢索器的第一名結果,並據此設定 α(Hsu and Tzeng, 2025)。BRIGHT 研究發現其以分數為基礎的變體並不穩定(Abdallah et al., 2026)。
**實用原則:**當兩個檢索器的品質相近、嵌入模型在其訓練領域之外運作,或許多查詢依賴嵌入所遺漏的精確詞項時,我們建議嘗試混合檢索。先從加權 RRF 開始:它只需要排名,而且在商店的訓練查詢上,排列檢定無法區分它與加權和。當兩個引擎都能為每個候選項給出分數時,改用正規化分數的加權和:有兩項研究發現它在其資料上優於 RRF(Bruch et al., 2023;Abdallah et al., 2026),而且在商店的訓練查詢上,它的權重更為穩定。當某個檢索器在保留查詢上被確立為較強,且沒有任何調校過的混合能在該處勝過它時,就單獨使用它,保留另一個作為基準線,並測試用於精確限制的詞彙過濾器。這就是商店的情況。
正式環境中會出什麼問題
**兩個檢索器,每一項成本都是兩倍。**這些範例是離線對目錄的一個樣本評分,但實際運作的混合搜尋必須同時維護兩份索引,且每次查詢都要執行兩次搜尋。兩者可以平行執行,但較慢的一方決定回應時間,任何一方失敗都會使結果劣化。請把這項成本和你量測到的增益相比較。
**清單太短會漏掉候選項目。**如果兩個檢索器都沒有回傳某項商品,融合就無法為它排名。在商店的訓練查詢上,兩份前 100 名清單的聯集涵蓋了每個查詢精確商品的 0.810,兩份前 20 名清單的聯集則涵蓋 0.445。某個引擎設定中的深度上限,可能在沒人察覺的情況下截掉候選項目。
**缺漏的分數需要一條規則。**當每個檢索器只回傳前 k 名時,只出現在其中一份清單的商品,在另一個檢索器那裡沒有分數。DPR 的重現實驗為它指定「被檢索到的文件中最低的稠密檢索分數」,BM25 則反過來處理(Ma et al., 2021)。Bruch 等人則改為精確計算缺漏的分數,lib.blend.candidates 也是這樣做,但正式環境的引擎不一定允許你這麼做(Bruch et al., 2023)。
**Min-max 是相對的。**對某個查詢而言,最佳候選項目一律得到 1,即使所有候選項目都很差也一樣。若某個查詢與目錄中任何商品都沒有共同的字詞,每個候選項目的 BM25 分數都是 0。單純的 min-max 此時會除以零,而 lib.blend 會回傳零,於是這個混合結果在無聲無息中變成只剩語意檢索。商店的查詢沒有一個如此,但實際的搜尋框會遇到。
**融合分數不是信心值。**檢索分數的校準很差:「各個模型家族的信心校準始終薄弱」(Abdallah et al., 2026)。不要依融合分數設定門檻,來判定某個查詢沒有好的結果。
**調校好的權重只屬於一組檢索器。**商店的 α 適用於這些 BM25 設定、這個嵌入模型和這份目錄。其中任何一項改變,都要先用訓練查詢重新調校,再決定是否信任舊權重。調校過的 RRF「在領域外的資料集上泛化能力不佳」(Bruch et al., 2023)。
**實用原則:**在融合結果旁記錄兩個檢索器各自的排名,檢查每個引擎設定中的清單深度,並在任一檢索器改變時重新調校 α。
本章做出的決定
日後若要更動其中任何一項,都必須有理由勝過原先的決定。
- 在訓練集上調校,在測試集上報告。 所有分析與所有調校後的設定,都使用 50 筆訓練查詢。本章只讀取測試查詢一次,用來報告結果。
- **兩項預先規劃的比較:**將各個調校後的混合方式與語意檢索比較,以 0.05 除以 2 作為門檻,由置換檢定決定,並在旁附上 t 檢定與自助法(bootstrap)區間。
- **融合之前先檢查各個檢索器,**使用 judged@10 與成對檢定,並逐一查詢計算兩者各自勝出的次數。
- **候選集:兩個檢索器前 100 名的聯集,**每個候選都查出兩種分數,因此不會有商品缺少分數。
- 正規化:逐查詢的 min-max。 在訓練集上,置換檢定在 0.05 的水準下無法區分另外兩種正規化與它的差異,而且 min-max 不需要知道任何一個評分函式的特性。
- RRF:k = 60,名單深度為 100。 訓練集上最佳的 k 相較於 60 並沒有帶來置換檢定在 0.05 的水準下能與隨機波動區分的增益,而且它會隨 BM25 同分項目的順序而改變。更深的名單只會增加候選。
- **單一權重 α,在 0 到 1 之間以 0.1 為間隔調校,**兩種混合方式皆然,依訓練集上的平均 nDCG@10 選定,接著進行自助法,並在反轉 BM25 同分順序後再調校一次。加權和的 α 通過這兩項檢查,加權 RRF 的 α 則沒有。
- **SPLADE 與 ColBERT 只作說明,不加以記錄,**因為回答商店的問題並不需要這兩者。
- 只有在測試集上勝過最佳的單一檢索器時,才採用混合方式。 兩種調校後的混合方式都沒有做到,因此商店要超越的排序器仍然是單獨的語意檢索,並保留未做詞幹還原的 BM25 作為基準。
- **本章所示:**混合方式只有在其檢索器於不同的查詢上失敗時才可能有幫助,而且在訓練集上勝出的權重必須在測試集上依然成立;在這間商店,兩種調校後的混合方式都沒有勝過語意檢索。
本章,於本版次中
版次 2026.11.10範例於 2026年11 月 19 通過測試本章 12 個範例 個範例中有 12 個通過
譯自英文書,版次 2026.11.10。