混合检索

阅读约需 83分钟 分钟,12节中含12个示例。

版次 2026.11.10 示例于 2026年11月19日 通过 132个示例中有132个通过 变化内容

AI 翻译。由人工以英文核验。 我们如何制作图书

译自英文版图书,版次 2026.11.10。 阅读原文

读完本章,你将能够把词法排序和语义排序合并为一个排序,方法是累加归一化后的分数,或使用倒数排名融合(reciprocal rank fusion);能够在训练查询上调节两种排序各自所占的权重;还能够在测试查询上通过配对检验,判断这种组合是否优于其两个输入中较好的一个。

该商店用两个检索器搜索其家具和家居用品目录,第 1 章把它们融合成了一个排序。这次融合没有胜过单独的语义检索:在测试查询上,融合的 nDCG@10 为 0.812,语义检索为 0.830,而第 3 章的检验对这一差距究竟是否存在也没有给出结论。因此,商店需要弄清楚,是否存在更好的混合方式能够胜过语义检索,后者是目前最好的排序器,也就是需要超越的对象。

所以,本章首先考察 BM25和语义检索是否在不同的查询上失效。前者是本书采用 bm25s 默认设置、不做词干提取的基线,后者使用开源模型 Snowflake/snowflake-arctic-embed-m-v1.5 生成嵌入。只有在两者失效之处不同,混合才可能有帮助,因此本章随后调节两种混合方式。

工业界的搜索系统常常把两者搭配使用。一个语义商品搜索模型被构建出来,是为了“在临时(ad hoc)商品检索中补充词法搜索引擎”(Nigam et al., 2019);某社交网络的搜索引擎则同时使用嵌入检索和关键词匹配为文档打分(Huang et al., 2020)。

商店有 50 条用于调节的训练查询,以及 30 条用于报告结果的测试查询。本章先找出每个检索器各自获胜的查询,并分别检查每个检索器单独的表现。随后构建两种混合方式(blend),即用一个权重来融合词法排序和语义排序、由该权重决定两者各占多大比重的方法,并调节这个权重。所有分析和所有调节步骤都在训练查询上进行。本章计划在测试查询上做两项比较,且只在最后做一次:把调节后的每种混合方式,即加权求和与加权倒数排名融合,分别与语义检索比较。

这部分尚未翻译,因此以英文显示。

Find the queries each retriever wins

最近一次修改见于版次 2026.11.8

Hybrid retrieval combines the rankings of a lexical and a semantic retriever into one ranking. Combining rankings is called fusion. Fusion can only help if the two retrievers fail on different queries: if both put poor products first for a query, no mix of their rankings fixes it. Published studies find that they do. Embeddings miss rare entities and selective keywords that BM25 matches (chapter 6), and on a dataset built to test the limits of one embedding per text, BM25 finds what embeddings miss but loses most of its score when words are replaced with synonyms (chapter 1). So the first check is per query, not on average.

Score both of the store's retrievers by nDCG@10 on every train query, keeping each one's top 100.

bm25-semantic-wins-by-query.py Python 3.12.15 · bm25s 0.3.16 · numpy 2.5.5 · pandas 3.1.1 · scipy 1.18.1 通过
"""Scores BM25 and semantic retrieval on each train query and counts the queries each one wins, ties included.The prose's point rests on the win counts and on the mean of the better score per query."""import numpy as npfrom lib.lexical import LexicalSearchfrom lib.metrics import evaluatefrom lib.semantic import SemanticSearchfrom lib.store import load_products, load_queries, qrelsproducts, queries, judgments = load_products(), load_queries("train"), qrels("train")bm25 = LexicalSearch(products).run(queries, k=100)semantic = SemanticSearch().run(queries, k=100)bm25_scores, semantic_scores = evaluate(bm25, judgments)["ndcg@10"], evaluate(semantic, judgments)["ndcg@10"]print("train queries:", len(queries))print(f"bm25 ndcg@10: {bm25_scores.mean():.3f}")print(f"semantic ndcg@10: {semantic_scores.mean():.3f}")# rounded to 3 places, so differences that are float noise count as tieslead = (bm25_scores - semantic_scores).round(3)              # per train query; above 0 means BM25 winsprint("bm25 better:", (lead > 0).sum())print("semantic better:", (lead < 0).sum())print("tied:", (lead == 0).sum())print(f"better of the two, chosen per query: {np.maximum(bm25_scores, semantic_scores).mean():.3f}")has_digit = queries["query"].str.contains(r"\d")print("queries with a digit:", has_digit.sum())print("queries with a digit, bm25 better:", (lead[has_digit] > 0).sum())print()print("queries where bm25 leads by 0.1 or more:", (lead >= 0.1).sum())print(queries.loc[lead[lead >= 0.1].index, "query"].to_string())
输出python bm25-semantic-wins-by-query.py
train queries: 50
bm25 ndcg@10: 0.695
semantic ndcg@10: 0.772
bm25 better: 15
semantic better: 31
tied: 4
better of the two, chosen per query: 0.802
queries with a digit: 5
queries with a digit, bm25 better: 1

queries where bm25 leads by 0.1 or more: 6
query_id
44     chrome shower head and valve
146          laundry vanity cabinet
277             barstool patio sets
313                    malachi sled
359                  french molding
432            butcher block island

Neither retriever wins every query. Semantic retrieval averages 0.772 nDCG@10 against BM25's 0.695, but BM25 still wins 15 train queries to semantic retrieval's 31, and 4 tie.

If you could choose the better of the two for each query, the mean would be 0.802. That isn't a ceiling for fusion, which mixes the lists rather than choosing one, but it shows what the differences could be worth.

Next, see whether the two lists hold different products, and what share of each query's exact products (those the judges graded as fully matching) each list finds:

overlap-and-union-recall.py Python 3.12.15 · bm25s 0.3.16 · numpy 2.5.5 · pandas 3.1.1 · scipy 1.18.1 通过
"""Measures how much BM25's and semantic retrieval's top 10s overlap, and the share of each query's exact productsthat each list, and their union, holds at depths 100 and 20. The prose's point rests on the union's shares at both depths."""import numpy as npfrom lib.lexical import LexicalSearchfrom lib.metrics import recall_at_kfrom lib.semantic import SemanticSearchfrom lib.store import load_products, load_queries, qrelsproducts, queries, judgments = load_products(), load_queries("train"), qrels("train")bm25 = LexicalSearch(products).run(queries, k=100)semantic = SemanticSearch().run(queries, k=100)shared = [len(set(bm25[query_id][:10]) & set(semantic[query_id][:10])) for query_id in queries.index]print(f"products in both top 10s, mean: {np.mean(shared):.1f}")def share_retrieved(run):    """The share of each query's exact products that the run's list holds, averaged over queries."""    # k is the whole list, so this is the share of exact products (grade 2) anywhere in it    shares = [recall_at_k(ranking, judgments[query_id], k=len(ranking)) for query_id, ranking in run.items()]    return np.mean(shares)for depth in [100, 20]:    lists = {"bm25": {query_id: ranking[:depth] for query_id, ranking in bm25.items()},             "semantic": {query_id: ranking[:depth] for query_id, ranking in semantic.items()}}    # the union in first-seen order, without repeats    lists["union"] = {query_id: list(dict.fromkeys(lists["bm25"][query_id] + lists["semantic"][query_id]))                      for query_id in queries.index}    for name, run in lists.items():        print(f"{name} top {depth}, share of exact products retrieved: {share_retrieved(run):.3f}")
输出python overlap-and-union-recall.py
products in both top 10s, mean: 3.7
bm25 top 100, share of exact products retrieved: 0.672
semantic top 100, share of exact products retrieved: 0.759
union top 100, share of exact products retrieved: 0.810
bm25 top 20, share of exact products retrieved: 0.297
semantic top 20, share of exact products retrieved: 0.353
union top 20, share of exact products retrieved: 0.445

The two top 10s share 3.7 products on average, so most of each is missing from the other. Fusion can't rank a product neither list holds, so each list's depth caps what fusion can reach. The union beats either list at both depths:

Share of exact products retrieved Top 20 Top 100
BM25 0.297 0.672
Semantic retrieval 0.353 0.759
Union of both 0.445 0.810

Two train queries show the kinds of failure:

two-queries-two-winners.py Python 3.12.15 · bm25s 0.3.16 · numpy 2.5.5 · pandas 3.1.1 · scipy 1.18.1 通过
"""Prints the top 5 products of BM25 and semantic retrieval for two train queries, with their judgments, and countsthe exact products in each top 10. The prose's point rests on those exact counts and on the two spellings of 7qt."""from lib.lexical import LexicalSearchfrom lib.semantic import SemanticSearchfrom lib.store import load_products, load_queries, product_text, qrelsLABELS = {2: "exact", 1: "partial", 0: "irrelevant"}products, queries, judgments = load_products(), load_queries("train"), qrels("train")bm25 = LexicalSearch(products).run(queries, k=10)semantic = SemanticSearch().run(queries, k=10)text = product_text(products)# the size written as one token, `7qt`print("products containing 7qt:", text.str.contains(r"\b7qt\b", case=False).sum())# the size written apart, `7 qt`print("products containing 7 qt:", text.str.contains(r"\b7 qt\b", case=False).sum())for query_id in [432, 62]:                                   # butcher block island, 7qt slow cooker    query = queries.at[query_id, "query"]    for name, run in [("bm25", bm25), ("semantic", semantic)]:        # grade 2 is exact, 1 partial, 0 irrelevant; None for a product nobody judged        grades = [judgments[query_id].get(product_id) for product_id in run[query_id]]        print()        print(f"exact in {name} top 10 for {query}: {grades.count(2)}")        for product_id, grade in list(zip(run[query_id], grades))[:5]:            print(f"  {LABELS.get(grade, 'unjudged'):<10}  {products.at[product_id, 'name'][:55]}")
输出python two-queries-two-winners.py
products containing 7qt: 0
products containing 7 qt: 9

exact in bm25 top 10 for butcher block island: 9
  exact       carol kitchen island with butcher block top
  exact       brunton kitchen island with butcher block top
  partial     brunton kitchen cart butcher block top
  exact       janigian kitchen island with butcher block
  exact       avelar kitchen island with butcher block

exact in semantic top 10 for butcher block island: 3
  partial     boosblock® prep table with butcher block top
  partial     boosblock® butcher block
  partial     prep table
  partial     pro chef kitchen cart with butcher block top
  exact       brunton kitchen island with butcher block top

exact in bm25 top 10 for 7qt slow cooker: 1
  partial     cuisinart 6.5 quart programmable slow cooker
  partial     midea 6.5 qt . slow cooker
  partial     delonghi cks1660d livenza programmable 6 qt . slow cook
  partial     nesco 1.5 qt . analog slow cooker
  partial     360 cookware gourmet slow cooker

exact in semantic top 10 for 7qt slow cooker: 7
  exact       7-qt . stay or go slow cooker
  exact       crock-pot 7 qt . slow cooker
  exact       zing 7 qt . digital slow cooker
  exact       black + decker 7 qt . digital slow cooker
  exact       crock-pot 7 qt . slow cooker

For butcher block island, BM25's top 10 holds 9 exact products to semantic retrieval's 3. BM25 rewards products whose text contains all three words: kitchen islands with butcher-block tops. Semantic retrieval ranks prep tables and carts with butcher-block tops near the top too, and the judges graded those partial: the right material, but not the island the query asks for.

For 7qt slow cooker it's the other way round: BM25's top 10 holds 1 exact and semantic retrieval's 7. The catalog has 0 products whose text contains 7qt and 9 that contain 7 qt, so BM25 matches only slow cooker and ranks cookers of every size, while semantic retrieval puts 7-quart cookers first.

Those are single queries: they show a kind of failure, not its frequency. The per-query count lists the 6 train queries where BM25 leads by 0.1 nDCG@10 or more. Most name a specific kind of product in two or more words that all have to match, and one, malachi sled, names a product line.

Chapter 1 found BM25 ahead on 2 of the 2 test queries that give a size in inches, yet of the 5 train queries that contain a digit, BM25 wins 1.

Working rule: Before you fuse, count per query where each retriever wins on your train queries, and check how many relevant products the union of their lists holds that the better one misses. Consider fusing when each wins a real share of queries and the union holds more relevant products than either list. That is necessary, not sufficient: next, check each retriever alone, and test the tuned blend on held-out queries before you adopt it.

融合之前先检查每个检索器

最近一次修改见于版次 2026.11.10

只知道各检索器在哪些查询上占优还不够:融合会让每个输入都对最终排序有发言权,较弱的输入也不例外。一项针对 11 个数据集的混合检索研究发现了“最弱环节”效应:一个较弱的检索器就可能显著拉低融合结果的准确率,因此融合之前应当逐一检查(Wang et al., PVLDB 2026)。所以,先比较各检索器的平均 nDCG@10,并在旁边列出 judged@10。然后用配对检验逐查询地比较:如果两者同样好,出现这么大的平均差异的可能性有多大?本书使用的检验是置换检验,旁边再配以配对 t 检验。

像第 1 章那样,用倒数排名融合(RRF)来融合 BM25 与语义检索:每个商品在每个排序中的得分为 1 / (60 + 其排名),再把两个排序中的得分相加,因此每个排序的票数相等。然后按胜者拆分训练查询,并将融合与单独使用语义检索作比较:

fusion-gain-by-winning-retriever.py Python 3.12.15 · bm25s 0.3.16 · numpy 2.5.5 · pandas 3.1.1 · scipy 1.18.1 通过
"""Compares BM25, semantic retrieval and their RRF fusion on the train queries, then splits the queries by whichretriever wins. The prose's point rests on the hybrid-minus-semantic lines, which give the net change, and on the meanchange within each winning group."""from lib.fusion import reciprocal_rank_fusionfrom lib.lexical import LexicalSearchfrom lib.metrics import evaluatefrom lib.semantic import SemanticSearchfrom lib.significance import comparefrom lib.store import load_products, load_queries, qrelsproducts, queries, judgments = load_products(), load_queries("train"), qrels("train")bm25 = LexicalSearch(products).run(queries, k=100)semantic = SemanticSearch().run(queries, k=100)hybrid = reciprocal_rank_fusion([bm25, semantic])           # k = 60 and equal votes, as in chapter 1bm25_scores, semantic_scores, hybrid_scores = (evaluate(run, judgments) for run in (bm25, semantic, hybrid))print(f"bm25 judged@10: {bm25_scores['judged@10'].mean():.3f}")print(f"semantic judged@10: {semantic_scores['judged@10'].mean():.3f}")for name, after, before in [("semantic minus bm25", semantic_scores, bm25_scores),                            ("hybrid minus semantic", hybrid_scores, semantic_scores)]:    result = compare(after["ndcg@10"], before["ndcg@10"])                    # paired by query    print(f"{name}, mean ndcg@10: {result['mean difference']:.3f}")    print(f"{name}, permutation p: {result['permutation p']:.3f}")    print(f"{name}, t-test p: {result['t-test p']:.3f}")print()# rounded to 3 places, so float noise can't make a query a win or a losschange = (hybrid_scores["ndcg@10"] - semantic_scores["ndcg@10"]).round(3)   # fusion against semantic alonelead = (bm25_scores["ndcg@10"] - semantic_scores["ndcg@10"]).round(3)       # above 0 means BM25 winsfor name, group in [("bm25 wins", lead > 0), ("semantic wins", lead < 0)]:    print(f"{name}, queries: {group.sum()}")    print(f"{name}, fusion better than semantic: {(change[group] > 0).sum()}")    print(f"{name}, fusion worse than semantic: {(change[group] < 0).sum()}")    print(f"{name}, mean change in ndcg@10: {change[group].mean():.3f}")
输出python fusion-gain-by-winning-retriever.py
bm25 judged@10: 0.930
semantic judged@10: 0.964
semantic minus bm25, mean ndcg@10: 0.077
semantic minus bm25, permutation p: 0.003
semantic minus bm25, t-test p: 0.004
hybrid minus semantic, mean ndcg@10: 0.002
hybrid minus semantic, permutation p: 0.863
hybrid minus semantic, t-test p: 0.872

bm25 wins, queries: 15
bm25 wins, fusion better than semantic: 12
bm25 wins, fusion worse than semantic: 2
bm25 wins, mean change in ndcg@10: 0.101
semantic wins, queries: 31
semantic wins, fusion better than semantic: 5
semantic wins, fusion worse than semantic: 25
semantic wins, mean change in ndcg@10: -0.045

BM25 的 judged@10 为 0.930,语义检索为 0.964,所以 BM25 的前 10 名中含有更多无人标注过相关性的商品,而第 2 章说明了这些商品如何拉低得分。语义检索的 nDCG@10 领先 0.077,置换检验的 p 值为 0.003,t 检验的 p 值为 0.004。在测试查询上,第 3 章确认了同样的领先。因此,这家商店有一个强检索器和一个较弱的检索器。

按胜者拆分后,融合在 BM25 占优的查询上平均有帮助,在语义检索占优的查询上则有损害:

占优的检索器 查询数 融合优于语义检索 融合劣于语义检索 nDCG@10 的平均变化
BM25 占优 15 12 2 0.101
语义检索占优 31 5 25 -0.045

按胜者给查询分组,本身就会造出这种模式,因为几乎任何混合在 BM25 占优时都有帮助,在它落败时都有损害。重要的是在所有训练查询上的净变化。融合与语义检索的平均 nDCG@10 相差 0.002,置换检验的 p 值为 0.863。在这些查询上,平权投票没有带来可测量的收益。

这种拆分用相关性标注来决定哪个检索器获胜,因此它只能用来诊断这种模式,不能为线上查询挑选检索器,因为线上查询没有相关性标注。

工作准则: 融合之前,先在训练查询上单独检查每个检索器,使用 judged@10 和配对检验。当其中一个较弱、赢下的查询较少时,不要给它平等的票数:为它加权并调整权重,或者不用它。平权投票适合质量大致相当的两个检索器。

先归一化分数再相加

最近一次修改见于版次 2026.11.10

要让较弱的检索器(商店里是 BM25)发挥较小的作用,最直接的办法是按权重把两个检索器的分数相加:α 乘以语义分数,加上 (1 − α) 乘以 BM25 分数。权重之和为 1 的加和称为凸组合,也就是加权和。只有当两种分数处于可比较的尺度上,这种做法才行得通,而这里并非如此:BM25 分数没有固定的取值范围,会随查询词的数量和稀有程度增大;语义检索的分数是余弦相似度,介于 −1 和 1 之间。分数归一化会在相加之前,针对每个查询对各检索器的分数重新缩放。按查询的 min-max 缩放,会把该查询已排序商品中的最低分映射为 0,最高分映射为 1。一项关于融合函数的研究称它是“文献中事实上的标准方法”,并检验了两种替代方法(Bruch et al., 2023)。理论 min-max 不用已见的最低值,而是用各分数可能取到的最低值:BM25 为 0,余弦为 −1。z 分数则减去查询分数的均值,再除以其标准差。各示例共用的模块 lib/blend.py 用几行代码就写出了这三种方法。

min-max(s) = (s − lowest score among the query's candidates) / (highest − lowest)
blend      = α · min-max(similarity) + (1 − α) · min-max(BM25)

求和要求每个参与排序的商品都有两个分数。一个查询的候选集是 BM25 前 100 名与语义检索前 100 名的并集,lib.blend.candidates 会为其中每个商品查出两个分数:BM25 分数(商品与查询没有共同词时为 0)和余弦相似度。

先看商店的原始分数:在训练查询上取 α = 0.5 相加,再在 min-max 缩放之后做同样的事。示例中手工写出了这两个过程,并用 lib.blend.weighted_sum 做了核对:

raw-scores-versus-min-max.py Python 3.12.15 · bm25s 0.3.16 · numpy 2.5.5 · pandas 3.1.1 · scipy 1.18.1 通过
"""Adds BM25 and cosine scores with equal weights, raw and after min-max scaling, and checks the scaled sum againstlib.blend.weighted_sum. The prose's point rests on the two sums' top 10 overlaps and nDCG@10, and on the count oftrain queries where the by-hand ranking matches the library's."""from lib.blend import candidates, weighted_sumfrom lib.lexical import LexicalSearchfrom lib.metrics import evaluatefrom lib.semantic import SemanticSearchfrom lib.significance import comparefrom lib.store import load_products, load_queries, qrelsproducts, queries, judgments = load_products(), load_queries("train"), qrels("train")lexical, semantic = LexicalSearch(products), SemanticSearch()tables = candidates(lexical, semantic, queries)              # both scores for each query's candidates# the candidate tables are keyed by query_idtable = tables[36]                                           # laundry basket with wheelsprint("candidates for laundry basket with wheels:", len(table))print(f"bm25 range: {table['bm25'].min():.3f} to {table['bm25'].max():.3f}")print(f"similarity range: {table['similarity'].min():.3f} to {table['similarity'].max():.3f}")def min_max(scores):    return (scores - scores.min()) / (scores.max() - scores.min())def blend(normalize, alpha=0.5):    run = {}    for query_id, table in tables.items():        score = alpha * normalize(table["similarity"]) + (1 - alpha) * normalize(table["bm25"])        # the top 100; the stable sort keeps the candidates' order on equal scores        run[query_id] = list(score.sort_values(ascending=False, kind="stable").index[:100])    return runtop_10s = {"bm25": lexical.run(queries, k=10), "semantic": semantic.run(queries, k=10)}per_query = {}for name, normalize in [("raw", lambda scores: scores), ("min-max", min_max)]:    run = blend(normalize)    per_query[name] = evaluate(run, judgments)["ndcg@10"]    print(f"{name} sum ndcg@10: {per_query[name].mean():.3f}")    for other, top in top_10s.items():        shared = [len(set(run[query_id][:10]) & set(top[query_id])) for query_id in queries.index]        print(f"{name} sum, products shared with the {other} top 10: {sum(shared) / len(shared):.1f}")# the library sorts equal scores by product_id; this blend keeps the candidates' orderby_hand, library = blend(min_max), weighted_sum(tables, 0.5)matches = sum(by_hand[query_id] == library[query_id] for query_id in tables)print(f"train queries ranked as lib.blend.weighted_sum ranks them: {matches} of {len(tables)}")p_value = compare(per_query["min-max"], per_query["raw"])["permutation p"]          # a tuning check on trainprint("min-max minus raw, permutation p:", "below 0.001" if p_value < 0.001 else f"{p_value:.3f}")
输出python raw-scores-versus-min-max.py
candidates for laundry basket with wheels: 125
bm25 range: 2.059 to 8.229
similarity range: 0.336 to 0.597
raw sum ndcg@10: 0.706
raw sum, products shared with the bm25 top 10: 9.6
raw sum, products shared with the semantic top 10: 3.9
min-max sum ndcg@10: 0.771
min-max sum, products shared with the bm25 top 10: 6.5
min-max sum, products shared with the semantic top 10: 6.2
train queries ranked as lib.blend.weighted_sum ranks them: 50 of 50
min-max minus raw, permutation p: below 0.001

对于 laundry basket with wheels,125 个候选商品的 BM25 分数范围为 2.059 to 8.229,相似度范围为 0.336 to 0.597。BM25 的跨度要大许多倍,因此原始分数之和的顺序由它决定。在训练查询上,原始分数之和的前 10 名平均与 BM25 的前 10 名共有 9.6 个商品,与语义检索的前 10 名共有 3.9 个。

权重说的是“各占一半”,但排序结果主要由 BM25 决定:原始分数之和的 nDCG@10 为 0.706,与 BM25 的 0.695 很接近。

经过 min-max 缩放之后,两个检索器都有发言权。缩放后的和,其前 10 名与 BM25 的前 10 名共有 6.5 个商品,与语义检索的共有 6.2 个,nDCG@10 升至 0.771。这一结果与原始分数之和的差异,其置换检验 p 值为 below 0.001。手工实现的版本对 50 of 50 个训练查询的排序,与 lib.blend.weighted_sum 的排序完全一致。

原始分数配合调好的权重也可以奏效。基于嵌入的检索器 DPR 的作者,用 BM25 加上 1.1 倍的 DPR 相似度对候选排序,其中的 1.1 是在开发集上选定的(Karpukhin et al., 2020)。归一化使 α 对每个查询都有同一种含义;没有归一化,BM25 所占的发言份额会随它在各查询间取值范围的变化而变化。不过,min-max 并不是绝对尺度:它相对于该查询的候选集,所以无论最好的候选有多差,它总是得到 1。

工作准则: 不要用猜出来的权重把原始的 BM25 分数和余弦分数直接相加。我们推荐把按查询的 min-max 作为默认做法,因为它不需要了解任何一个评分函数。当你知道各分数可能取到的最低值,并希望下限不随候选集变动时,使用理论 min-max。无论哪种情况,都应在训练查询上调节 α。

这部分尚未翻译,因此以英文显示。

Test reciprocal rank fusion's settings on train queries

最近一次修改见于版次 2026.11.8

Ranks, unlike scores, share one scale, so fusing by rank needs no normalization. Reciprocal rank fusion (RRF) scores each product by adding, over the rankings that contain it, 1 divided by the sum of a constant k and its rank (Cormack et al., 2009), as chapter 1's formula writes it. The price is information: "because RRF is a function of ranks, it disregards the distribution of scores and, as such, discards useful information" (Bruch et al., 2023). A product a hair behind the leader and one far behind it get scores set by their ranks alone.

The constant k sets how much the top ranks dominate. Chapter 1 took k = 60 from RRF's authors, who found in a pilot study that 60 came close to the best value and that the exact choice mattered little. Their pilots fused 30 configurations of Wumpus Search, one search engine, on collections from TREC, a series of shared retrieval evaluations (Cormack et al., 2009). Bruch et al. fused one lexical and one semantic retriever, gave each its own constant, η, and found the results sensitive to the balance between the constants, which acts as a weight: "where the semantic model already performs strongly, for example, discounting the lexical system by increasing η_Lex leads to better performance". One k shared by both rankings, as in the store, sets no balance between them.

The depth of each list matters too. A product in one retriever's top 20 but not in the other's earns nothing from the second ranking, though it has a low rank there. As Bruch et al. put it, "ranks cannot be computed exactly unless the entire collection D is ranked by retrieval system o for every query." On the store's train queries, the union of two top-20 lists holds far fewer exact products than the union of two top-100 lists, as the shares of exact products in each list show.

Try six values of k on the train queries, fusing lists 100 deep. One check is planned before running: does the best k beat the default, k = 60?

rrf-k-on-train.py Python 3.12.15 · bm25s 0.3.16 · numpy 2.5.5 · pandas 3.1.1 · scipy 1.18.1 通过
"""Fuses BM25 and semantic retrieval with six values of k on the train queries, and tests the best k against thedefault k = 60. The prose's point rests on the rank ratios at k = 1 and k = 60, on the best k, and on its permutationp-value."""from lib.fusion import reciprocal_rank_fusionfrom lib.lexical import LexicalSearchfrom lib.metrics import evaluatefrom lib.semantic import SemanticSearchfrom lib.significance import comparefrom lib.store import load_products, load_queries, qrelsproducts, queries, judgments = load_products(), load_queries("train"), qrels("train")bm25 = LexicalSearch(products).run(queries, k=100)semantic = SemanticSearch().run(queries, k=100)# RRF gives rank r the score 1 / (k + r), so rank 1 over rank 10 is (k + 10) / (k + 1)for k in [1, 60]:    print(f"rank 1 over rank 10 at k={k}: {(1 / (k + 1)) / (1 / (k + 10)):.3f}")scores = {k: evaluate(reciprocal_rank_fusion([bm25, semantic], k=k), judgments)["ndcg@10"]          for k in [1, 10, 30, 60, 100, 300]}for k, per_query in scores.items():    print(f"k={k} ndcg@10: {per_query.mean():.3f}")best = max(scores, key=lambda k: scores[k].mean())           # ties go to the smaller kprint("best k on train:", best)result = compare(scores[best], scores[60])                     # the planned check: best k against the defaultprint(f"best k minus k=60: {result['mean difference']:.3f}")print(f"best k minus k=60, permutation p: {result['permutation p']:.3f}")print(f"best k minus k=60, t-test p: {result['t-test p']:.3f}")print(f"best k minus k=60, 95% interval: {result['ci low']:.3f} to {result['ci high']:.3f}")print(f"best k minus k=60, wins, losses, ties: {result['wins']}, {result['losses']}, {result['ties']}")
输出python rrf-k-on-train.py
rank 1 over rank 10 at k=1: 5.500
rank 1 over rank 10 at k=60: 1.148
k=1 ndcg@10: 0.753
k=10 ndcg@10: 0.770
k=30 ndcg@10: 0.771
k=60 ndcg@10: 0.774
k=100 ndcg@10: 0.778
k=300 ndcg@10: 0.777
best k on train: 100
best k minus k=60: 0.004
best k minus k=60, permutation p: 0.093
best k minus k=60, t-test p: 0.103
best k minus k=60, 95% interval: 0.001 to 0.011
best k minus k=60, wins, losses, ties: 6, 3, 41

At k = 1, rank 1 earns 5.500 times what rank 10 earns. At k = 60, it earns 1.148 times as much, so a product near the top of both lists beats one at the top of only one. From k = 1 to k = 300, the six settings score 0.753, 0.770, 0.771, 0.774, 0.778 and 0.777.

The best setting on train, k = 100, leads k = 60 by 0.004, with a permutation p-value of 0.093 and a t-test p-value of 0.103. The book takes its verdict from the permutation test, which can't tell that gain from chance at 0.05. The gain's 95% bootstrap interval, 0.001 to 0.011, excludes 0, so the interval disagrees with both tests.

The two settings' rankings differ on only a handful of queries (wins, losses and ties: 6, 3, 41). Which k is best is fragile too: with BM25's tied products listed the other way, k = 300 wins instead. This fits Cormack et al.'s finding that the choice of k matters little.

Working rule: Keep k = 60 as the default, and fuse lists at least 100 deep, because a deeper list can only add candidates. Move off k = 60 only when the best k on train beats it by a margin the permutation test can tell from chance, with p below 0.05. When one retriever is the stronger one, as in the store, k isn't the setting to change: RRF's equal vote is. Give the rankings weights and tune them.

在训练查询上调整融合权重

最近一次修改见于版次 2026.11.8

本章的两种融合各带一个权重 α,即分给语义检索的份额。在加权和中,α 乘以归一化后的相似度,1 − α 乘以归一化后的 BM25 分数。在加权 RRF 中,一个商品的得分是 (1 − α) / (k + 其 BM25 排名) 加上 α / (k + 其语义排名)。对两者而言,α = 0 是仅用 BM25,α = 1 是仅用语义检索。对 RRF 而言,α = 0.5 就是第 1 章的非加权融合,只是每个得分减半。下面以 α = 0.75 作为示意,给出一条训练查询中排名第一的商品的计算过程:

weighted-rrf-one-product.py Python 3.12.15 · bm25s 0.3.16 · numpy 2.5.5 · pandas 3.1.1 · scipy 1.18.1 通过
"""Scores every candidate for one train query by weighted RRF, by hand, and checks the ranking againstlib.blend.weighted_rrf. The prose's point rests on the top product's two ranks and its score, and on the count ofplaces that match the library."""from lib.blend import weighted_rrffrom lib.lexical import LexicalSearchfrom lib.semantic import SemanticSearchfrom lib.store import load_products, load_queriesalpha, k = 0.75, 60                                          # an illustrative weight; tuning picks one on trainproducts, queries = load_products(), load_queries("train")bm25 = LexicalSearch(products).run(queries, k=100)semantic = SemanticSearch().run(queries, k=100)query_id = 36                                                # laundry basket with wheelsdef part(ranking, weight, product_id):    """weight / (k + rank) from one ranking, or 0 when the product isn't in it."""    # index counts from 0, so one is added to get the rank    return weight / (k + ranking.index(product_id) + 1) if product_id in ranking else 0.0candidates = dict.fromkeys(bm25[query_id] + semantic[query_id])   # the union, in the order first seenscore = {product_id: part(bm25[query_id], 1 - alpha, product_id) + part(semantic[query_id], alpha, product_id)         for product_id in candidates}ranking = sorted(score, key=lambda product_id: (-score[product_id], product_id))   # ties by product_idfirst = ranking[0]print("first product:", products.at[first, "name"])print("its bm25 rank:", bm25[query_id].index(first) + 1)print("its semantic rank:", semantic[query_id].index(first) + 1)print(f"its weighted rrf score: {score[first]:.5f}")library = weighted_rrf(bm25, semantic, alpha)[query_id]          # the top 100matches = sum(mine == theirs for mine, theirs in zip(ranking, library))print(f"places that match lib.blend.weighted_rrf: {matches} of {len(library)}")
输出python weighted-rrf-one-product.py
first product: laundry sorter basket on wheels
its bm25 rank: 1
its semantic rank: 2
its weighted rrf score: 0.01620
places that match lib.blend.weighted_rrf: 100 of 100

对于 laundry basket with wheels,排名第一的商品是 laundry sorter basket on wheels,它在 BM25 排名中位于第 1 名,在语义检索排名中位于第 2 名,因此得分为 0.25 / (60 + 1) + 0.75 / (60 + 2) = 0.01620。

该示例按这种方式为每个候选商品打分;它的排序与 lib.blend.weighted_rrf 在 100 of 100 处一致。

调整的做法是:在训练查询上尝试一组 α 取值,保留平均 nDCG@10 最高的那个。Bruch 等人(2023)报告了关于调整凸组合(CC)的三项发现:调整后的 CC 无论使用哪种分数归一化,表现都大致相当;它在领域内和领域外都优于 RRF;它的样本效率很高。样本效率高是指 α 在“使用不到 5% 的训练数据”时就已收敛,而这只是远大于本商店的训练集中的一小部分。

对加权和,在三种归一化下分别在 0、0.1、…、1 上调整 α;对 k = 60 的加权 RRF 也同样调整。随后作为调整检验,将每种归一化与 min-max 比较,并将留给测试查询的两种融合与语义检索比较:

tune-blend-weights-on-train.py Python 3.12.15 · bm25s 0.3.16 · numpy 2.5.5 · pandas 3.1.1 · scipy 1.18.1 通过
"""Tunes α on the train queries for the weighted sum under three normalizations and for weighted RRF, then comparesthe two blends that go to test with semantic retrieval. The prose's point rests on each setting's best α and nDCG@10,on the p-values against tuned min-max, and on the two gains over semantic retrieval."""from lib.blend import ALPHAS, NORMALIZATIONS, best_alpha, candidates, scores_by_alpha, weighted_rrf, weighted_sumfrom lib.lexical import LexicalSearchfrom lib.semantic import SemanticSearchfrom lib.significance import comparefrom lib.store import load_products, load_queries, qrelsproducts, queries, judgments = load_products(), load_queries("train"), qrels("train")lexical, semantic = LexicalSearch(products), SemanticSearch()tables = candidates(lexical, semantic, queries)bm25, semantic_run = lexical.run(queries, k=100), semantic.run(queries, k=100)per_query = {}                                               # nDCG@10 per query (rows) and alpha (columns)for name in NORMALIZATIONS:    per_query[name] = scores_by_alpha(lambda alpha: weighted_sum(tables, alpha, name), judgments)per_query["weighted rrf"] = scores_by_alpha(lambda alpha: weighted_rrf(bm25, semantic_run, alpha), judgments)semantic_alone = per_query["min-max"][1.0]                   # alpha = 1 is semantic retrieval in either blendprint("alphas tried:", len(ALPHAS))print("min-max ndcg@10 for alpha 0 to 1:", " ".join(f"{mean:.3f}" for mean in per_query["min-max"].mean()))print(f"semantic alone ndcg@10: {semantic_alone.mean():.3f}")# alpha 0.5 is unweighted RRF, chapter 1's fusionprint(f"unweighted rrf (alpha 0.5) ndcg@10: {per_query['weighted rrf'][0.5].mean():.3f}")best = {name: frame[best_alpha(frame)] for name, frame in per_query.items()}for name, frame in per_query.items():    print(f"{name} best alpha: {best_alpha(frame)}")    print(f"{name} best ndcg@10: {best[name].mean():.3f}")for name in ["theoretical min-max", "z-score", "weighted rrf"]:          # each tuned, against tuned min-max    print(f"{name} against min-max, permutation p: {compare(best[name], best['min-max'])['permutation p']:.3f}")for name in ["min-max", "weighted rrf"]:                                 # the two blends that go to test    gain = compare(best[name], semantic_alone)    print(f"{name} gain over semantic: {gain['mean difference']:.3f}")    print(f"{name} gain over semantic, permutation p: {gain['permutation p']:.3f}")    print(f"{name} gain over semantic, t-test p: {gain['t-test p']:.3f}")
输出python tune-blend-weights-on-train.py
alphas tried: 11
min-max ndcg@10 for alpha 0 to 1: 0.695 0.709 0.736 0.746 0.758 0.771 0.781 0.784 0.777 0.778 0.772
semantic alone ndcg@10: 0.772
unweighted rrf (alpha 0.5) ndcg@10: 0.774
min-max best alpha: 0.7
min-max best ndcg@10: 0.784
theoretical min-max best alpha: 0.9
theoretical min-max best ndcg@10: 0.789
z-score best alpha: 0.7
z-score best ndcg@10: 0.787
weighted rrf best alpha: 0.6
weighted rrf best ndcg@10: 0.787
theoretical min-max against min-max, permutation p: 0.164
z-score against min-max, permutation p: 0.349
weighted rrf against min-max, permutation p: 0.776
min-max gain over semantic: 0.013
min-max gain over semantic, permutation p: 0.177
min-max gain over semantic, t-test p: 0.161
weighted rrf gain over semantic: 0.015
weighted rrf gain over semantic, permutation p: 0.215
weighted rrf gain over semantic, t-test p: 0.205

在相同的查询上,仅用语义检索的得分为 0.772。各设置的最佳 α 和得分如下:

设置 最佳 α 平均 nDCG@10 相对于调整后的 min-max 的置换检验 p 值
加权和,min-max 0.7 0.784 —
加权和,理论 min-max 0.9 0.789 0.164
加权和,z-score 0.7 0.787 0.349
加权 RRF,k = 60 0.6 0.787 0.776

在 0.05 的水平上,训练查询上的检验无法把任何一种替代归一化与 min-max 区分开,这与 Bruch 等人关于归一化几乎无关紧要的发现相符。该检验也无法把加权 RRF 与调整后的 min-max 区分开。Bruch 等人的第二项发现,即加权和优于 RRF,在这里并未显现,而这些查询也无法对此作出定论。

由于网格中包含 α = 1,每个调整后的融合在训练查询上的得分都至少与语义检索相当。然而,即使在选出它们的那批查询上,置换检验在 0.05 的水平上也无法把任一融合的增益与随机波动区分开。加权和的增益为 0.013,置换 p 值为 0.177;加权 RRF 的增益为 0.015,p 值为 0.215。每个增益还是在同一批查询上 11 次尝试中的最佳值,这使它进一步被高估。

工作准则: 在训练查询上用较粗的网格调整一个权重,并将归一化保持为 min-max,除非另一种归一化的优势大到训练查询上的检验能将其与随机波动区分开,即 p 低于 0.05。把相对于最佳单一检索器的训练增益视为偏乐观的数值,并预期它在测试查询上会缩小;对任何从多个候选中挑出最佳者的设置,这都是一条经验法则。在信任该设置之前,先检查它是否稳定,再在测试查询上报告它。

检查调优后的权重是否稳定

最近一次修改见于版次 2026.11.10

调优后的权重来自几十条查询,因此可能取决于你手头恰好有哪些查询。自助法(bootstrap)可以说明这种依赖有多大:对训练查询重采样,在每个样本上调优 α,并重复多次。如果选出的 α 来回跳动,说明调优值反映的更多是样本,而不是商店本身。第 3 章使用同样的重采样方法,为评分给出区间。

用固定的随机种子对训练查询重采样 1000 次,并在相同的重采样样本上,统计两种调优混合方法(min-max 分数加权和与加权 RRF)中各个 α 获胜的次数:

blend-weight-bootstrap.py Python 3.12.15 · bm25s 0.3.16 · numpy 2.5.5 · pandas 3.1.1 · scipy 1.18.1 通过
"""Resamples the train queries with a fixed seed and counts which α wins each resample, for the weighted sum and forweighted RRF on the same resamples. The prose's point rests on the α tuned on all train queries and on the counts ofresamples choosing each α."""import numpy as npimport pandas as pdfrom lib.blend import best_alpha, candidates, scores_by_alpha, weighted_rrf, weighted_sumfrom lib.lexical import LexicalSearchfrom lib.semantic import SemanticSearchfrom lib.store import load_products, load_queries, qrelsproducts, queries, judgments = load_products(), load_queries("train"), qrels("train")lexical, semantic = LexicalSearch(products), SemanticSearch()tables = candidates(lexical, semantic, queries)bm25, semantic_run = lexical.run(queries, k=100), semantic.run(queries, k=100)blends = {"weighted sum": scores_by_alpha(lambda alpha: weighted_sum(tables, alpha), judgments),          "weighted rrf": scores_by_alpha(lambda alpha: weighted_rrf(bm25, semantic_run, alpha), judgments)}resamples = 1000print("resamples:", resamples)# a fixed seed, so every run draws the same resamplesrng = np.random.default_rng(2026)draws = [rng.integers(0, len(queries), len(queries)) for _ in range(resamples)]   # the same draws for bothfor name, per_query in blends.items():                       # per_query: train queries x alphas    print(f"{name} alpha tuned on all train queries: {best_alpha(per_query)}")    # best alpha per resample; idxmax takes the first, smaller alpha when two tie    chosen = pd.Series([per_query.iloc[draw].mean().idxmax() for draw in draws])    for alpha, count in chosen.value_counts().sort_index().items():        print(f"{name} resamples choosing alpha {alpha}: {count}")
输出python blend-weight-bootstrap.py
resamples: 1000
weighted sum alpha tuned on all train queries: 0.7
weighted sum resamples choosing alpha 0.5: 13
weighted sum resamples choosing alpha 0.6: 171
weighted sum resamples choosing alpha 0.7: 686
weighted sum resamples choosing alpha 0.8: 11
weighted sum resamples choosing alpha 0.9: 74
weighted sum resamples choosing alpha 1.0: 45
weighted rrf alpha tuned on all train queries: 0.6
weighted rrf resamples choosing alpha 0.5: 2
weighted rrf resamples choosing alpha 0.6: 513
weighted rrf resamples choosing alpha 0.7: 76
weighted rrf resamples choosing alpha 0.8: 365
weighted rrf resamples choosing alpha 0.9: 26
weighted rrf resamples choosing alpha 1.0: 18

加权和的权重保持稳定。在全部训练查询上,它的 α 为 0.7,α = 0.7 在 686 次重采样中胜出,多于其他任何取值。α = 0.6 在 171 次重采样中胜出,单独使用语义检索(α = 1)则在 45 次中胜出。

加权 RRF 在 α = 0.6 处调优,稳定性较差:它的权重在 0.6 到 0.8 之间移动,α = 0.6 在 513 次重采样中胜出,α = 0.8 胜出 365 次,α = 0.7 胜出 76 次。

除了等权重 α = 0.5 之外,两个权重都落在网格中偏向语义检索的一侧;α = 0.5 在加权和的重采样中胜出 13 次,在加权 RRF 的重采样中胜出 2 次。

重采样并不是唯一的检验。BM25 分数相等的商品需要有某种排序,而 lib.lexical 按 product_id 排列它们,这种顺序不携带任何信息。按 product_id 降序排列同样不携带信息。加权和对并列商品使用相同的 BM25 项,因此它们的顺序对加权和几乎没有影响;但 RRF 会给它们连续的名次,因而得到不同的分数。现在按 product_id 降序列出 BM25 的并列项,并在训练查询上重新调优最佳 RRF k和两个权重:

tuning-with-ties-reversed.py Python 3.12.15 · bm25s 0.3.16 · numpy 2.5.5 · pandas 3.1.1 · scipy 1.18.1 通过
"""Tunes RRF's k and both blends' α again with BM25's equal scores listed in descending product_id order, to testwhether the order of ties matters. The prose's point rests on each setting's pair of lines, ties by product_id andties reversed."""import numpy as npfrom lib.blend import best_alpha, candidates, scores_by_alpha, weighted_rrf, weighted_sumfrom lib.fusion import reciprocal_rank_fusionfrom lib.lexical import LexicalSearchfrom lib.metrics import evaluatefrom lib.semantic import SemanticSearchfrom lib.store import load_products, load_queries, qrelsclass TiesReversed(LexicalSearch):    """BM25 with equal scores in descending product_id order, an order as arbitrary as ascending."""    def search(self, query, k=100):        scores = super().search(query, k=len(self.ids))           # every match, so the cut at k can move too        # highest score first, then descending product_id (np.lexsort sorts by its last key first)        order = np.lexsort((-scores.index.to_numpy(), -scores.to_numpy()))        return scores.iloc[order[:k]]products, queries, judgments = load_products(), load_queries("train"), qrels("train")semantic = SemanticSearch()semantic_run = semantic.run(queries, k=100)# each order runs in turn, against the same semantic runfor ties, lexical in [("ties by product_id", LexicalSearch(products)), ("ties reversed", TiesReversed(products))]:    bm25 = lexical.run(queries, k=100)    by_k = {k: evaluate(reciprocal_rank_fusion([bm25, semantic_run], k=k), judgments)["ndcg@10"].mean()            for k in [1, 10, 30, 60, 100, 300]}    print(f"best rrf k, {ties}: {max(by_k, key=by_k.get)}")                # ties go to the smaller k    rrf = scores_by_alpha(lambda alpha: weighted_rrf(bm25, semantic_run, alpha), judgments)    print(f"weighted rrf alpha, {ties}: {best_alpha(rrf)}")    tables = candidates(lexical, semantic, queries)    total = scores_by_alpha(lambda alpha: weighted_sum(tables, alpha), judgments)    print(f"weighted sum alpha, {ties}: {best_alpha(total)}")
输出python tuning-with-ties-reversed.py
best rrf k, ties by product_id: 100
weighted rrf alpha, ties by product_id: 0.6
weighted sum alpha, ties by product_id: 0.7
best rrf k, ties reversed: 300
weighted rrf alpha, ties reversed: 0.8
weighted sum alpha, ties reversed: 0.7

反转并列项的顺序会改变 RRF 的设置,但不会改变加权和的设置:

设置 并列项按 product_id 排序 并列项反转
最佳 RRF k 100 300
加权 RRF α 0.6 0.8
加权和 α 0.7 0.7

当一种不携带信息的顺序就能改变某项设置时,说明训练查询无法区分这些设置,所以 RRF 的 k 和权重是脆弱的,而加权和的权重不是。商店没有更多的训练查询可用来确定加权 RRF 的权重,而它的测试对比已按并列项按 product_id 排序时调优出的权重规划好,因此送去测试的是 α = 0.6。

稳定性的含义是训练查询对某个权重意见一致,并不表示它能改进其他查询上的搜索:重采样反复使用同一批查询,因此这些查询的共同之处也被样本共享。

工作准则: 对训练查询做自助法重采样,并在每个样本上调优;然后再用另一种不携带信息的顺序列出并列项,重新调优一次。如果选出的权重在网格上分散,就使用更粗的网格或更多的训练查询,或者选择更简单的设置。如果仅仅是并列项的顺序就让它发生变化,就在报告它的每一处说明这一点。如果它在两项检验下都保持稳定,就把它送去测试查询。

在测试查询上报告调参后的融合方法

最近一次修改见于版次 2026.11.8

只有留出的查询才能说明,调参得到的设置在选出它的那些查询之外是否仍然有效。把每个调参后的融合方法在测试查询上各运行一次,并使用第 3 章的评测框架,逐查询与语义检索(最佳的单一检索器)比较。本章在查看结果之前就确定了这两项比较。依照第 3 章,Bonferroni 校正把门槛定为置换检验的 p 值低于 0.025,即 0.05 除以两项计划内的比较。t 检验和置信水平相应的 bootstrap 区间(0.975)作为参照。

在商店的训练查询上,调参后的融合方法相对语义检索分别提高了 0.013 和 0.015,而置换检验无法把这两个提升与随机波动区分开。请先做出预测:在测试查询上,α = 0.7 的加权求和与 α = 0.6 的加权 RRF,得分会高于语义检索、与之持平,还是低于它?

tuned-blends-on-test.py Python 3.12.15 · bm25s 0.3.16 · numpy 2.5.5 · pandas 3.1.1 · scipy 1.18.1 通过
"""Chooses each blend's α on the train queries, runs both once on the test queries, and compares each with semanticretrieval at the corrected threshold. The prose's point rests on the two mean differences and their permutationp-values, and the assert checks that neither clears the threshold."""from lib.blend import best_alpha, candidates, scores_by_alpha, weighted_rrf, weighted_sumfrom lib.lexical import LexicalSearchfrom lib.metrics import evaluatefrom lib.semantic import SemanticSearchfrom lib.significance import comparefrom lib.store import load_products, load_queries, qrelslexical, semantic = LexicalSearch(load_products()), SemanticSearch()def blends(queries):    tables, bm25, semantic_run = candidates(lexical, semantic, queries), lexical.run(queries), semantic.run(queries)    return {"weighted sum": lambda alpha: weighted_sum(tables, alpha),          # each blend's run at a given alpha            "weighted rrf": lambda alpha: weighted_rrf(bm25, semantic_run, alpha)}train, test = blends(load_queries("train")), blends(load_queries("test"))threshold = 0.05 / len(test)                                 # two planned comparisons, Bonferroni-correctedsemantic_scores = evaluate(semantic.run(load_queries("test")), qrels("test"))print("test queries:", len(semantic_scores))print(f"semantic ndcg@10: {semantic_scores['ndcg@10'].mean():.3f}")unweighted = evaluate(test["weighted rrf"](0.5), qrels("test"))          # alpha 0.5 is chapter 1's fusionprint(f"unweighted rrf ndcg@10: {unweighted['ndcg@10'].mean():.3f}")print(f"p-value threshold for each comparison: {threshold:.3f}")print(f"confidence level of each interval: {1 - threshold:.3f}")judged = [semantic_scores["judged@10"].mean()]for name, blend in test.items():    alpha = best_alpha(scores_by_alpha(train[name], qrels("train")))       # train only    scores = evaluate(blend(alpha), qrels("test"))    judged.append(scores["judged@10"].mean())    # the interval's level matches the corrected threshold    result = compare(scores["ndcg@10"], semantic_scores["ndcg@10"], confidence=1 - threshold)    print(f"{name} alpha chosen on train: {alpha}")    print(f"{name} ndcg@10: {scores['ndcg@10'].mean():.3f}")    print(f"{name} minus semantic: {result['mean difference']:.3f}")    print(f"{name} permutation p: {result['permutation p']:.3f}")    print(f"{name} t-test p: {result['t-test p']:.3f}")    print(f"{name} interval: {result['ci low']:.3f} to {result['ci high']:.3f}")    # neither difference is established: each permutation p-value stays at or above the corrected threshold    assert result['permutation p'] >= thresholdprint(f"lowest judged@10: {min(judged):.3f}")print(f"highest judged@10: {max(judged):.3f}")
输出python tuned-blends-on-test.py
test queries: 30
semantic ndcg@10: 0.830
unweighted rrf ndcg@10: 0.812
p-value threshold for each comparison: 0.025
confidence level of each interval: 0.975
weighted sum alpha chosen on train: 0.7
weighted sum ndcg@10: 0.816
weighted sum minus semantic: -0.014
weighted sum permutation p: 0.229
weighted sum t-test p: 0.221
weighted sum interval: -0.042 to 0.008
weighted rrf alpha chosen on train: 0.6
weighted rrf ndcg@10: 0.825
weighted rrf minus semantic: -0.005
weighted rrf permutation p: 0.650
weighted rrf t-test p: 0.643
weighted rrf interval: -0.029 to 0.016
lowest judged@10: 0.980
highest judged@10: 0.990

两种融合方法都没有超过语义检索。语义检索的得分为 0.830,min-max 分数的加权求和为 0.816,加权 RRF 为 0.825。

这些运行结果的 judged@10 取值介于 0.980 与 0.990 之间,因此缺失的判定对三者的影响大致相当。

两项差异都不成立:每个置换检验的 p 值都不低于校正后的门槛 0.025。加权求和与语义检索相差 -0.014,置换检验 p 值为 0.229,t 检验 p 值为 0.221,区间为 -0.042 to 0.008。加权 RRF 相差 -0.005,两个 p 值分别为 0.650 和 0.643,区间为 -0.029 to 0.016。两个区间都包含 0,检验结果与之一致,所以结论是“没有改进”,而不是“更差”。对加权 RRF 而言,这一结论针对的是 α = 0.6,而这个权重可能因 BM25 并列项的顺序而改变。

若把结果说成“高于”,就等于相信了训练集上的提升。即使在选出该设置的那些查询上,置换检验也无法把这一提升与随机波动区分开,而且它只是多次尝试中最好的一次。

工作准则: 只有当融合方法在留出查询上胜过最佳的单一检索器,且置换检验的 p 值低于按计划内比较次数校正后的门槛时,才采用它。如果做不到,就保留较简单的系统。这样,语义检索单独成为商店中需要被超越的排序器。请保留调参得到的权重和代码,并在检索器或目录发生变化时重新测试这些融合方法。

这部分尚未翻译,因此以英文显示。

Learn sparse term weights with SPLADE

最近一次修改见于版次 2026.11.10

Instead of fusing two retrievers, you can change one so that it misses less. BM25 only matches words that appear in a product's text. Learned sparse retrieval keeps BM25's machinery, an inverted index of terms, but has a neural model choose each product's terms and their weights. SPLADE is one learned sparse retrieval model (Formal et al., 2021). A language model of the BERT family, a transformer that reads text as subword tokens, reads the product's text and, at each token i, gives every term j in its vocabulary a raw score w_ij. ReLU sets the negative scores to 0, and the original paper sums a logarithm of what's left over the text's tokens:

weight of term j = sum over the text's tokens i of  log(1 + ReLU(w_ij))

The logarithm is a saturation, like BM25's: in the authors' words, it "prevents some terms to dominate and naturally ensures sparsity in representations". SPLADE v2 replaced the sum with the maximum over the tokens, and the authors made max pooling their default (Formal et al., 2021, SPLADE v2). A SPLADE vector has one dimension per vocabulary entry, 30,522 for a BERT vocabulary, and typically more than 99% of them are zero (Sentence Transformers). The nonzero terms of a SPLADE vector include expansion terms: words the text doesn't contain but the model weights anyway, so a product can match a query word it never uses.

Expansion terms target one of BM25's failures in the store. Count, on the train queries, how many exact products lack at least one of the query's words, as BM25's tokenizer sees them:

exact-products-missing-query-words.py Python 3.12.15 · bm25s 0.3.16 · numpy 2.5.5 · pandas 3.1.1 · scipy 1.18.1 通过
"""Counts the exact train pairs whose product lacks a word of its query, as BM25's tokenizer sees them, and counts theexact products of one query by term. The prose's point rests on the share missing a query word and on the per-termcounts for midcentury tv unit."""from collections import Counterimport bm25sfrom lib.store import load_products, load_queries, product_text, qrelsproducts, queries, judgments = load_products(), load_queries("train"), qrels("train")def words(texts):    """The tokens BM25 matches on: bm25s's default tokenizer with English stopwords, as in lib.lexical."""    tokenized = bm25s.tokenize(texts, stopwords="en", return_ids=False, show_progress=False)    return [set(tokens) for tokens in tokenized]product_words = dict(zip(products.index, words(product_text(products).tolist())))query_words = dict(zip(queries.index, words(queries["query"].tolist())))exact_pairs = [(query_id, product_id) for query_id in queries.index               for product_id, grade in judgments[query_id].items() if grade == 2]missed = Counter(word for query_id, product_id in exact_pairs                 for word in query_words[query_id] - product_words[product_id])# a pair counts as missing when some query word is absent from the product's set of tokensmissing = sum(not query_words[query_id] <= product_words[product_id] for query_id, product_id in exact_pairs)print("exact train pairs:", len(exact_pairs))print("exact pairs missing a query word:", missing)print(f"share missing a query word: {missing / len(exact_pairs):.3f}")print("most often missing:", ", ".join(f"{word} ({count})" for word, count in missed.most_common(8)))exact = [product_id for query_id, product_id in exact_pairs if query_id == 453]   # midcentury tv unitprint("exact products for midcentury tv unit:", len(exact))for word in ["midcentury", "unit", "stand"]:    print(f"exact products containing {word}:", sum(word in product_words[product_id] for product_id in exact))mid_century = product_text(products)[exact].str.contains("mid-century")print("exact products whose text says mid-century:", mid_century.sum())
输出python exact-products-missing-query-words.py
exact train pairs: 1937
exact pairs missing a query word: 1454
share missing a query word: 0.751
most often missing: ligth (193), decorative (134), modern (102), freestanding (78), floats (58), queen (57), dining (56), infant (43)
exact products for midcentury tv unit: 25
exact products containing midcentury: 0
exact products containing unit: 0
exact products containing stand: 25
exact products whose text says mid-century: 20

Of 1937 exact (query, product) pairs, 1454 miss at least one query word, a share of 0.751. The words missed most often, with their counts, are ligth (193), decorative (134), modern (102), freestanding (78), floats (58), queen (57), dining (56), infant (43). Some are typos, which chapter 8's corrector targets, and plural forms, which chapter 5 stems; others are descriptive words the product text leaves out, or different words for one thing.

For midcentury tv unit, the exact products use other words. Of the 25 exact products, 0 contain midcentury and 0 contain unit. Their text says mid-century instead in 20 and stand in 25.

Expansion terms are aimed at gaps like these, though whether a model learns that a TV stand is a TV unit is something you'd measure.

The SPLADE authors' own technical report gives SPLADE-v3 an MRR@10 above 0.40 on the MS MARCO development set, a passage-ranking benchmark where MRR@10 measures how high the first relevant passage lands, and describes it as "comparing well to cross-encoder re-rankers" (Lassance et al., 2024). The book doesn't run SPLADE. The reference checkpoints, naver/splade-cocondenser-ensembledistil and naver/splade-v3, are licensed CC-BY-NC-SA-4.0, which rules out commercial use. An Apache-2.0 model, opensearch-neural-sparse-encoding-doc-v3-distill, runs its neural model on products only, and encodes queries with a tokenizer and a table of weights.

Working rule: Consider learned sparse retrieval when BM25's misses are mostly vocabulary misses and you want to keep an inverted index. Check the model's licence first, and measure it like any retriever: alone, per query, on train queries, then on test.

用 ColBERT 逐词元匹配

最近一次修改见于版次 2026.11.10

SPLADE 改变的是词汇检索,延迟交互(late interaction)改变的是语义检索。商店的语义检索是单向量的:每个商品一个嵌入,每个查询一个嵌入,只比较一次。延迟交互为每个词元保留一个向量,而不是为每段文本保留一个向量,因此也称为多向量检索。ColBERT 像嵌入模型一样,分别对查询和商品编码,只在最后才将二者比较。对于查询中的每个词元,ColBERT 找出与之最相似的商品词元,再把这些最大值相加,这一运算称为 MaxSim(Khattab and Zaharia, 2020):

score(query, product) = sum over query tokens q of  max over product tokens d of  cosine(q, d)

由于商品是在不带查询的情况下编码的,它们的词元向量可以提前算好,而交叉编码器做不到这一点:它把两段文本作为一个输入来读取。ColBERT 论文报告称,其效果“与现有基于 BERT 的模型相当”,这些模型把查询和段落放在一起读取,而速度“快两个数量级”。代价是存储:每个词元一个向量,而不是每个商品一个向量。ColBERTv2 对这些向量进行压缩,使延迟交互模型所需的空间减少 6 到 10 倍(Santhanam et al., 2022)。

回到 midcentury tv unit,其确切的商品写的是 stand,从不写 unit。单个向量把整个查询折叠成一个方向,把整段商品文本折叠成另一个方向。采用延迟交互时,查询中的 unit 会与每个商品词元比较,其中包括 stand,其最佳匹配单独计入得分。词元级匹配对一些困难情形有帮助。NevIR 是一个否定基准,随机选择的得分为 25%,ColBERTv1 得分为 19.7%,ColBERTv2 为 13.0%,高于所有受测的单向量模型(12% 或更低),但仍低于随机水平(Weller et al., 2024)。BEIR 是一个以零样本方式测试模型的基准,使用模型未经训练的数据集;在其中,重排序器和延迟交互模型“平均取得最佳的零样本表现”,但计算成本很高(Thakur et al., 2021)。已有开源工具,例如 PyLate 库和 Apache-2.0 许可的模型 lightonai/GTE-ModernColBERT-v1(Chaffin and Sourty, 2025)。本书也没有运行 ColBERT。

工作准则: 当单向量检索漏掉取决于某一个词的细节,而且你负担得起每个词元一个向量的存储时,可以考虑延迟交互。对于与商店规模相当的目录,我们建议先在候选短名单上测试重排序器(第 9 章):它在查询时读取同样的细节,且不需要新的索引,不过第 9 章中的重排序器都没有显示出相对于商店语义检索的确定提升。

判断混合检索何时值得使用

最近一次修改见于版次 2026.11.8

融合是否有帮助,取决于被融合的检索器,已发表的研究结论也并不一致。词法检索与稠密检索(稠密指基于嵌入)即使在稠密模型较弱时也可以互补:在领域之外,零样本混合方法在三个数据集上平均比深度检索模型相对提升了 20.4%(T. Chen et al., 2022)。词法基线很重要。稠密检索器 DPR 的作者发现混合方法收益甚微,而用更强的 BM25 做的复现则发现了“统计上显著的提升”(Ma et al., 2021)。语义检索器的强度同样重要。BRIGHT 是一个以推理密集型查询为主的基准,在该基准上,以 α = 0.5 对 min-max 分数做加权求和来融合 BM25,使中档稠密检索器的 nDCG@10 提高了 3.5 到 4.9 个点,却使最强的两个检索器下降了 2.0 到 2.9 个点。作者提醒,融合可能损害本已很强的检索器,因为“融合后的列表会提升词法上相似但语义上无关的文档”(Abdallah et al., 2026)。

生产环境的报告通常是将混合方法与单纯的词法搜索相比较。一家零售商的预印本报告称,在线 A/B 测试中,词法与向量混合搜索优于单纯的词法搜索,点击率提高了 0.97%,转化率提高了 0.98%(Sonagara et al., 2026)。这说明的是在词法系统中加入语义检索有益,而不是在强语义检索中加入词法检索有益,后者才是商店面对的问题。

商店属于强检索器的情形:在测试查询上,第 3 章确立了语义检索相对 BM25 的优势,而且经过调优的两种混合方式都没有超过它。商店的 BM25 是本书未做词干提取的基线。第 5 章构建了一个更强的基线,Ma 等人的复现也给出了理由,值得用它重新测试这些混合方式。

融合分数并不是混合的唯一形式。搜索引擎还可以把词法匹配用作语义检索的过滤器:有一种引擎在其查询语言中加入了嵌入运算符,使“由嵌入与词项任意组合而成的混合检索表达式”可以作为一个查询运行(Huang et al., 2020)。商店的过滤器可以要求 48 这样的尺寸精确匹配,再对匹配结果按嵌入排序。这是待检验的假设,而不是结论,而且朴素的过滤器会在 7qt slow cooker 上失败,因为没有任何商品使用这种写法。判断一个查询是否带有这类约束,属于查询理解(第 8 章),第 8 章会在购物者提到颜色、形状、尺寸或表面处理的后续消息上尝试这样的过滤器。为每个查询选择权重同样属于查询理解,本书不会构建它。为每个训练查询在两种排序中挑选更好的一个,得分将是 0.802,而按查询设置权重会混合两个列表,不受这个数字的限制。一种按查询设置权重的方法,是请 LLM(大语言模型)评判每个检索器的首位结果,并据此设定 α(Hsu and Tzeng, 2025)。BRIGHT 研究发现它的一种基于分数的变体并不稳定(Abdallah et al., 2026)。

工作准则: 我们建议在以下情况尝试混合方法:两个检索器的质量接近,嵌入模型在其训练领域之外工作,或者许多查询依赖嵌入所遗漏的精确词项。先从加权 RRF 开始:它只需要排名,而且在商店的训练查询上,置换检验无法区分它与加权求和。当两个引擎都为每个候选给出分数时,改用归一化分数的加权求和:有两项研究发现它在各自的数据上优于 RRF(Bruch et al., 2023;Abdallah et al., 2026),而且在商店的训练查询上,它的权重更为稳定。当某个检索器在留出查询上被确认更强,且没有任何调优后的混合方式在其上胜出时,就单独使用它,把另一个保留为基线,并测试针对精确约束的词法过滤器。这正是商店的情形。

生产环境中会出什么问题

最近一次修改见于版次 2026.11.8

两个检索器意味着一切都要付出两倍的成本。 示例是在离线状态下对目录的一个样本打分,而在线的混合搜索需要同时维护两个索引的更新,并且每次查询要执行两次搜索。两者可以并行运行,但较慢的一方决定响应时间,任何一方出现故障都会使结果变差。请把这笔成本与你测得的收益放在一起衡量。

过短的列表会丢失候选项。 融合无法为两个检索器都没有返回的商品排序。在该商店的训练查询上,两个前 100 名列表的并集包含了每个查询精确商品的 0.810,两个前 20 名列表的并集则包含 0.445。某个引擎配置中的深度限制可能在无人察觉的情况下裁掉候选项。

缺失的分数需要一条规则。 当每个检索器只返回各自的前 k 个结果时,未出现在某个列表中的商品,在该检索器那里没有分数。DPR 复现实验为它赋予“已检索文档中的最低稠密检索分数”,对 BM25 则反过来处理(Ma et al., 2021)。Bruch 等人则像 lib.blend.candidates 那样精确计算缺失的分数,而生产环境中的引擎可能不允许你这样做(Bruch et al., 2023)。

最小-最大归一化是相对的。 对一个查询而言,最好的候选项总是得到 1,即使所有候选项都很差。如果一个查询与目录中任何商品都没有共同的词,每个候选项的 BM25 分数都是 0。此时普通的最小-最大归一化会除以零,而 lib.blend 返回全零,于是这个混合检索会悄然退化为单纯的语义检索。该商店的查询中没有出现这种情况,但线上的搜索框迟早会遇到。

融合分数不是置信度。 检索分数的校准很差:“各类模型的置信度校准始终较弱”(Abdallah et al., 2026)。不要用融合分数设定阈值,来判断某个查询没有好的结果。

调好的权重只属于某一对检索器。 该商店的 α 适配的是这些 BM25 设置、这个嵌入模型和这份目录。其中任何一项发生变化,都应先在训练查询上重新调参,再决定是否信任旧权重。调优后的 RRF“在域外数据集上的泛化能力较差”(Bruch et al., 2023)。

工作准则: 在融合结果旁记录两个检索器各自的排名,检查每个引擎配置中的列表深度,并在任一检索器变化时重新调整 α。

本章做出的决定

最近一次修改见于版次 2026.11.8

此后的任何改动,都必须有理由胜过下面每一项决定。

  • 在训练集上调参,在测试集上报告。 所有分析和所有调好的设置,都使用 50 条训练查询。本章只读取一次测试查询,用于报告。
  • 两项事先计划的比较: 每个调好的混合方法各自与语义检索比较,由置换检验判定,显著性水平为 0.05 除以 2,并同时给出 t 检验和自助法(bootstrap)区间。
  • 融合之前先检查每个检索器, 使用 judged@10 和配对检验,并按查询统计各个检索器在哪些查询上胜出。
  • 候选集:两个检索器各自前 100 名的并集, 对每个候选都查出两个分数,因此没有任何商品的分数缺失。
  • 归一化:逐查询的 min-max。 在训练集上,置换检验在 0.05 的水平下无法区分另外两种归一化与它的差别,而且 min-max 不需要了解任何一个打分函数。
  • RRF:k = 60,列表深度为 100。 训练集上最优的 k 相比 60 并无增益,置换检验在 0.05 的水平下无法把这点差别与随机波动区分开,而且它会随 BM25 并列项的顺序而变化。更深的列表只会增加候选。
  • 一个权重 α,在 0 到 1 之间以 0.1 为步长调参, 两种混合方法都如此,依据训练集上的平均 nDCG@10 选定,然后做自助法检验,并在把 BM25 的并列项顺序反转后再次调参。加权和的 α 在两项检查下都成立,加权 RRF 的 α 则不成立。
  • SPLADE 和 ColBERT 只作介绍,不做记录, 因为回答商店的问题并不需要它们。
  • 只有当混合方法在测试集上胜过最好的单个检索器时,才采用它。 两个调好的混合方法都没有做到,因此商店要超越的排序器仍然是单独的语义检索,而未做词干提取的 BM25 则保留为基线。
  • 本章所显示的: 只有当各检索器在不同的查询上失败时,混合才可能有帮助;在训练集上获胜的权重必须在测试集上同样成立。在这家商店,两个调好的混合方法都没有胜过语义检索。

本章,在本版次中

版次 2026.11.10示例于 2026年11月19日 通过本章 12个示例 个示例中有 12 个通过

译自英文版图书,版次 2026.11.10。

引用本节

格式

每种格式均引用英文原书,本译本以其为准。

每种格式都会注明该版次。其链接会打开当前版次,节标题处会说明该节文本最后一次修改的时间。访问日期和检索日期即为您打开此页的当天。

本页内容

第 7 章,共 11 章: 混合检索

  1. Find the queries each retriever wins
  2. 融合之前先检查每个检索器
  3. 先归一化分数再相加
  4. Test reciprocal rank fusion's settings on train queries
  5. 在训练查询上调整融合权重
  6. 检查调优后的权重是否稳定
  7. 在测试查询上报告调参后的融合方法
  8. Learn sparse term weights with SPLADE
  9. 用 ColBERT 逐词元匹配
  10. 判断混合检索何时值得使用
  11. 生产环境中会出什么问题
  12. 本章做出的决定

返回顶部全部章节