端到端的搜索
阅读约需 77分钟 分钟,10节中含11个示例。
读完本章,你可以为商品目录搭建一条粗糙但完整的搜索流水线,从购物者的查询一直到排好序的商品列表,并且可以用一个数字衡量对这条流水线所做的改动是否有帮助。后面的每一章都会选取这条流水线的一个部分,把它做扎实。
一家线上家具和家居用品商店的购物者,会在搜索框中输入诸如 clear shower curtain 或 48 inch kitchen hood 这样的查询。本书全程围绕这家商店展开,并称之为“商店”。商店需要两个答案:它的搜索到底有多好,以及某项改动是否让搜索变得更好。无论面对哪种搜索系统,你都会遇到同样的两个问题,所以本书在商店上尝试每一种方法,让你能看到每种方法对真实查询产生的影响。
商店具备回答这两个问题所需的条件:一个商品目录、一组真实的查询,以及人们对哪些商品与哪条查询相匹配所做的判断。因此在本章中,你将用三种方式搜索商品目录(匹配词语、匹配含义,以及将两者结合),用一个更慢的模型对最佳结果重新排序,并依据这些判断为每个版本打分。并非每个阶段都能提高分数。这正是本章的主要教训:对每一项改动都要进行度量。
这部分尚未翻译,因此以英文显示。
Load the store's data
Before you can improve a search system, you need three things: the items it searches, the queries people type, and judgments that say which items match which query. Search engines call the items documents. In this book they are products. A judgment is a person's label for one pair of a query and a product, and a grade is the number that stands for a judgment's label. Without judgments, you can look at results and form opinions, but you can't tell a fix from a regression.
The store's data, which the book's examples search and score, is real. It is a sample of WANDS, a public product-search dataset that Wayfair published with a paper at ECIR 2022 (Chen et al., 2022), under the MIT licence (licence). The full dataset has 42,994 products, 480 queries and 233,448 judgments (repository). The queries are a stratified sample of real shopper queries from a quarter's search log in 2021, and each label is the majority vote of up to three trained annotators (Chen et al., sections 4.1 and 5). The script prep/wands_sample.py draws the book's sample, and the sample's data card, examples/data/wands/README.md, says how. Setup code that many examples need lives in the book's shared modules, short files in examples/lib/ that you can open and read. lib.store, one of those modules, loads the sample, and the next example prints what comes back:
1"""Loads the store's sample and prints its size, how its queries split, and how its judgments are labelled. The prose quotes the split from the `train queries` and `test queries` lines."""2import pandas as pd34from lib.store import DATA, load_labels, load_products, load_queries56products = load_products()7queries = load_queries()8labels = load_labels() # one row per judged (query, product) pair9conflicts = pd.read_csv(DATA / "label_conflicts.csv") # pairs WANDS labelled two ways1011print(f"products: {len(products):,}")12print("queries:", len(queries))13print("train queries:", (queries["split"] == "train").sum())14print("test queries:", (queries["split"] == "test").sum())15print(f"judged pairs: {len(labels):,}")16print("pairs with conflicting labels:", len(conflicts))17for label in ["exact", "partial", "irrelevant"]:18 print(f"{label} pairs: {(labels['label'] == label).sum():,}")19print(f"products judged for at least one query: {labels['product_id'].nunique():,}")2021print()22print(queries.head(4).to_string())
python load-the-store.pyproducts: 12,000
queries: 80
train queries: 50
test queries: 30
judged pairs: 13,077
pairs with conflicting labels: 7
exact pairs: 3,252
partial pairs: 7,972
irrelevant pairs: 1,853
products judged for at least one query: 11,917
query query_class split
query_id
6 acrylic clear chair Dining Chairs train
10 king poster bed Beds train
20 foutains with brick look Wallpaper train
36 laundry basket with wheels Hampers & Baskets train
The sample has 12,000 products and 80 queries. It splits the queries once, before any ranker runs: 50 train queries for tuning and 30 test queries for reporting. Every score in this book is reported on the test queries, and nothing is tuned on them. A score on queries you tuned on measures how well you fit those queries, not how well search works.
WANDS judged some pairs more than once. The sample keeps one row per pair, 13,077 in all. When a pair's labels differ, it keeps the lower grade and lists the pair in label_conflicts.csv, which holds 7 pairs. Code that scores a ranking reads the judgments through lib.store.qrels, which returns them as {query_id: {product_id: grade}}. Chapter 2 reads the judgments closely.
Each judgment is one of three labels, defined by the dataset's authors (Chen et al., section 3.2). Exact means the product fully matches the search query. Partial means it "only matches the target entity of the query, but does not satisfy the modifiers for the query." Irrelevant means the product is not relevant to the query. The book's grades are 2, 1 and 0. Of the judged pairs, 3,252 are exact, 7,972 partial and 1,853 irrelevant. In this book, a relevant product is one judged exact for the query, unless a sentence says otherwise.
Three warnings come with the store's data. First, each query's query_class is WANDS's predicted class for it, not a judgment (Chen et al., Table 1): foutains with brick look is classed as Wallpaper. Treat it as a hint. Second, nobody judged every product for every query. That would have taken about 60 million judgments, so WANDS judged candidates gathered from products shoppers clicked or added to their carts and from several search systems, and stopped when few new exact matches turned up (Chen et al., sections 4.2 and 4.3). A product without a judgment for a query is unjudged: unknown, not irrelevant. Third, nearly every product in the sample, 11,917 of 12,000, is judged for at least one of its queries, so the sample holds fewer distractors than a full catalog would: products that compete for a query's top places without matching it.
Working rule: Split your queries into train and test before you change anything. Read test results to understand a ranker, but choose and tune every change on train queries, and report on test. Treat unjudged products as unknown. If you have no judgments yet, collect some before you change the ranking.
用 BM25 对商品排序
搜索商店的商品目录包含两项工作。检索是从整个目录中找出可能与查询匹配的商品。排序是把检索到的商品按顺序排列,最好的排在最前。词法方法通过比较查询中的词与每件商品文本中的词,并对重合部分打分,来完成这两项工作。
BM25 是一种词法打分函数,由三部分组成。商品包含查询中的词时得分更高,词越罕见,加分越多,因为一个词出现在越少的商品中,就越能说明这些商品的特点。同一个词重复出现,所增加的分数越来越少,直至达到上限,BM25 文献称之为饱和。较长的文本相对较短的文本会被打折,因此它不能靠包含更多的词取胜;参数 b 决定折扣的幅度,取值从 0(不打折)到 1(完全打折)(Robertson and Zaragoza, 2009)。第 5 章会手工推导这一公式。
本书的 BM25 运行在 bm25s 0.3.16 上,这是一个用于 BM25 的 Python 库。默认情况下,它使用 Lucene 搜索库所采用的 BM25 变体打分,参数为 k1=1.5(饱和设置)和 b=0.75(bm25s 源码)。它的得分与教科书中的公式略有差异,但一项在三个新闻专线语料集上进行的复现研究发现,BM25 各变体在效果上没有显著差异(Kamphuis et al., 2020)。打分之前,bm25s 会先把文本切分为词元。它会将词元转为小写,保留两个或更多字符的词,去除英文停用词,例如 and,并且不会把词还原为共同的词干(bm25s 分词器)。本书使用的排序器 lib.lexical.LexicalSearch 会为每件商品建立索引,索引内容是商品的名称、类别、品类和描述连接而成的一段文本。
商店的测试查询 clear shower curtain 展示了 BM25 的实际表现。输入这个查询的购物者想要的是透明的帘子,而目录中有许多并不透明的帘子,还有一些含有相同词的配件。下面是 BM25 排名前 10 的结果:
1"""Searches the store's catalog with BM25 for one test query and prints the top 10 products. The prose's point that BM25 matches words rather than things rests on the names at the bottom of the list."""2from lib.lexical import LexicalSearch3from lib.store import load_products, load_queries45products = load_products()6query = load_queries().at[246, "query"] # a test query7lexical = LexicalSearch(products) # BM25 over name, class, category and description89scores = lexical.search(query, k=10) # the 10 highest scores, best first10print("query:", query)11print()12# names cut to 55 characters so that each row fits on one line13for rank, (product_id, score) in enumerate(scores.items(), start=1):14 print(f"{rank:>2} {product_id:>5} {score:5.2f} {products.at[product_id, 'name'][:55]}")
python first-bm25-search.pyquery: clear shower curtain
1 9752 7.11 rodas vinyl shower curtain
2 20428 6.97 woolley vinyl single shower curtain
3 20429 6.75 woolley vinyl solid color single shower curtain
4 20425 6.65 siniard single shower curtain
5 20430 6.48 simmon vinyl single shower curtain liner
6 3564 6.10 71 '' x71 '' 4pcs shower curtain sets beach theme showe
7 20132 5.95 newburg check 3d vinyl single shower curtain + hooks
8 33346 5.76 gilded pineapple shower curtain hooks
9 3197 5.72 double shower curtain hooks
10 1050 5.68 stainless steel shower curtain hooks
这十条结果大多是浴帘或浴帘内衬,但最后几条是浴帘挂钩。BM25 在挂钩的名称中找到了“shower curtain”这两个词,并给它们打出了几乎与浴帘一样高的分数,因为它匹配的是词,而不是事物:它并不知道挂钩不是帘子。这里打印出的内容都没有说明这些帘子是否透明。要知道这一点,需要商店的判断标注。
**工作准则:**以 BM25 的库默认设置作为基线。它不需要训练数据,也不需要模型,出错时你可以看出是哪些词造成的。在你能够用训练查询衡量出改动效果之前,不要调整它的参数。当购物者和商品对同一事物使用不同的词时,再加入语义检索。
对照判定检查结果
在弄清购物者想要什么之前,排序本身意义不大。判定逐对提供这一信息。就这一用途而言,它们比点击更可靠。WANDS 的作者指出:“User behavior log data suffer from positional biases and would favor the rankings similar to the production system”(Chen et al.,第 3.1 节)。购物者会点击最先展示的商品,因此点击大多只是在印证你已有的排序。
接下来,把 BM25 针对 clear shower curtain 返回的前 10 个商品逐一与其判定并排列出。如果某个商品在这条查询下无人判定,则显示为 unjudged:
1"""Puts BM25's top 10 for `clear shower curtain` beside each product's judgment, and counts the labels. The prose's reading of the top 10 rests on the labelled rows and the `in the top 10` lines after them."""2from lib.lexical import LexicalSearch3from lib.store import load_products, load_queries, qrels45LABELS = {2: "exact", 1: "partial", 0: "irrelevant"}67products = load_products()8query_id = 2469query = load_queries().at[query_id, "query"]10judgments = qrels()[query_id] # {product_id: grade} for this query1112top = LexicalSearch(products).search(query, k=10).index13# a product with no judgment for this query is unjudged, not irrelevant14results = [LABELS.get(judgments.get(product_id), "unjudged") for product_id in top]15print("query:", query)16print()17for rank, (product_id, label) in enumerate(zip(top, results), start=1):18 print(f"{rank:>2} {label:<10} {products.at[product_id, 'name'][:55]}")1920print()21for label in ["exact", "partial", "irrelevant", "unjudged"]:22 print(f"{label} in the top 10: {results.count(label)}")23print("products judged for this query:", len(judgments))24print("exact products judged for this query:", sum(grade == 2 for grade in judgments.values()))
python judge-first-results.pyquery: clear shower curtain
1 partial rodas vinyl shower curtain
2 exact woolley vinyl single shower curtain
3 exact woolley vinyl solid color single shower curtain
4 exact siniard single shower curtain
5 exact simmon vinyl single shower curtain liner
6 partial 71 '' x71 '' 4pcs shower curtain sets beach theme showe
7 partial newburg check 3d vinyl single shower curtain + hooks
8 irrelevant gilded pineapple shower curtain hooks
9 irrelevant double shower curtain hooks
10 irrelevant stainless steel shower curtain hooks
exact in the top 10: 4
partial in the top 10: 3
irrelevant in the top 10: 3
unjudged in the top 10: 0
products judged for this query: 253
exact products judged for this query: 12
BM25 的前 10 名中有 4 个完全相关商品、3 个部分相关商品、3 个不相关商品(即挂钩)和 0 个未判定商品。第一个结果属于部分相关:按数据集的定义,它与购物者所要的物品(浴帘)相符,但不符合修饰词 clear(透明)。
判定还能说明最理想的前 10 名应包含什么。评审者为这条查询标注了 253 个商品,其中 12 个为完全相关。因此,理想的排序可以让十个位置全部由完全相关的商品占据,而 BM25 只占了 4 个。指标要衡量的正是这一差距。
以这种方式阅读结果,可以看出排序器犯的是哪一类错误。在这里,BM25 匹配了配件的词语,却漏掉了修饰词。对许多查询取平均会掩盖这一点,而你亲眼看到的错误才是你能修复的错误。
**工作准则:**在计算任何指标之前,先读五到十条查询的靠前结果及其判定。留意其中的规律:配件、漏掉的修饰词、错误的尺寸。当靠前结果中有若干个未判定时,应先让它们得到判定,再信任该查询的得分。
这部分尚未翻译,因此以英文显示。
Score one ranking with nDCG@10
To compare rankings, you need one number per ranking. It should reward matching products, and exact ones more than partial ones, and count a product near the top more than one near the bottom, because shoppers read from the top. Normalized discounted cumulative gain at 10, nDCG@10, does all three.
In words: each of the top 10 results earns a gain equal to its grade: 2 for exact, 1 for partial, 0 for irrelevant. The gain is divided by a discount that grows with position: log2 of the position plus one, so the first result counts in full and the third counts half. Adding up the discounted gains gives the discounted cumulative gain, DCG. Dividing by the DCG of the best possible top 10 for that query, the ideal DCG, puts the score between 0 and 1, where 1 means no better top 10 exists. An unjudged product earns 0.
DCG@10 = sum over positions i from 1 to 10 of grade(i) / log2(i + 1)
nDCG@10 = DCG@10 / ideal DCG@10
The WANDS paper scores search with nDCG@10 too, with gains of 1.0, 0.5 and 0 for exact, partial and irrelevant (Chen et al., section 6). The book's grades are twice those gains, which doubles both DCG and ideal DCG, so nDCG doesn't change.
Take BM25's ranking of clear shower curtain: a partial product first, then exact and partial curtains, then three irrelevant hooks. Compute DCG, the ideal DCG and their ratio by hand, then compare with lib.metrics, the shared module of the book's metrics:
1"""Computes nDCG@10 for one test query step by step, then checks it against `lib.metrics`. The prose's point rests on the `ndcg@10` line and on the library line that prints the same value."""2import math34from lib.lexical import LexicalSearch5from lib.metrics import ndcg_at_k6from lib.store import load_products, load_queries, qrels78products = load_products()9query = load_queries().at[246, "query"] # "clear shower curtain"10judgments = qrels("test")[246] # {product_id: grade}11ranking = list(LexicalSearch(products).search(query, k=10).index)1213grades = [judgments.get(product_id, 0) for product_id in ranking] # unjudged counts as 014# positions count from 1, so the first result is discounted by log2(2) = 115dcg = sum(grade / math.log2(position + 1) for position, grade in enumerate(grades, start=1))16best = sorted(judgments.values(), reverse=True)[:10] # the best possible top 1017ideal = sum(grade / math.log2(position + 1) for position, grade in enumerate(best, start=1))1819print("grades:", grades)20print("best grades:", best)21print(f"dcg@10: {dcg:.3f}")22print(f"ideal dcg@10: {ideal:.3f}")23print(f"ndcg@10: {dcg / ideal:.3f}")24print(f"ndcg@10 from lib.metrics: {ndcg_at_k(ranking, judgments, 10):.3f}")
python ndcg-for-one-query.pygrades: [1, 2, 2, 2, 2, 1, 1, 0, 0, 0]
best grades: [2, 2, 2, 2, 2, 2, 2, 2, 2, 2]
dcg@10: 5.586
ideal dcg@10: 9.087
ndcg@10: 0.615
ndcg@10 from lib.metrics: 0.615
The ranking earns a DCG of 5.586. The best possible top 10 holds only exact products and earns 9.087. Their ratio, 0.615, matches what ndcg_at_k in lib.metrics returns: 0.615. The partial product at position 1 costs the most of any single result, because it holds the place with no discount. Each hook costs less, because it sits lower, but together the hooks cost more.
That one number sums up what the judgments showed: a ranking that found the right kind of product and made two kinds of mistake. You can compute it for any ranking of any judged query, and compare.
Working rule: Use nDCG@10 as your headline metric when your judgments are graded and the first page of results matters most, as it does in a store's search. When you have only yes-or-no labels, or when shoppers look at one result only, another metric fits better. Chapter 3 builds the alternatives and says when to choose each.
为每个测试查询打分
单个查询只是一则轶事。要评判一个排序器,就要在一组固定的查询上为它打分并取平均,本书中这组查询就是测试查询。一组查询中每个查询的排序结果合起来称为一次运行(run),写作 {query_id: [product_id, ...]},最相关的排在最前。lib.metrics.evaluate 会对照相关性判断为一次运行打分,并为每个查询返回一行结果。
除了 nDCG@10,evaluate 还会报告 judged@10:前 10 个结果中,针对该查询有任何判断的产品所占的比例。未经判断的产品按等级 0 计算,所以 judged@10 偏低,就意味着分数有一部分是猜出来的。它无法区分一个不相关的产品和一个相关但无人判断的产品。
现在对每个测试查询运行 BM25,为每个查询保留前 100 个产品,并为这次运行打分:
1"""Scores BM25 on every test query, keeping each query's top 100, and prints the averages and the spread. The prose's point rests on the mean and judged@10 lines and on the three lowest queries printed after them."""2from lib.lexical import LexicalSearch3from lib.metrics import evaluate4from lib.store import load_products, load_queries, qrels56LABELS = {2: "exact", 1: "partial", 0: "irrelevant"}78products, queries, judgments = load_products(), load_queries("test"), qrels("test")9run = LexicalSearch(products).run(queries, k=100) # {query_id: [product_id, ...]}10scores = evaluate(run, judgments) # one row per test query11scores["query"] = queries["query"]1213print("test queries:", len(scores))14print(f"mean ndcg@10: {scores['ndcg@10'].mean():.3f}")15print(f"mean judged@10: {scores['judged@10'].mean():.3f}")16print(f"lowest ndcg@10: {scores['ndcg@10'].min():.3f}")17print(f"median ndcg@10: {scores['ndcg@10'].median():.3f}")18print(f"highest ndcg@10: {scores['ndcg@10'].max():.3f}")19print("queries with ndcg@10 of 1:", (scores["ndcg@10"].round(3) == 1).sum())20print()21lowest = scores.nsmallest(3, "ndcg@10")22print(lowest[["query", "ndcg@10", "judged@10"]].round(3).to_string())23print("lowest query:", lowest["query"].iloc[0])24print(f"judged@10 of the lowest query: {lowest['judged@10'].iloc[0]:.3f}")25print()26# query 133, outdoor light fixtures, is the second-lowest scorer in the table above27print(f"judged@10 for {queries.at[133, 'query']}: {scores.at[133, 'judged@10']:.3f}")28print("bm25 top 5 for", queries.at[133, "query"])29for product_id in run[133][:5]:30 label = LABELS.get(judgments[133].get(product_id), "unjudged")31 print(f" {label:<10} {products.at[product_id, 'name'][:60]}")
python bm25-test-scores.pytest queries: 30
mean ndcg@10: 0.725
mean judged@10: 0.927
lowest ndcg@10: 0.212
median ndcg@10: 0.687
highest ndcg@10: 1.000
queries with ndcg@10 of 1: 4
query ndcg@10 judged@10
query_id
46 beach blue headboard 0.212 0.5
133 outdoor light fixtures 0.286 0.2
292 decorative white pillow 0.467 1.0
lowest query: beach blue headboard
judged@10 of the lowest query: 0.500
judged@10 for outdoor light fixtures: 0.200
bm25 top 5 for outdoor light fixtures
exact malikah 1 -bulb outdoor bulkhead light
unjudged 60 watt equivalent e12 led candle light bulb
unjudged 5w ( 40 watt equivalent ) , b11 led , dimmable light bulb e1
unjudged 60 watt equivalent , a19 led , non-dimmable light bulb , war
unjudged 40 watt equivalent e12 dimmable led edison candle light bulb
在 30 个测试查询上,BM25 的平均 nDCG@10 为 0.725,平均 judged@10 为 0.927。这个平均值是商店第一个测得的结果,此后每个排序器都要与它比较。
平均值掩盖了很大的差异。分数从 0.212 到 1.000,中位数为 0.687,有 4 个查询得到满分 1。
得分最低的查询 beach blue headboard 的 judged@10 为 0.500。另一个得分较低的查询 outdoor light fixtures,其 judged@10 为 0.200:BM25 的前 5 个结果中,有一个确切匹配的户外灯具,随后是几个灯泡,它们与查询共有一些词,但没有人针对这个查询对它们做过判断。灯泡不是灯具,所以这里的等级 0 大概是合理的。但指标无从知道这一点,换一个排序器或换一个查询,未经判断的产品可能恰恰是好结果。
工作准则: 报告测试查询上的平均 nDCG@10 时,要同时给出平均 judged@10,并且每次为一次运行打分时,都要查看得分最低的那些查询。当一个排序器的 judged@10 低于另一个时,先查明原因,再比较它们的 nDCG:差距可能来自缺失的判断,而不是结果更差。
这部分尚未翻译,因此以英文显示。
Search by meaning with embeddings
Lexical retrieval fails when the shopper's words and the product's words differ, or when shared words mean different things, as with the hooks that BM25 ranks for clear shower curtain. Semantic retrieval compares the meanings of the query and the product instead of their words. It rests on embeddings. An embedding is a list of numbers, a vector, that a neural network computes for a piece of text. The network is trained so that texts with similar meanings get vectors pointing in similar directions. You embed every product once, ahead of time. At search time, you embed the query and return the products whose vectors are closest to it. With every vector scaled to length 1, closeness is the dot product of two vectors, which equals the cosine of the angle between them.
Embedding the products ahead of time is what makes this fast. In the paper that introduced Sentence-BERT, finding the most similar pair among 10,000 sentences took about 65 hours on a V100 GPU when BERT read every pair. Embedding each sentence once and comparing the embeddings took about 5 seconds, almost all of it spent computing the embeddings: the comparison itself took about 0.01 seconds (Reimers and Gurevych, 2019).
The store's embeddings come from the open model Snowflake/snowflake-arctic-embed-m-v1.5. Examples never call a model, so that every example runs offline and gives the same output each time. Instead, a recording, here the script recordings/embeddings.py, runs the model once and saves its vectors, and the examples read them offline through lib.recorded. We chose this model because it is open and under the Apache 2.0 licence, so anyone can download the pinned version and rerun the recording. The recording adds to each query the instruction that the model's card asks for ("Represent this sentence for searching relevant passages:") and embeds products as they are (model card). The book's semantic retrieval, built as lib.semantic.SemanticSearch, multiplies every product vector by the query vector and sorts the results. Chapter 6 builds it step by step.
Every product is a vector of 768 numbers, and semantic retrieval compares the query's vector with each one; here are the ten closest, with their judgments:
1"""Ranks the store's products for `clear shower curtain` by the similarity of their recorded embeddings to the query's, and prints the top 10 with each judgment. The prose's point rests on the top 10 counts and the similarity spread line."""2from lib.metrics import ndcg_at_k3from lib.recorded import manifest4from lib.semantic import SemanticSearch5from lib.store import load_products, load_queries, qrels67LABELS = {2: "exact", 1: "partial", 0: "irrelevant"}89products = load_products()10query_id = 24611judgments = qrels()[query_id]12semantic = SemanticSearch() # recorded embeddings of every product and query1314print("embedding model:", manifest("embeddings")["model"])15print("dimensions:", semantic.products.shape[1])16print("query:", load_queries().at[query_id, "query"])17print()18# the query's vector against every product's: unit-length vectors, so each dot product is a cosine19similarity = semantic.search(query_id, k=10)20for rank, (product_id, score) in enumerate(similarity.items(), start=1):21 label = LABELS.get(judgments.get(product_id), "unjudged")22 print(f"{rank:>2} {score:.3f} {label:<10} {products.at[product_id, 'name'][:50]}")2324print()25labels = [LABELS.get(judgments.get(product_id), "unjudged") for product_id in similarity.index]26for label in ["exact", "partial", "irrelevant", "unjudged"]:27 print(f"{label} in the top 10: {labels.count(label)}")28print(f"similarity spread in the top 10: {similarity.max() - similarity.min():.3f}")29print(f"ndcg@10: {ndcg_at_k(list(similarity.index), judgments, 10):.3f}")
python first-semantic-search.pyembedding model: Snowflake/snowflake-arctic-embed-m-v1.5
dimensions: 768
query: clear shower curtain
1 0.577 exact woolley vinyl solid color single shower curtain
2 0.572 exact woolley vinyl single shower curtain
3 0.569 partial newburg check 3d vinyl single shower curtain + hoo
4 0.556 partial rodas vinyl shower curtain
5 0.554 exact siniard single shower curtain
6 0.552 partial 71 '' x71 '' 4pcs shower curtain sets beach theme
7 0.550 exact almazan lightweight single shower curtain
8 0.549 exact simmon vinyl single shower curtain liner
9 0.546 partial vinyl striped shower curtain set
10 0.542 partial cindy window vinyl single shower curtain
exact in the top 10: 5
partial in the top 10: 5
irrelevant in the top 10: 0
unjudged in the top 10: 0
similarity spread in the top 10: 0.035
ndcg@10: 0.793
On this query, semantic retrieval does better than BM25. Its top 10 holds 5 exact products, against BM25's 4, and 0 irrelevant ones. Its nDCG@10 is 0.793, against BM25's 0.615. Notice how close the similarities are: the first and the tenth differ by 0.035. Semantic retrieval always returns something, ranked by small differences.
One query can mislead, so compare the two rankers on every test query:
1"""Scores BM25 and semantic retrieval on every test query and counts the queries each one is better on. The prose's point that the test set is too small to show a pattern rests on the `better` and `size in inches` lines."""2from lib.lexical import LexicalSearch3from lib.metrics import evaluate4from lib.semantic import SemanticSearch5from lib.store import load_products, load_queries, qrels67products, queries, judgments = load_products(), load_queries("test"), qrels("test")8bm25 = evaluate(LexicalSearch(products).run(queries, k=100), judgments)9semantic = evaluate(SemanticSearch().run(queries, k=100), judgments)1011for name, scores in [("bm25", bm25), ("semantic", semantic)]:12 print(f"{name} ndcg@10: {scores['ndcg@10'].mean():.3f}")13 print(f"{name} judged@10: {scores['judged@10'].mean():.3f}")1415# per test query, rounded to three decimals: queries that differ only beyond the third place count as tied16difference = (semantic["ndcg@10"] - bm25["ndcg@10"]).round(3)17print("semantic better:", (difference > 0).sum())18print("bm25 better:", (difference < 0).sum())19print("tied:", (difference == 0).sum())20sized = queries["query"].str.contains(r"\d+\s*(?:inch|in\b|\"|'')") # a size in inches21print("test queries with a size in inches:", sized.sum())22print("of those, bm25 better:", (difference[sized] < 0).sum())23print()24print("queries where bm25 is better:")25print(queries.loc[difference[difference < 0].index, "query"].to_string())
python compare-lexical-semantic.pybm25 ndcg@10: 0.725
bm25 judged@10: 0.927
semantic ndcg@10: 0.830
semantic judged@10: 0.990
semantic better: 21
bm25 better: 7
tied: 2
test queries with a size in inches: 2
of those, bm25 better: 2
queries where bm25 is better:
query_id
63 monthly calendar
72 fold in half table
94 kohler whitehaven farmhouse kitchen sink
285 48" sliding single track , barn door for laundry
292 decorative white pillow
293 wood floor protector mat
482 48 inch kitchen hood
Averaged over the test queries, semantic retrieval is ahead: an nDCG@10 of 0.830 against BM25's 0.725. Its judged@10, 0.990, is also higher than BM25's 0.927, so part of BM25's deficit could come from products nobody judged. For outdoor light fixtures, at least, those products are light bulbs, and a score of 0 is fair.
Semantic retrieval is better on 21 queries, BM25 on 7, and 2 are tied. The store has only 30 test queries, so chapter 3 tests whether a gap like this one is larger than chance.
BM25 is better on 2 of the 2 test queries that give a size in inches. Exact tokens such as sizes and model numbers may favour matching words, but a count this small can't show a pattern. On the train queries that contain a digit, BM25 is better on 1 of 5 (Find the queries each retriever wins), and the embedding model matches digits too, as text (Check queries that contain digits).
A theoretical result shows where single vectors fall short. Weller et al. prove that an embedding's dimension limits which combinations of top results a single vector can return. On LIMIT, a dataset they built to test that limit, BM25 finds 93.6% of the relevant items in its top 100, and the best single-vector model 18.9%. On a variant of LIMIT with words replaced by synonyms, BM25's score drops by more than 89% (Weller et al., 2025). Lexical and semantic retrieval fail in different places.
Working rule: Add semantic retrieval when shoppers describe what they want in their own words. Don't assume lexical retrieval wins on exact terms such as sizes, brands and model numbers: before you rely on it there, count which retriever wins the train queries that contain them. On this data, semantic retrieval alone scores highest so far, so it is the ranker to beat.
用倒数排名融合合并两个排名
词法检索和语义检索的失误出现在不同的地方,因此混合检索把两者各自的排名合并起来,这里指 BM25 的排名和语义检索的排名,希望同时保留两者的长处。它们的分数不能直接相加,因为二者处在不同的尺度上(第 7 章说明了原因)。倒数排名融合(RRF)只使用排名,从而避开了这个问题。对于每个包含某个商品的排名,该商品得到的分数是 1 除以一个常数与其排名之和,各个分数再相加:
RRF(product) = sum over rankings r of 1 / (k + rank of product in r), with k = 60
常数 k 会把头部压平:k = 60 时,排名第 1 得 1/61,排名第 10 得 1/70。因此,在两个排名中都靠近前列的商品,会胜过只在其中一个排名里居于榜首的商品。作者在一项预研中把 k 定为 60,并发现“这一选择并不关键”(Cormack et al., 2009)。他们融合了两类运行结果:提交给 TREC(文本检索会议的共同评测)的结果,以及同一个搜索系统的 30 种配置。在这些实验中,以平均准确率均值(MAP)衡量,RRF 平均比最好的单个运行高出 4% 到 5%;MAP 先对每个查询取各相关结果所在位置上的准确率并求平均,再对所有查询求平均。这与融合一个词法排名和一个语义排名是不同的情形,而且后来一项关于混合搜索的研究发现,经过调优的归一化分数加权和优于 RRF(Bruch et al., 2023)。第 7 章会在训练查询上对两者都进行调优。这里的 RRF 不做调优。
RRF 很短,可以手写成两个函数:rrf_scores 为一个查询的商品打分,reciprocal_rank_fusion 为每个查询排序,分数相同的商品按 product_id 的顺序排列,这样两个排名都不会因为先传入而在平局中占便宜。hybrid-rrf 示例对每个测试查询融合 BM25 和语义检索各自的前 100 个结果,检查 clear shower curtain 排在最前的两个商品的分数,并给混合运行打分。
BM25 的得分为 0.725,语义检索为 0.830,所以较弱的检索器拥有同等的投票权。它的一票会拖低融合结果吗?还是两者能很好地弥补对方的遗漏,从而胜过语义检索?在阅读输出之前,先从三个答案中选定一个:高于语义检索、介于两者之间,或低于 BM25。
1"""Fuses BM25 and semantic retrieval with reciprocal rank fusion, by hand for one query and then for every test query. The prose's point rests on the hybrid's `ndcg@10` line and on the two scores printed for the first product, which agree."""2from lib.lexical import LexicalSearch3from lib.metrics import evaluate4from lib.semantic import SemanticSearch5from lib.store import load_products, load_queries, qrels678def rrf_scores(rankings, k=60):9 """Each product scores the sum of 1 / (k + rank) over the rankings that contain it."""10 scores = {}11 for ranking in rankings:12 for rank, product_id in enumerate(ranking, start=1):13 scores[product_id] = scores.get(product_id, 0) + 1 / (k + rank)14 return scores151617def reciprocal_rank_fusion(runs, k=60, depth=100):18 """For each query, products in order of their RRF scores, best first; equal scores by product_id."""19 fused = {}20 for query_id in runs[0]:21 scores = rrf_scores([run[query_id] for run in runs], k)22 # equal scores fall back to product_id, so neither ranking wins ties by being passed first23 fused[query_id] = sorted(scores, key=lambda product_id: (-scores[product_id], product_id))[:depth]24 return fused252627products, queries, judgments = load_products(), load_queries("test"), qrels("test")28bm25, semantic = LexicalSearch(products).run(queries, k=100), SemanticSearch().run(queries, k=100)29hybrid = reciprocal_rank_fusion([bm25, semantic])3031for place, product_id in zip(["first", "second"], hybrid[246]): # query 246, "clear shower curtain"32 # list.index counts from 0, and a rank counts from 133 bm25_rank, semantic_rank = bm25[246].index(product_id) + 1, semantic[246].index(product_id) + 134 print(f"{place} product: {products.at[product_id, 'name']}")35 print(f"{place} product's bm25 rank: {bm25_rank}")36 print(f"{place} product's semantic rank: {semantic_rank}")37 # the formula with the two ranks in it, where k = 6038 print(f"{place} product's score by hand: {1 / (60 + bm25_rank) + 1 / (60 + semantic_rank):.5f}")39 print(f"{place} product's score from rrf_scores: {rrf_scores([bm25[246], semantic[246]])[product_id]:.5f}")4041hybrid_scores, semantic_scores = evaluate(hybrid, judgments), evaluate(semantic, judgments)42print(f"hybrid ndcg@10: {hybrid_scores['ndcg@10'].mean():.3f}")43print(f"hybrid judged@10: {hybrid_scores['judged@10'].mean():.3f}")44difference = (hybrid_scores["ndcg@10"] - semantic_scores["ndcg@10"]).round(3)45print("better than semantic:", (difference > 0).sum())46print("worse than semantic:", (difference < 0).sum())47print("tied with semantic:", (difference == 0).sum())
python hybrid-rrf.pyfirst product: woolley vinyl solid color single shower curtain
first product's bm25 rank: 3
first product's semantic rank: 1
first product's score by hand: 0.03227
first product's score from rrf_scores: 0.03227
second product: woolley vinyl single shower curtain
second product's bm25 rank: 2
second product's semantic rank: 2
second product's score by hand: 0.03226
second product's score from rrf_scores: 0.03226
hybrid ndcg@10: 0.812
hybrid judged@10: 0.980
better than semantic: 9
worse than semantic: 14
tied with semantic: 7
融合后的第一个商品 woolley vinyl solid color single shower curtain 在 BM25 排名中位于第 3 位,在语义检索排名中位于第 1 位。它的分数由两个排名各贡献一项相加而成,即 1/(60 + 3) + 1/(60 + 1),手算结果为 0.03227,与 rrf_scores 给出的相同。
第二个商品 woolley vinyl single shower curtain 在 BM25 排名中位于第 2 位,在语义检索排名中位于第 2 位,得分为 0.03226,因此排在第二。
混合运行的得分为 0.812:高于 BM25,低于单独使用语义检索。逐个查询来看,它在 9 个查询上胜过语义检索,在 14 个查询上不如语义检索,在 7 个查询上与之持平。它的 judged@10 为 0.980。
因此,融合在这里没有改进语义检索,但这并不等于让搜索变差了。为什么没有帮助,是第 7 章要回答的问题,该章会在融合之前检查每个检索器,并在训练查询上调优混合比例。
如果你选的是高于语义检索,那么这里的教训是:合并两个排名并不保证有收益。融合和其他改动一样,必须经过测量。
工作准则: 融合之前,先单独测量每个检索器。把 k = 60 的 RRF 作为未调优的默认做法,只有当融合后的排名在留出查询上胜过最好的单个检索器时,才保留它。当其中一个检索器明显较弱时,应尝试对排名加权,或在训练查询上调优 k,而不是接受同等的投票权。第 7 章在该商店的数据上尝试了这两种做法,它在测试查询上报告的调优后混合结果,没有一个胜过语义检索(在测试查询上报告调优后的混合结果)。
用交叉编码器重排前 50 个结果
到目前为止,每个阶段都属于检索。检索必须为每个查询给目录中的每件商品打分,因此成本必须低。重排(reranking)则用一个更慢的模型,对检索得到的短列表重新排序。由于只需处理几十件商品,这个模型有余力把查询和商品放在一起看。
一类重排模型是交叉编码器(cross-encoder)。它把查询和商品文本作为同一个输入一起读取,为这一对输出一个相关性分数,每一对都独立打分(Nogueira and Cho, 2019)。嵌入模型只读取商品本身,所以商品向量可以提前算好。交叉编码器需要查询才能计算,因此无法提前算出任何东西,对每一对都运行它会耗费时间:Sentence-BERT 对比中慢的一方,就是 BERT 以交叉编码器的方式读取每一对句子。因此,交叉编码器只重排由成本更低的第一阶段检索出的候选短列表(Nogueira and Cho,第 2 节;Retrieve & Re-Rank)。作为这一成本的回报,它可以通过注意力(attention,即模型中让每个词参考输入中其他词的部分)把查询中的每个词与商品中的每个词联系起来。Reimers and Gurevych(2019,第 4.3 节)指出,BERT 可以利用注意力直接比较两个句子。2019 年,在 MS MARCO 段落基准上,用这种方式对 BM25 的前 1,000 个段落重排,使 MRR@10(衡量第一个相关段落排位高低的指标)相对此前最佳结果提高了 27%(Nogueira and Cho)。
交叉编码器是一种语言模型,但不是大语言模型(large language model,LLM)。LLM 是在大量文本上训练、能根据提示生成文本的模型。后面的章会用 LLM 来评判、改写和重排。交叉编码器不生成任何文本,它只输出一个数。
本店的交叉编码器是开源模型 Alibaba-NLP/gte-reranker-modernbert-base,选择它的理由与嵌入模型相同:它是开源的,采用 Apache 2.0 许可证,任何人都可以重新运行这份录制结果。它的录制脚本 recordings/cross_encoder.py 把模型的分数保存一次,覆盖每个查询下 BM25 与语义检索各自前 100 名的并集中的每件商品,因此混合检索前 50 名中的每件商品都有分数。重排步骤按这些分数,对混合检索结果(用 RRF 融合的 BM25 与语义检索)的前 50 名重新排序,其余部分保持原位,然后给结果打分。本书称这一结果为重排后的混合检索:
1"""Reorders the top 50 of the hybrid run by cross-encoder score, keeps the rest in place, and scores the result. The prose's point rests on the reranked nDCG@10 line, set beside the hybrid's score, and on the better and worse counts."""2from lib.fusion import reciprocal_rank_fusion3from lib.lexical import LexicalSearch4from lib.metrics import evaluate5from lib.recorded import cross_encoder_scores, manifest6from lib.semantic import SemanticSearch7from lib.store import load_products, load_queries, qrels89products, queries, judgments = load_products(), load_queries("test"), qrels("test")10bm25 = LexicalSearch(products).run(queries, k=100)11semantic = SemanticSearch().run(queries, k=100)12hybrid = reciprocal_rank_fusion([bm25, semantic]) # the same function as in the hybrid section1314recorded = cross_encoder_scores() # one score per (query, product) pair15score = {(query_id, product_id): value for query_id, product_id, value in recorded.itertuples(index=False)}16print("cross-encoder model:", manifest("cross-encoder")["model"])1718depth = 50 # rerank the top 5019reranked = {}20for query_id, ranking in hybrid.items():21 # the top 50 are reordered by score; equal scores keep the hybrid's order22 top = sorted(ranking[:depth], key=lambda product_id: score[(query_id, product_id)], reverse=True)23 # products below the top 50 keep their hybrid order24 reranked[query_id] = top + ranking[depth:]2526before, after = evaluate(hybrid, judgments), evaluate(reranked, judgments)27print(f"reranked ndcg@10: {after['ndcg@10'].mean():.3f}")28print(f"reranked judged@10: {after['judged@10'].mean():.3f}")29difference = (after["ndcg@10"] - before["ndcg@10"]).round(3)30print("better than hybrid:", (difference > 0).sum())31print("worse than hybrid:", (difference < 0).sum())
python cross-encoder-rerank.pycross-encoder model: Alibaba-NLP/gte-reranker-modernbert-base
reranked ndcg@10: 0.798
reranked judged@10: 0.980
better than hybrid: 11
worse than hybrid: 12
这里重排器没有帮上忙。它的得分为 0.798,而它所基于的混合检索结果为 0.812。它改善了 11 个查询,损害了 12 个查询。本章在测试查询上的结果汇总如下:
| 排序器 | nDCG@10 | judged@10 |
|---|---|---|
| BM25 | 0.725 | 0.927 |
| 语义检索 | 0.830 | 0.990 |
| 混合检索(RRF,k = 60) | 0.812 | 0.980 |
| 重排后的混合检索(前 50 名) | 0.798 | 0.980 |
它搞砸的一个查询 refrigerator with ice an water in door 说明了重排器是如何失分的:
1"""Shows the hybrid and reranked top 5 for one query whose nDCG@10 the reranker lowered, with each product's cross-encoder score and judgment. The prose's point rests on the two rank lines and the two nDCG@10 lines."""2import re34from lib.fusion import reciprocal_rank_fusion5from lib.lexical import LexicalSearch6from lib.metrics import ndcg_at_k7from lib.recorded import cross_encoder_scores8from lib.semantic import SemanticSearch9from lib.store import load_products, load_queries, product_text, qrels1011LABELS = {2: "exact", 1: "partial", 0: "irrelevant"}1213products = load_products()14query_id = 40615query = load_queries().at[query_id, "query"]16judgments = qrels()[query_id]17recorded = cross_encoder_scores()18score = recorded[recorded["query_id"] == query_id].set_index("product_id")["score"]1920bm25 = {query_id: list(LexicalSearch(products).search(query, k=100).index)}21semantic = {query_id: list(SemanticSearch().search(query_id, k=100).index)}22hybrid = reciprocal_rank_fusion([bm25, semantic])[query_id]23# the top 50 are reordered by score; the rest stay in the hybrid's order24reranked = sorted(hybrid[:50], key=score.get, reverse=True) + hybrid[50:]2526print("query:", query)27for name, ranking in [("hybrid", hybrid), ("reranked", reranked)]:28 first_exact = next(rank for rank, product_id in enumerate(ranking, start=1) if judgments.get(product_id) == 2)29 print()30 print(f"{name} ndcg@10: {ndcg_at_k(ranking, judgments, 10):.3f}")31 print(f"{name} rank of the first exact product: {first_exact}")32 for product_id in ranking[:5]:33 label = LABELS.get(judgments.get(product_id), "unjudged")34 print(f" {score[product_id]:.2f} {label:<10} {products.at[product_id, 'name']}")353637exact = next(product_id for product_id in hybrid if judgments.get(product_id) == 2)38words = set(re.findall(r"\w+", product_text(products).at[exact].lower())) # name, class, category, description39print()40print("hybrid's first exact product:", products.at[exact, "name"])41print("its text says ice or water:", "yes" if {"ice", "water"} & words else "no")
python rerank-one-query.pyquery: refrigerator with ice an water in door
hybrid ndcg@10: 0.660
hybrid rank of the first exact product: 3
1.95 partial apec water icemaker-kit-38-14-ro refrigerator/icemaker installation accessory
2.06 partial 36 '' french door 26.1 cu . ft. smart energy star refrigerator with interior ice maker
1.44 exact 36 '' french door refrigerator 27 cu . ft. energy star refrigerator
2.19 partial 36 '' energy star counter-depth french door 22.6 cu . ft. refrigerator with ice maker
2.19 partial 800 series 21 cu . ft. smart energy star counter depth french door refrigerator with internal water dispenser
reranked ndcg@10: 0.533
reranked rank of the first exact product: 9
2.20 partial smeg classic 36 '' french door 22 cu . ft. refrigerator with ice maker
2.19 partial 36 '' energy star counter-depth french door 22.6 cu . ft. refrigerator with ice maker
2.19 partial 800 series 21 cu . ft. smart energy star counter depth french door refrigerator with internal water dispenser
2.10 partial 28 '' energy star counter depth top freezer 12.56 cu . ft. refrigerator with auto ice maker
2.06 partial 36 '' french door 26.1 cu . ft. smart energy star refrigerator with interior ice maker
hybrid's first exact product: 36 '' french door refrigerator 27 cu . ft. energy star refrigerator
its text says ice or water: no
交叉编码器把第一件完全匹配的商品埋没了,使它从第 3 名降到第 9 名,并使该查询的 nDCG@10 从 0.660 降到 0.533。重排后的前 5 名都是名称中带有 ice maker 或 water dispenser 的冰箱,而评判者把它们标为部分匹配:它们符合所查找的对象,即冰箱,但不符合修饰条件,即门上的冰和水。混合检索结果中第一件完全匹配的商品是一台法式对开门冰箱,它在混合检索前 5 名中得到的交叉编码器分数最低。
评判者把这台冰箱标为完全匹配,所以按数据集的定义,它与查询完全相符。然而它的文本没有提到冰或水,因此任何只读取文本的模型都无法依据这一修饰条件为它排序。WANDS 的评判者除了商品名称,还看到了每件商品的图片和网页(标注指南),这很可能让他们看到了文本没有说出的信息。通用重排器从未见过本店的评判结果。第 9 章会用训练查询的评判结果来训练排序器。
工作准则: 对短列表重排,这里是 50 件商品,因为成本会随每一对而增长。只有当重排器在留出查询上胜过单独使用的最佳检索时,才采用它。这个重排器既没有改进混合检索结果,也没有改进语义检索,所以本店目前最好的流水线仍是单独的语义检索。在放弃重排之前,可以尝试不同的深度,或用你自己的评判结果训练的排序器。在第 9 章的测试查询上,不同的深度和 LambdaMART 都没有胜过语义检索(按预算设定重排深度、在测试查询上报告 LambdaMART)。
这部分尚未翻译,因此以英文显示。
What goes wrong in production
The tokenizer drops words without telling you. bm25s's default tokenizer keeps only tokens of two or more characters and doesn't stem words (bm25s tokenizer). Run it on a train query and a test query, and count the products it can't match:
1"""Tokenizes one train query, one test query and the products' text with bm25s, and counts the products that have `chairs` or only `chair`. The prose's points rest on the printed token lists and on the two product counts."""2import bm25s34from lib.store import load_products, product_text567def tokenize(texts):8 """The tokenizer lib.lexical uses for products and queries, with bm25s's default settings."""9 # stopwords="en" removes English words such as "and"; no stemmer is set, so "chairs" and "chair" stay different tokens10 return bm25s.tokenize(texts, stopwords="en", return_ids=False, show_progress=False)111213for query in ["ge top loading washer 4.5", "bistro table and chairs"]:14 print(f"{query!r} -> {tokenize([query])[0]}")1516products = tokenize(product_text(load_products()).tolist())17print()18print("products with the token chairs:", sum("chairs" in tokens for tokens in products))19print("products with chair but not chairs:", sum("chair" in tokens and "chairs" not in tokens for tokens in products))
python tokenizer-drops-terms.py'ge top loading washer 4.5' -> ['ge', 'top', 'loading', 'washer']
'bistro table and chairs' -> ['bistro', 'table', 'chairs']
products with the token chairs: 895
products with chair but not chairs: 359
The 4.5 in ge top loading washer 4.5 disappears, because 4 and 5 are one character each, so that number plays no part in the ranking. And chairs matches only products that say chairs: 895 products do, while 359 say chair and never chairs, and none of those can match that word of bistro table and chairs. Neither case raises an error. Chapter 5 shows both problems and adds stemming.
Unjudged results look like failures. A ranker that surfaces products nobody judged gets lower scores than it may deserve, and a ranker that surfaces the judged ones gets an advantage. Report judged@10 with every score, and have new products judged when it drops.
Scores from a sample are probably optimistic. The sample keeps 12,000 products, nearly all judged for some query. The full WANDS catalog has 42,994 (repository), and a store's catalog may be far larger, with more near misses competing for the top 10. Nothing in this chapter measures that, but expect lower scores on a full catalog. Don't compare the scores here with the WANDS paper's either: it ran the BM25 of Solr, a search server, over product titles and descriptions (Chen et al.).
Model outputs belong to one model version. Product and query embeddings must come from the same model at the same version, or their similarities aren't comparable. Change the model and you re-embed the whole catalog. Each folder in examples/data/recorded/ has a manifest.json that pins the model and its version.
Reranking costs time on every query. Product embeddings are computed once. A cross-encoder runs on every pair, for every query, while the shopper waits. In this chapter, a recording script computes the scores ahead of time on a GPU, which hides that cost. Chapter 9 counts what each rerank depth costs (Set the rerank depth from a budget).
Working rule: Check the tokens your queries become, track judged@10, and record the model version next to every vector you store.
本章做出的决定
在后续某一章用训练查询检验其中某项之前,这些默认设置一直有效。
- 在测试集上报告,在训练集上调优。 本章的每个分数都是在 30 条测试查询上得出的。本章不做任何调优;后续各章只在 50 条训练查询上调优。
- 主要指标:nDCG@10,并列出 judged@10。 相关性等级用作增益,未经评判的商品按等级 0 计算,judged@10 则显示这一假设在何时影响重大。
- 基线:使用 bm25s 默认参数的 BM25, 即
k1=1.5和b=0.75,检索范围为每件商品的名称、类别、分类和描述。调优要等到可以用训练查询衡量时再进行。 - 语义检索使用已记录的嵌入向量, 来自一个开源模型,通过单位长度向量的点积进行比较,因此每个示例都可以离线运行,且每次输出相同。在不同模型之间做选择是后面的问题。
- 混合检索:RRF,k = 60,权重相等,未经调优, 与其作者的用法一致。
- 重排深度:混合检索的前 50 条, 由已记录的交叉编码器打分,因此每件被重排的商品都有一个已记录的分数。在线上商店中,深度既是质量问题,也是成本问题。
- 要超越的排序器是单独的语义检索, 因为在这份数据上,融合和这个重排器都没有超过它。第 3 章将在评估框架中检验这类差距是否大于随机波动。
- 本章的结论: 融合或重排这类本应有帮助的环节,在这份数据上并未超过语义检索,因此每一项改动都要先经过衡量,才能保留。
本章,在本版次中
版次 2026.11.10示例于 2026年11月19日 通过本章 11个示例 个示例中有 11 个通过
译自英文版图书,版次 2026.11.10。