理解查询

阅读约需 89分钟 分钟,10节中含12个示例。

版次 2026.11.10 示例于 2026年11月19日 通过 132个示例中有132个通过 变化内容

AI 翻译。由人工以英文核验。 我们如何制作图书

译自英文版图书,版次 2026.11.10。 阅读原文

读完本章,你可以在检索之前改写购物者的查询:纠正拼写,统一单位的写法,用同义词扩展查询,预测其商品类别,用大语言模型改写查询,用该模型设想出的商品进行搜索,并把追问转成独立的查询。你还可以依据人工判断,确定每一种改动是否对其所服务的检索器有帮助。

购物者在商店搜索框中输入的一些短查询,无法匹配到他们想要的商品。ligth bulb 拼错了一个词。7qt slow cooker 把尺寸写成了一个词项,而 0 商品中包含这个词项。对于 midcentury tv unit,第 7 章统计了 25 件精确匹配的商品,其中 0 件包含 midcentury,0 件包含 unit。

购物者在搜索书桌之后输入 in white,指的是白色书桌,但仅凭这几个词并不能表明这一点。查询理解(query understanding)是搜索系统在检索之前对查询所做的一切处理,目的是弥合此类差距:它包括纠错、规范化、扩展、分类和改写。第 5 章到第 7 章改变的是检索器。本章改变的是查询,并保持检索器不变:一个是 BM25,即本书在 lib.lexical 中不做词干提取的基线;另一个是语义检索,它将查询的嵌入与每件商品的嵌入进行比较,二者均由 Snowflake/snowflake-arctic-embed-m-v1.5 预先计算。

其中三种技术使用商店的大语言模型。它是 gemma4:12b-it-q4_K_M,是 Google 以 Apache 2.0 许可证发布的开放模型(模型卡)。示例中没有任何地方调用它。recordings/llm_rewrites.py 和 recordings/llm_conversations.py 这两个脚本在温度为 0、种子固定的条件下运行过它一次,并把输出保存为录制结果,示例通过 lib.recorded 读取这些结果。每份录制结果的 manifest.json 中保存了完整的提示词。这些录制结果使用的开放模型和设置与第 4 章相同。大语言模型只能看到购物者的用词,从不接触任何商品。本章的每一项得分都使用人工判断。

每项设置都是在 50 条训练查询上选定的,30 条测试查询则留作报告之用。本章在查看任何测试结果之前,就已在测试查询上确定了 5 项计划中的比较。每一项比较的对象,都是只做了一处改动的检索器,以及对照用的、使用原样查询的同一检索器:

  1. 使用同义词表的 BM25
  2. 带有预测类别加权的语义检索
  3. 基于大语言模型改写结果的 BM25
  4. 基于大语言模型改写结果的语义检索
  5. 使用 HyDE 的语义检索,它把查询的嵌入与大语言模型设想出的一件商品的嵌入取平均

每项计划中的比较都必须满足本书的规则:其配对置换检验的 p 值低于 0.01,即 0.05 除以计划中的比较数。与之并列的还有配对 t 检验和 99% 的自助法区间。拼写纠正、单位规范化和追问改写各自涉及的查询太少,无法检验,因此这几节属于探索性分析。

依据目录中的词语纠正拼写

最近一次修改见于版次 2026.11.10

第一个缺口是拼写错误,例如 ligth bulb。在 BM25 中,拼错的词只能匹配到有相同拼写错误的商品,它还会使查询的嵌入偏离其本意所指的商品,正如第 6 章所示。搜索查询中有许多这类错误。一项关于网页搜索的研究,即 Cucerzan and Brill (2004),报告称发送给搜索引擎的查询中大约有 10% 到 15% 含有错误。经典的纠错器采用噪声信道模型:对于键入的词 t,选择使 P(c) × P(t | c) 最大的纠正词 c(Kernighan et al., 1990)。P(c) 表示候选词有多常见。P(t | c) 是错误模型,表示本意为 c 的购物者键入 t 的可能性有多大。Kernighan 等人把与拼写错误相差一次编辑的词作为候选词:删除一个字母、插入一个字母、替换一个字母,或交换两个相邻字母。把一个字符串变成另一个字符串所需的插入、删除和替换的次数,就是二者的编辑距离(Manning et al., 2008),这里的纠错器还把一次交换算作一次编辑。一本教科书用平实的话陈述了这两条原则:在正确的拼写中,选择最近的;在距离相近的候选词之间,选择更常见的(Manning et al., 2008)。Norvig 用寥寥几行代码就写出了这样的纠错器(Norvig)。

correction(t) = argmax over candidates c of  P(c) × P(t | c)

词典,即被视为拼写正确的词的列表,决定了哪些词会被纠正。网页搜索引擎根据查询日志构建词典,因此它们选出的纠正词是其他用户的查询中最常见的那个(Manning et al.)。本商店没有日志,所以它的词典来自目录中的文本。词典的来源对品牌名称很重要,因为商品搜索中的纠错器必须能区分真正的拼写错误与拼法不合常规的新品牌(Guo et al., 2024)。

这里的纠错器使用最粗糙的错误模型:词典中的词保持不变,一次编辑优于两次编辑,而 P(c),即每个候选词有多常见,用来在编辑次数相同的候选词之间决出胜负。在全部 80 条查询上运行它,看看哪些查询被改动:

catalog-spelling-corrector.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 · scikit-learn 1.9.1 · scipy 1.18.1 通过
"""A spelling corrector built from the catalog's word counts, run on every query and checked againstlib.queries.correct_query. The prose's point rests on the "queries changed" line and on the line that says whether both correctors agree."""import refrom collections import Counterfrom lib.queries import correct_queryfrom lib.store import load_products, load_queries, product_texttexts = product_text(load_products()).str.lower()counts = Counter(re.findall(r"[a-z]+", " ".join(texts)))       # the lexicon, and how common each word isdef edits1(word):    """Every string one edit away: a deletion, a swap of two neighbouring letters, a replacement or an insertion."""    splits = [(word[:cut], word[cut:]) for cut in range(len(word) + 1)]    letters = "abcdefghijklmnopqrstuvwxyz"    return ({left + right[1:] for left, right in splits if right}            | {left + right[1] + right[0] + right[2:] for left, right in splits if len(right) > 1}            | {left + letter + right[1:] for left, right in splits if right for letter in letters}            | {left + letter + right for left, right in splits for letter in letters})def correct(word):    """Keep a word the catalog uses. Otherwise take the commonest catalog word one edit away, else two."""    if not word.isalpha() or word in counts:        return word    candidates = {edit for edit in edits1(word) if edit in counts} or {        two for edit in edits1(word) for two in edits1(edit) if two in counts}    # the most common candidate; on a tie in counts, the alphabetically first one    return max(sorted(candidates), key=counts.get) if candidates else wordqueries = load_queries()corrected = queries["query"].map(lambda query: " ".join(correct(word) for word in query.lower().split()))changed = queries[corrected != queries["query"].str.lower()]print(f"catalog words: {len(counts):,}")print("queries changed:", len(changed))print("of them, train queries:", (changed["split"] == "train").sum())for query_id, query in changed["query"].items():    print(f"  {query} ({changed.at[query_id, 'split']}), corrected: {corrected[query_id]}")same = corrected.equals(queries["query"].map(lambda query: correct_query(query, counts)))print("same as lib.queries.correct_query:", same)for word in ["tye", "tie", "the", "kohler", "whitehaven", "malachi", "luau"]:      # P(c), as counts    print(f"catalog count of {word}:", counts[word])
输出python catalog-spelling-corrector.py
catalog words: 19,680
queries changed: 4
of them, train queries: 3
  foutains with brick look (train), corrected: fountains with brick look
  twin over full bunk beds cool desins (train), corrected: twin over full bunk beds cool designs
  ligth bulb (train), corrected: light bulb
  tye dye duvet cover (test), corrected: the dye duvet cover
same as lib.queries.correct_query: True
catalog count of tye: 0
catalog count of tie: 45
catalog count of the: 20389
catalog count of kohler: 71
catalog count of whitehaven: 60
catalog count of malachi: 5
catalog count of luau: 4

目录使用了 19,680 个不同的词,纠错器改动了 4 条查询。其中有 3 条是训练查询。共享函数 lib.queries.correct_query 做出相同的纠正,并且纠错器拼写每条训练查询的方式与第 6 章的拼写探针相同:那是第 6 章为本书写出的、拼写正确的拼写错误查询副本。

纠错器不会动品牌名称,因为目录用到了它们:kohler 出现 71 次,whitehaven 出现 60 次。它保留的一些词很罕见,例如 malachi,出现 5 次,以及 luau,出现 4 次。如果词典漏掉了罕见词,就会把这两个词都当作拼写错误。

纠错器的失败出现在测试查询中。tye dye duvet cover 变成了 the dye duvet cover:tye 与 tie(在目录中出现 45 次)和 the(20389)都只相差一次编辑,而 P(c) 选中了更常见的词。而 refrigerator with ice an water in door 仍保留着它的 an。那个 an 是一个真词错误,即每个词拼写都正确的拼写错误,例如 "flew form Heathrow"(Manning et al., 2008)。两者都需要上下文。对于 tye 和 an,统计与它相差一次编辑的每个目录词紧接在查询的下一个词之前出现的次数,这个下一个词是 dye 或 water:

spelling-context-counts.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 · scikit-learn 1.9.1 · scipy 1.18.1 通过
"""For the misspelled word in two test queries, `tye` and the real-word error `an`, counts how often each nearby catalog word comes before the query's nextword, and shows the context-free corrector's choice. The prose's point rests on the "catalog count of ... thecorrector's choice" lines."""import refrom collections import Counterfrom lib.queries import correct_word, edits1, word_countsfrom lib.store import load_products, product_texttexts = product_text(load_products()).str.lower()counts = word_counts(texts)                                   # the corrector's lexiconwords_per_product = [re.findall(r"[a-z]+", text) for text in texts]pairs = Counter(pair for words in words_per_product for pair in zip(words, words[1:]))      # neighbouring words# tye in "tye dye duvet cover", an in "refrigerator with ice an water in door": each word, the word after itfor word, after in [("tye", "dye"), ("an", "water")]:    # the typed word and each catalog word one edit away, ranked by how often each comes before the next word    options = sorted(((pairs[(edit, after)], edit) for edit in edits1(word) | {word} if edit in counts), reverse=True)    for count, edit in options[:3]:        print(f"catalog count of {edit} {after}:", count)    choice = correct_word(word, counts)                        # what the corrector without context picks    print(f"the corrector's choice for {word}: {choice}")    print(f"catalog count of {choice} {after}, the corrector's choice:", pairs[(choice, after)])
输出python spelling-context-counts.py
catalog count of tie dye: 18
catalog count of type dye: 0
catalog count of tyke dye: 0
the corrector's choice for tye: the
catalog count of the dye, the corrector's choice: 0
catalog count of and water: 59
catalog count of a water: 20
catalog count of in water: 9
the corrector's choice for an: an
catalog count of an water, the corrector's choice: 0

在 dye 之前,目录文本中 tie 出现 18 次,而纠错器的选择 the 为 0 次。在 water 之前,and 出现 59 次,领先于 a 的 20 次,而纠错器的选择 an 为 0。对这两条查询,如果纠错器用相邻的词为每个候选词打分,就会选出 tie 和 and。现在可以依据各自的判断为每条纠正后的查询打分,以此看出这一改动是否有帮助:

measure-spelling-corrections.py Python 3.12.15 · bm25s 0.3.16 · numpy 2.5.5 · pandas 3.1.1 · scikit-learn 1.9.1 · scipy 1.18.1 通过
"""Scores each typo query, and the variant spelling tye dye, with BM25, and with recorded embeddings where the correctedtext has one, as typed and as corrected. The prose's point rests on the nDCG@10 lines for queries 20 and 224 (train) and the corrected BM25 line for 459 (test)."""from lib.lexical import LexicalSearchfrom lib.metrics import ndcg_at_kfrom lib.queries import correct_query, word_countsfrom lib.recorded import probesfrom lib.semantic import SemanticSearchfrom lib.store import load_products, load_queries, product_text, qrelsproducts, queries, judgments = load_products(), load_queries(), qrels()counts = word_counts(product_text(products))bm25, semantic = LexicalSearch(products), SemanticSearch()probe_queries, probe_vectors = probes()      # recorded embeddings of the four typo queries, spelled correctlyspelling = probe_queries[probe_queries["kind"] == "spelling"]# the probes' embeddings, keyed by the correctly spelled text, so a corrected query can find its ownspelled = {row["query"]: probe_vectors[index] for index, row in spelling.iterrows()}typo_queries = spelling["of_query_id"].astype(int).tolist()print("typo queries:", len(typo_queries))print("of them, test queries:", (queries.loc[typo_queries, "split"] == "test").sum())for query_id in typo_queries + [459]:        # the typo queries, then the one variant spelling, tye dye    typed = queries.at[query_id, "query"]    corrected = correct_query(typed, counts)    label = f"{query_id} ({queries.at[query_id, 'split']})"    print(f"{label}: {typed} -> {corrected}")    for name, text in [("typed", typed), ("corrected", corrected)]:        ranking = list(bm25.search(text, k=10).index)        print(f"{label}, bm25 ndcg@10 {name}: {ndcg_at_k(ranking, judgments[query_id], 10):.3f}")    if corrected == typed.lower():        print(f"{label}: the corrector changes nothing")    elif corrected in spelled:        for name, vector in [("typed", semantic.query_vectors[query_id]), ("corrected", spelled[corrected])]:            ranking = list(semantic.search_vector(vector, k=10).index)            print(f"{label}, semantic ndcg@10 {name}: {ndcg_at_k(ranking, judgments[query_id], 10):.3f}")    else:        print(f"{label}: no recorded embedding of the corrected text")
输出python measure-spelling-corrections.py
typo queries: 4
of them, test queries: 1
305 (train): ligth bulb -> light bulb
305 (train), bm25 ndcg@10 typed: 0.945
305 (train), bm25 ndcg@10 corrected: 0.945
305 (train), semantic ndcg@10 typed: 1.000
305 (train), semantic ndcg@10 corrected: 1.000
20 (train): foutains with brick look -> fountains with brick look
20 (train), bm25 ndcg@10 typed: 0.356
20 (train), bm25 ndcg@10 corrected: 0.507
20 (train), semantic ndcg@10 typed: 0.353
20 (train), semantic ndcg@10 corrected: 0.645
224 (train): twin over full bunk beds cool desins -> twin over full bunk beds cool designs
224 (train), bm25 ndcg@10 typed: 0.488
224 (train), bm25 ndcg@10 corrected: 0.488
224 (train), semantic ndcg@10 typed: 0.565
224 (train), semantic ndcg@10 corrected: 0.616
406 (test): refrigerator with ice an water in door -> refrigerator with ice an water in door
406 (test), bm25 ndcg@10 typed: 0.501
406 (test), bm25 ndcg@10 corrected: 0.501
406 (test): the corrector changes nothing
459 (test): tye dye duvet cover -> the dye duvet cover
459 (test), bm25 ndcg@10 typed: 0.953
459 (test), bm25 ndcg@10 corrected: 0.953
459 (test): no recorded embedding of the corrected text

纠正对 foutains with brick look(训练)的影响最大。语义检索从 0.353 升至 0.645,BM25 从 0.356 升至 0.507。

在另外两条训练集拼写错误中,twin over full bunk beds cool desins 只影响语义检索,从 0.565 升至 0.616,而 ligth bulb 对两种检索器都没有影响。对于 refrigerator with ice an water in door,纠错器没有做任何改动,即使正确地修正也无助于 BM25。对于键入的 an 和正确的 and,bm25s 把 an, and 作为停用词丢弃,而第 6 章中用 and 拼写的查询副本得分为 0.501,与键入的查询相同。

对于 tye dye duvet cover,无论哪种方式,BM25 的得分都是 0.953,因为 the 是停用词,而 tye 不匹配任何商品,所以两个版本搜索的词项相同。语义检索无法在纠正后的文本上评分,因为记录中没有它的嵌入。本商店有 4 条拼写错误查询,其中 1 条是测试查询,如此少的查询所能给出的最小双侧置换 p 值是 0.125。

**工作准则:**依据目录构建词典,有了查询日志之后也依据日志构建,并且只纠正词典中没有的词,这样品牌名称才能保留下来。纠正一个词之前,先检查它的相邻词:没有上下文的纠错器会把变体拼写变成常见词,并且漏掉真词错误。依据拼写错误查询自身的判断为每次纠正打分,并且在测试纠错器之前,收集比本商店多得多的拼写错误查询。

这部分尚未翻译,因此以英文显示。

Normalize units the way the catalog writes them

最近一次修改见于版次 2026.11.8

A typo has one right spelling; a size, as in 7qt slow cooker, often has several. Normalization maps the different ways of writing one thing to a single form, on the products and on the query alike. Sizes in this catalog need it. Chapter 5 found that 7qt and 7 qt never match each other, because the tokenizer drops the one-character 7. A normalizer that joins each number to its unit, written one way, turns both into one term, 7qt. It has to run on the product text when you build the index, and on every query. It also has to produce terms the tokenizer keeps whole. The tokenizer splits words at a decimal point, so 3.7qt would become 7qt. To prevent that, the shared function lib.queries.normalize_units writes the point as an underscore, which the tokenizer keeps inside a word. It also leaves in alone when a number or one follows, as in 3 in 1.

A rule written from the queries would cover qt, in, inch and ". Before you trust it, count how the catalog writes inches. We count them first, print the terms the tokenizer makes of a few normalized texts, and build BM25 three times: over the text as written, normalized by that first rule, and normalized by normalize_units, which also handles the catalog's two apostrophes, '', for inches. We score every query, train or test, that gives a number with a unit:

normalize-units-both-sides.py Python 3.12.15 · bm25s 0.3.16 · numpy 2.5.5 · pandas 3.1.1 · pystemmer 3.1.0 · scikit-learn 1.9.1 · scipy 1.18.1 通过
"""Counts how the catalog writes inches, prints the terms the tokenizer makes of normalized texts, and scores BM25 onevery query with a unit under three rules. The prose's point rests on the "bm25 ndcg@10" lines for queries 62 and 482."""from lib.lexical import LexicalSearchfrom lib.lexical_by_hand import tokenizefrom lib.metrics import ndcg_at_kfrom lib.queries import INCH_SPELLINGS, normalize_units, unit_rulesfrom lib.store import load_products, load_queries, product_text, qrelsproducts, queries, judgments = load_products(), load_queries(), qrels()text = product_text(products).str.lower()# the catalog's spellings of inches after a number; -? also takes a hyphen, as in 48-inchfor name, unit in [("''", r"''"), ('"', r"\""), ("inch", r"-?inch")]:    print(f"products that write a number then {name}:", text.str.contains(rf"\d\s*{unit}").sum())# the first rule, written from the queries' units: lib's inch spellings (inches, inch, in, ") without the two apostrophes; quarts are always includedfirst_try = unit_rules([spelling for spelling in INCH_SPELLINGS if spelling != "''"])   # written from queries# None leaves the text as written, with no normalizationrules = {"as written": None, "first try": first_try, "normalized": unit_rules()}checks = [("3.7-quart slow cooker", "normalized"), ("3 in 1 sofa bed", "normalized"), ("1 in twin", "normalized"),          ("48 '' range hood", "first try"), ("48 '' range hood", "normalized")]for example, name in checks:                                                   # what the tokenizer reads    print(f"terms of {example}, {name}:", " ".join(tokenize(normalize_units(example, rules[name]))))searches = {}for name, rule in rules.items():    rewritten = products.copy()    if rule is not None:        for column in ["name", "product_class", "category", "description"]:   # the columns product_text joins            rewritten[column] = rewritten[column].map(lambda value: normalize_units(value, rule))    searches[name] = LexicalSearch(rewritten)# queries with a number and a unit: qt, in, inch or a double quotewith_units = queries[queries["query"].str.contains(r"\d\s*(?:qt|in\b|inch|\")")]print("queries with a unit:", len(with_units))print("of them, test queries:", (with_units["split"] == "test").sum())for query_id, query in with_units["query"].items():    label = f"{query_id} ({queries.at[query_id, 'split']})"    print(f"{label}: {query} -> {normalize_units(query)}")    for name, rule in rules.items():        typed = query if rule is None else normalize_units(query, rule)        ranking = list(searches[name].search(typed, k=10).index)        print(f"{label}, bm25 ndcg@10 {name}: {ndcg_at_k(ranking, judgments[query_id], 10):.3f}")
输出python normalize-units-both-sides.py
products that write a number then '': 3014
products that write a number then ": 65
products that write a number then inch: 215
terms of 3.7-quart slow cooker, normalized: 3_7qt slow cooker
terms of 3 in 1 sofa bed, normalized: sofa bed
terms of 1 in twin, normalized: 1in twin
terms of 48 '' range hood, first try: 48 range hood
terms of 48 '' range hood, normalized: 48in range hood
queries with a unit: 4
of them, test queries: 2
62 (train): 7qt slow cooker -> 7qt slow cooker
62 (train), bm25 ndcg@10 as written: 0.554
62 (train), bm25 ndcg@10 first try: 0.930
62 (train), bm25 ndcg@10 normalized: 0.930
101 (train): 48 in entry table with side by side drawer -> 48in entry table with side by side drawer
101 (train), bm25 ndcg@10 as written: 0.569
101 (train), bm25 ndcg@10 first try: 0.569
101 (train), bm25 ndcg@10 normalized: 0.569
285 (test): 48" sliding single track , barn door for laundry -> 48in sliding single track , barn door for laundry
285 (test), bm25 ndcg@10 as written: 0.592
285 (test), bm25 ndcg@10 first try: 0.592
285 (test), bm25 ndcg@10 normalized: 0.592
482 (test): 48 inch kitchen hood -> 48in kitchen hood
482 (test), bm25 ndcg@10 as written: 1.000
482 (test), bm25 ndcg@10 first try: 0.824
482 (test), bm25 ndcg@10 normalized: 1.000

The catalog's usual way to write inches is the one a rule written from the queries misses: two apostrophes after a number, in 3014 products, against 65 for " and 215 for inch.

The tokenizer reads 3.7-quart slow cooker, normalized, as 3_7qt slow cooker, and 3 in 1 sofa bed as sofa bed. normalize_units still misreads 1 in twin as 1in twin, which is why the terms are worth printing.

Normalizing lifts the train query 7qt slow cooker: BM25's nDCG@10 goes from 0.554 to 0.930, and the first rule gives 0.930. The test query 48 inch kitchen hood shows what the first rule loses by missing the two apostrophes. It scores 1.000 as written and 0.824 under the first rule. The query becomes 48in, while the first rule leaves a hood written 48 '' as 48 range hood, and the full rule makes it 48in range hood. With the full rule, the hood query scores 1.000.

The other train query, 48 in entry table with side by side drawer, scores 0.569 as written and 0.569 normalized, and the test query about a sliding barn door scores 0.592 and 0.592. That makes 4 queries with a unit, 2 of them test queries: too few to test.

Working rule: Write normalization rules from the catalog's text, not from the queries, by counting every way it writes each unit. Print the terms the tokenizer makes of the normalized text, because a rule that looks right can produce terms that match the wrong products. Apply the same rules to products and queries in one release, and rebuild the index. When the catalog keeps sizes in fields, such as the width in attributes, match the size against the field instead, as chapter 6 suggests.

这部分尚未翻译,因此以英文显示。

Expand queries with a synonym list

最近一次修改见于版次 2026.11.10

Some gaps are neither typos nor units but different words for one thing, as when the exact products for midcentury tv unit say stand. Query expansion adds terms to a query so that it can match products that use other words for the same thing. Expansion generally increases recall, the share of the relevant products that a search finds (Manning et al., 2008). It can also significantly decrease precision, the share of found products that are relevant, particularly when the query contains ambiguous terms (Manning et al., 2008). A word that expansion adds can match products the shopper didn't ask for, and move the query away from its meaning, which is called query drift. Expansion terms can come from a hand-built list of synonyms, or from the results. Pseudo-relevance feedback assumes the top results are relevant and adds their terms to the query, which makes drift its known risk (Manning et al., 2008).

Here lib.queries.SYNONYMS holds 14 groups of interchangeable words, such as stand and unit, written by reading the train queries whose exact products miss a query word. When a query contains a word or phrase from a group, expand appends the group's other members, and BM25 searches the result. Measure it on the train queries, then make the first of the chapter's planned comparisons on the test queries:

synonym-expansion-train-and-test.py Python 3.12.15 · bm25s 0.3.16 · numpy 2.5.5 · pandas 3.1.1 · scikit-learn 1.9.1 · scipy 1.18.1 通过
"""Expands each query with the synonym list, scores BM25 on the train and test queries as typed and expanded, andruns the list's planned test comparison. The prose's point rests on the "synonyms, bm25" lines of that comparison."""from lib.lexical import LexicalSearchfrom lib.metrics import evaluatefrom lib.queries import PLANNED_TEST_COMPARISONS, SYNONYMS, THRESHOLD, expand, report_plannedfrom lib.store import load_products, load_queries, qrelsproducts, judgments = load_products(), qrels()bm25 = LexicalSearch(products)print("synonym groups:", len(SYNONYMS))scores = {}for split in ["train", "test"]:    queries = load_queries(split)    expanded = queries.assign(query=queries["query"].map(expand))    changed = queries.index[expanded["query"] != queries["query"]]    print(f"{split} queries the list changes:", len(changed))    for name, frame in [("typed", queries), ("expanded", expanded)]:        scores[split, name] = evaluate(bm25.run(frame), qrels(split))        print(f"{split} ndcg@10 {name}: {scores[split, name]['ndcg@10'].mean():.3f}")        print(f"{split} judged@10 {name}: {scores[split, name]['judged@10'].mean():.3f}")for query in queries.loc[changed, "query"]:                   # the test queries the list changes    print(f"test query changed: {query}")    # expand returns the query, a space, then the added words, so this slice keeps only the added words    print(f"words the list adds to it: {expand(query)[len(query) + 1:]}")change = (scores["train", "expanded"]["ndcg@10"] - scores["train", "typed"]["ndcg@10"]).round(3)print(f"train queries better, worse: {(change > 0).sum()}, {(change < 0).sum()}")       # for choosing, not evidenceprint(f"share of the net train gain from the largest gain: {change.max() / change.sum():.3f}")for kind, query_id in [("gain", change.idxmax()), ("loss", change.idxmin())]:    query = load_queries().at[query_id, "query"]    # the same slice: everything after the query and the space that expand puts between them    added = expand(query)[len(query) + 1:]    print(f"largest train {kind}: {query}")    print(f"largest train {kind}, words added: {added}")    print(f"largest train {kind}, change: {change[query_id]:+.3f}")named = [product_id for product_id in bm25.search(expand(query), k=10).index if added in products.at[product_id, "name"]]print(f"largest train loss, top 10 names with {added}:", len(named))      # query and added: the largest loss'sprint("largest train loss, of those, unjudged:", sum(product_id not in judgments[query_id] for product_id in named))print("\nplanned test comparisons:", len(PLANNED_TEST_COMPARISONS))print(f"p-value threshold: {THRESHOLD:.2f}")report_planned("synonyms, bm25", scores["test", "expanded"]["ndcg@10"], scores["test", "typed"]["ndcg@10"])
输出python synonym-expansion-train-and-test.py
synonym groups: 14
train queries the list changes: 14
train ndcg@10 typed: 0.695
train judged@10 typed: 0.930
train ndcg@10 expanded: 0.722
train judged@10 expanded: 0.944
test queries the list changes: 1
test ndcg@10 typed: 0.725
test judged@10 typed: 0.927
test ndcg@10 expanded: 0.725
test judged@10 expanded: 0.927
test query changed: indoor multi tiered plant stand
words the list adds to it: unit
train queries better, worse: 6, 5
share of the net train gain from the largest gain: 0.473
largest train gain: midcentury tv unit
largest train gain, words added: stand mid century
largest train gain, change: +0.632
largest train loss: wire basket with dividers
largest train loss, words added: hamper
largest train loss, change: -0.176
largest train loss, top 10 names with hamper: 4
largest train loss, of those, unjudged: 4

planned test comparisons: 5
p-value threshold: 0.01
synonyms, bm25, mean difference: 0.000
synonyms, bm25, wins, losses, ties: 0, 0, 30
synonyms, bm25, permutation p: 1.000
synonyms, bm25, established at 0.01: no
synonyms, bm25, t-test p: 1.000
synonyms, bm25, t-test agrees: yes
synonyms, bm25, 99% interval: 0.000 to 0.000
synonyms, bm25, interval agrees: yes

The list changes 14 train queries. Mean nDCG@10 goes from 0.695 to 0.722, and judged@10, the share of the top 10 with a judgment, rises too, from 0.930 to 0.944.

Of the changed train queries, 6, 5 score better and worse, respectively, and one query, midcentury tv unit, accounts for 0.473 of the net gain, gaining +0.632 with stand mid century added. The list was written from these queries, so even these numbers flatter it.

The largest loss, -0.176, shows drift: wire basket with dividers gets hamper, and 4 products whose names contain hamper enter its top 10. Of those, 4 are unjudged, so the metric counts them as irrelevant, which for hampers is a fair guess.

On the test queries, the list changes 1 query, indoor multi tiered plant stand, by adding unit. Judged@10 stays at 0.927, and the difference isn't established (planned comparisons). On the queries it was written from, the list helped about as many queries as it hurt, and queries it wasn't written for rarely contain its words.

Working rule: Keep a synonym list for vocabulary gaps you have seen, write each entry from a failing query, and read the products each entry adds before you keep it. Never report a list's gain on the queries you wrote it from, and keep an entry only if a planned comparison on the test queries shows a gain. Test any expansion of semantic retrieval first: the LLM's rewrites, which add synonyms and also broaden and drop words, hurt it. When the gaps are too many to list, try stemming or learned sparse retrieval instead.

基于商品文本训练查询分类器

最近一次修改见于版次 2026.11.10

系统除了改写查询中的词,还可以预测查询要找的是哪一类商品。查询分类(query classification)把查询归入商店分类体系中的某个类别,例如 Range Hoods。在商品搜索中,“将查询映射到预先定义的商品分类体系,是电商查询理解流程中的重要一步”(Skinner and Kallumadi, 2019):预测出的类别可以引导检索转向目录中正确的部分。训练分类器通常需要带标注的查询。Skinner 和 Kallumadi 报告称,利用现有的商品分类映射可以改进查询分类,并减少对标注数据的需求。他们先在类别已知的商品标题上做预训练,再用较少的带标注查询进行微调。

这家商店走得更远,完全不使用带标注的查询,因为它根本没有。WANDS 为每个查询给出一个 query_class,即“查询所属的类别”(WANDS),但第 2 章表明它是一个预测值,其中有些取值是错的。而且本书所用的 WANDS 样本中,每个类别至多只有一个查询(examples/data/wands/README.md),因此没有哪个类别有两个查询可供学习。所以我们在带有 product_class 的商品上训练,再把分类器应用于查询。

朴素贝叶斯(naive Bayes)分类器只需几行代码,就能从商品文本中学到类别。用文字来说,当某个类别在商品中很常见,且查询中的词在该类别的商品文本中也很常见时,朴素贝叶斯就认为该类别很可能是这个查询的类别。它把各个词视为相互独立,这就是“朴素”之处。每个计数都加 1,这样一个在某类别中从未出现过的词不会把该类别排除在外:

predicted class = argmax over classes c of  log P(c) + sum over the query's words w of  log P(w | c)
P(w | c)        = (count of w in class c's text + 1) / (number of words in class c's text + vocabulary size)

我们根据商品文本的词频手工实现它,用 scikit-learn 1.9.1 版本中的 MultinomialNB 对其进行核对,然后把它的预测与 query_class 以及每个查询的精确匹配商品所属的类别进行比较:

query-classes-by-hand.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 · scikit-learn 1.9.1 · scipy 1.18.1 通过
"""A naive Bayes query classifier built by hand from product text, checked against scikit-learn's MultinomialNB andcompared with each query's exact products. The prose's point rests on the two "queries where most exact productshave" lines and on the agreement line with MultinomialNB."""import numpy as npfrom lib.queries import class_rows, train_classifierfrom lib.store import load_labels, load_products, load_queries, product_textproducts, queries, labels = load_products(), load_queries(), load_labels()rows = class_rows(products)                     # one row per (product, class) for products with a classvectorizer, model = train_classifier(products)  # the same, by scikit-learn's MultinomialNBwords = vectorizer.transform(product_text(rows))  # word counts: one row per (product, class), one column per wordclasses = np.array(sorted(rows["label"].unique()))print("product classes:", len(classes))print("products without a class:", (products["product_class"] == "").sum())print("products with several classes:", products["product_class"].str.contains("|", regex=False).sum())# query classes whose name matches no product classmissing = queries["query_class"].dropna()[lambda names: ~names.isin(classes)]print("query classes that no product has:", len(missing))print("query classes that no product has, names:", "; ".join(missing))# Naive Bayes: log P(class) + sum over the query's words of log P(word | class), each word count smoothed by 1word_counts = np.vstack([np.asarray(words[(rows["label"] == name).to_numpy()].sum(axis=0)) for name in classes])log_prior = np.log(rows["label"].value_counts()[classes].to_numpy() / len(rows))log_likelihood = np.log((word_counts + 1) / (word_counts.sum(axis=1, keepdims=True) + word_counts.shape[1]))query_words = vectorizer.transform(queries["query"])predicted = classes[np.asarray(query_words @ log_likelihood.T + log_prior).argmax(axis=1)]print("same predictions as MultinomialNB:", (predicted == model.predict(query_words)).sum(), "of", len(queries))queries = queries.assign(predicted=predicted)print("predicted class equals query_class:", (queries["predicted"] == queries["query_class"]).sum())# judgment grade 2 is exact; a product's classes are joined by |, so they're split before matchingexact = labels[labels["grade"] == 2].join(products["product_class"].str.split("|"), on="product_id")for column, name in [("predicted", "predicted class"), ("query_class", "query_class")]:    share = [exact.loc[exact["query_id"] == query_id, "product_class"].apply(lambda names: label in names).mean()             for query_id, label in queries[column].items()]    print(f"queries where most exact products have the {name}:", (np.array(share) > 0.5).sum())# outdoor light fixtures, desk and chair set, and pool floatsfor query_id in [133, 255, 410]:    query, split, label = queries.loc[query_id, ["query", "split", "predicted"]]    print(f"{query_id} ({split}): {query} -> {label}"          f" (query_class: {queries.at[query_id, 'query_class']})")print("predicted class for 7qt slow cooker:", queries.at[62, "predicted"])       # its query_class has no products
输出python query-classes-by-hand.py
product classes: 410
products without a class: 800
products with several classes: 620
query classes that no product has: 1
query classes that no product has, names: Crock Pots & Slow Cookers
same predictions as MultinomialNB: 80 of 80
predicted class equals query_class: 60
queries where most exact products have the predicted class: 63
queries where most exact products have the query_class: 63
133 (test): outdoor light fixtures -> Light Bulbs (query_class: Outdoor Wall Lights)
255 (test): desk and chair set -> Desks (query_class: Office Chairs)
410 (train): pool floats -> Deck Boxes (query_class: Lawn Games)
predicted class for 7qt slow cooker: Slow Cookers

商品共使用 410 个类别。有 800 个商品没有类别,被排除在外;有 620 个商品有多个类别,因此其中每个商品按每个类别各占一条训练行。

手工实现的分类器与 MultinomialNB 对 80 of 80 个查询做出了相同的预测。其中有 60 个查询的预测等于 query_class,但 query_class 本身也只是一种猜测。对照相关性判断来看,这两种猜测不分高下:有 63 个查询的大多数精确匹配商品带有预测类别,有 63 个查询的大多数精确匹配商品带有 query_class。其中三个预测展示了它仅凭词语来理解查询的方式:outdoor light fixtures 被归为 Light Bulbs,desk and chair set 被归为 Desks,pool floats 被归为 Deck Boxes。

样本的设计限制了这一检验所能说明的问题。每个类别至多只由一个查询来检验,因此无法度量各类别的准确率,而且 80 个查询只覆盖了 410 个类别中的一部分,所以这些计数无法说明其余类别的情况。根本无法用这些查询训练查询分类器,比较两个分类器也只能看它们各自对这几个查询判断得是否正确。此外,有 1 个查询类别(Crock Pots & Slow Cookers)根本不对应任何商品类别,因此任何在商品上训练的分类器都无法预测它。

工作准则: 没有带标注的查询时,用你的商品类别训练查询分类器,并对照经过相关性判断的精确匹配商品所属的类别来检验它,而不是对照另一个分类器的猜测。一旦日志为你提供了带标注的查询,就改用这些查询来训练。把预测出的类别当作排序的提示,绝不要把它当作标签。

提升预测类别的得分,而不是按类别过滤

最近一次修改见于版次 2026.11.10

这里的预测类别来自在商品文本上训练的朴素贝叶斯分类器。预测类别可以通过两种方式作用于检索。过滤只保留属于预测类别的商品。提升则给预测类别中商品的得分加上一个固定的量,使这些商品排名上移,同时不删除任何其他商品。过滤的效果完全取决于分类器:预测类别之外的每一个相关商品都会消失。

商店的训练集判定显示了过滤会付出多大代价:对于训练查询,有 501 of 1937 个精确匹配的商品位于其查询的预测类别之外。我们在语义检索的相似度上加入提升。语义检索是商店最好的排序器,因此也是它的待超越排序器。我们在训练查询上尝试五种提升权重和一种过滤。随后,我们把最佳提升用于测试查询,进行本章的第二项计划比较:

class-boost-train-and-test.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 · scikit-learn 1.9.1 · scipy 1.18.1 通过
"""Counts the exact train products outside each query's predicted class, scores a class filter and several boosts onthe train queries, and takes the best boost to the test queries as a planned comparison. The prose's point rests onthe "exact train products outside the predicted class" line and on the planned comparison's lines."""from lib.metrics import evaluatefrom lib.queries import THRESHOLD, class_boost_run, in_class, report_planned, train_classifierfrom lib.semantic import SemanticSearchfrom lib.store import load_products, load_queries, qrelsproducts, queries = load_products(), load_queries()vectorizer, model = train_classifier(products)               # trained on product text, never on queriespredicted = dict(zip(queries.index, model.predict(vectorizer.transform(queries["query"]))))semantic = SemanticSearch()train, train_judgments = load_queries("train"), qrels("train")exact, outside = 0, 0for query_id, judged in train_judgments.items():    member = in_class(products, predicted[query_id])    exact += sum(grade == 2 for grade in judged.values())    outside += sum(grade == 2 and not member[product_id] for product_id, grade in judged.items())print(f"exact train products outside the predicted class: {outside} of {exact}")base = evaluate(semantic.run(train), train_judgments)print(f"train ndcg@10 semantic: {base['ndcg@10'].mean():.3f}")print(f"train judged@10 semantic: {base['judged@10'].mean():.3f}")train_scores = {}for weight in [None, 0.01, 0.02, 0.05, 0.1, 0.2]:                          # None: keep only the predicted class    train_scores[weight] = evaluate(class_boost_run(semantic, train.index, predicted, products, weight), train_judgments)    name = "filter" if weight is None else f"boost {weight}"    print(f"train ndcg@10 {name}: {train_scores[weight]['ndcg@10'].mean():.3f}")# every weight except None, the filterboosts = [weight for weight in train_scores if weight is not None]best = max(boosts, key=lambda weight: train_scores[weight]["ndcg@10"].mean())for name, weight in [("filter", None), ("best boost", best)]:    print(f"train judged@10 {name}: {train_scores[weight]['judged@10'].mean():.3f}")print("best boost on train:", best)print(f"\nthe planned test comparison, threshold {THRESHOLD:.2f}:")test, test_judgments = load_queries("test"), qrels("test")boosted = evaluate(class_boost_run(semantic, test.index, predicted, products, best), test_judgments)plain = evaluate(semantic.run(test), test_judgments)print(f"test judged@10 semantic: {plain['judged@10'].mean():.3f}")print(f"test judged@10 boosted: {boosted['judged@10'].mean():.3f}")report_planned("class boost, semantic", boosted["ndcg@10"], plain["ndcg@10"])
输出python class-boost-train-and-test.py
exact train products outside the predicted class: 501 of 1937
train ndcg@10 semantic: 0.772
train judged@10 semantic: 0.964
train ndcg@10 filter: 0.675
train ndcg@10 boost 0.01: 0.773
train ndcg@10 boost 0.02: 0.774
train ndcg@10 boost 0.05: 0.768
train ndcg@10 boost 0.1: 0.742
train ndcg@10 boost 0.2: 0.692
train judged@10 filter: 0.866
train judged@10 best boost: 0.966
best boost on train: 0.02

the planned test comparison, threshold 0.01:
test judged@10 semantic: 0.990
test judged@10 boosted: 0.993
class boost, semantic, mean difference: 0.005
class boost, semantic, wins, losses, ties: 7, 4, 19
class boost, semantic, permutation p: 0.174
class boost, semantic, established at 0.01: no
class boost, semantic, t-test p: 0.164
class boost, semantic, t-test agrees: yes
class boost, semantic, 99% interval: -0.002 to 0.017
class boost, semantic, interval agrees: yes

在训练查询上,过滤的表现比普通排序器更差:nDCG@10 从 0.772 降至 0.675,judged@10 从 0.964 降至 0.866,因此它的前 10 名中也混入了无人判定过的商品。

较小的提升几乎不改变训练得分,较大的提升则会损害得分:提升权重为 0.01 时,nDCG@10 为 0.773;为 0.2 时,为 0.692。训练集上的最佳提升权重是 0.02,此时 nDCG@10 为 0.774,judged@10 为 0.966。

在测试查询上,不使用提升时 judged@10 为 0.990,使用提升后为 0.993,提升带来的增益并未得到确认(计划比较)。

工作准则: 不要按预测类别过滤。用在训练查询上调好的权重来提升预测类别,并且只有在其计划比较得到确认时才保留该提升。商店的提升没有得到确认,因此其语义检索保持不提升。只有当购物者自己选择了类别时,例如点击某个分类,我们才建议使用类别过滤。

这部分尚未翻译,因此以英文显示。

Rewrite queries with an LLM

最近一次修改见于版次 2026.11.8

Query rewriting replaces the query with a text written to retrieve better. Where the corrector, the unit rules and the synonym list each fix one kind of gap, an LLM can fix spelling, expand abbreviations and add synonyms in one step, with no list to maintain. In rewrite-retrieve-read, the rewritten query goes to the retriever in place of the original (Ma et al., 2023). Studies report gains. With query2doc, which appends an LLM-written passage, a pseudo-document, to the query, BM25 improved by 3% to 15% on ad-hoc IR datasets, with text-davinci-003 writing the passages for MS MARCO and TREC DL queries (Wang et al., 2023). Other studies report failures. A comparison of 11 expansion methods, 12 datasets and 24 retrievers found that expansion helps weaker models and generally harms stronger ones (Weller et al., 2024). Another study found that LLM-based expansion can significantly degrade retrieval when the LLM lacks the knowledge or the query is highly ambiguous (Abe et al., 2025).

The rewrites that the examples read were recorded with the store's LLM, the open model named in the chapter's opening, run locally on each query alone. The prompt's instruction is: "Rewrite this search query from an online furniture and home-goods store so that a search engine finds what the shopper wants. Fix spelling, expand abbreviations and add common synonyms for the product type, but keep every requirement the shopper stated and add none of your own. Return a single line." The rest of the prompt gives the query and asks for JSON, and the recording's manifest holds it in full. The recording script embedded each rewrite as a query, with the same embedding model as the products. BM25 searches the rewrite's text, and semantic retrieval its embedding. The train queries come first, to see what the rewrites change before any test:

llm-rewrites-on-train.py Python 3.12.15 · bm25s 0.3.16 · numpy 2.5.5 · pandas 3.1.1 · scipy 1.18.1 通过
"""Prints the store's recorded LLM rewrites for a few train queries, then scores BM25 and semantic retrieval on thetrain queries, typed and rewritten. The prose's point rests on the nDCG@10 lines for both retrievers."""from lib.lexical import LexicalSearchfrom lib.metrics import evaluatefrom lib.recorded import llm_rewrites, manifestfrom lib.semantic import SemanticSearchfrom lib.store import load_products, load_queries, qrelsrecording = manifest("llm-rewrites")print("model:", recording["model"])print("embedding model:", recording["embedding_model"])print("rewrite prompt, first line:", recording["prompts"]["rewrite"].split("\n")[0])table, rewrite_vectors, _ = llm_rewrites()rewrites = table.set_index("query_id")["rewrite"]# the embeddings come in the table's row order, not query order: row maps each query_id to its rowrow = {query_id: position for position, query_id in enumerate(table["query_id"])}# a spelling fix, a product code, and malachi sledfor query_id in [20, 205, 313]:    print(f"rewrite of {load_queries().at[query_id, 'query']}: {rewrites[query_id]}")queries, judgments = load_queries("train"), qrels("train")products = load_products()# judgment grade 2 is exactexact = [product_id for product_id, grade in judgments[313].items() if grade == 2]print("exact products for malachi sled:", "; ".join(products.loc[exact, "name"]))bm25, semantic = LexicalSearch(products), SemanticSearch()runs = {"bm25, typed": bm25.run(queries),        "bm25, rewrite": bm25.run(queries.assign(query=rewrites[queries.index].to_numpy())),        "semantic, typed": semantic.run(queries),        "semantic, rewrite": {query_id: list(semantic.search_vector(rewrite_vectors[row[query_id]]).index)                              for query_id in queries.index}}scores = {name: evaluate(run, judgments) for name, run in runs.items()}for name, frame in scores.items():    print(f"train ndcg@10 {name}: {frame['ndcg@10'].mean():.3f}")    print(f"train judged@10 {name}: {frame['judged@10'].mean():.3f}")for retriever in ["bm25", "semantic"]:                       # for choosing, not evidence    change = (scores[f"{retriever}, rewrite"]["ndcg@10"] - scores[f"{retriever}, typed"]["ndcg@10"]).round(3)    print(f"train queries better, worse, {retriever}: {(change > 0).sum()}, {(change < 0).sum()}")    print(f"largest train loss, {retriever}: {queries.at[change.idxmin(), 'query']}")    print(f"largest train loss, {retriever}, change: {change.min():+.3f}")
输出python llm-rewrites-on-train.py
model: gemma4:12b-it-q4_K_M
embedding model: Snowflake/snowflake-arctic-embed-m-v1.5
rewrite prompt, first line: Rewrite this search query from an online furniture and home-goods store so that a search engine finds what the shopper wants. Fix spelling, expand abbreviations and add common synonyms for the product type, but keep every requirement the shopper stated and add none of your own. Return a single line.
rewrite of foutains with brick look: fountains with brick look, brick style water features, masonry fountains, faux brick fountains
rewrite of ines shag rug w003096809: Ines shag rug w003096809 area rug carpet
rewrite of malachi sled: Malachi sled sofa couch settee loveseat
exact products for malachi sled: malachi sled end table; malachi 44 '' console table; malachi sled coffee table
train ndcg@10 bm25, typed: 0.695
train judged@10 bm25, typed: 0.930
train ndcg@10 bm25, rewrite: 0.646
train judged@10 bm25, rewrite: 0.868
train ndcg@10 semantic, typed: 0.772
train judged@10 semantic, typed: 0.964
train ndcg@10 semantic, rewrite: 0.759
train judged@10 semantic, rewrite: 0.924
train queries better, worse, bm25: 23, 24
largest train loss, bm25: malachi sled
largest train loss, bm25, change: -0.644
train queries better, worse, semantic: 18, 25
largest train loss, semantic: wall design shelf
largest train loss, semantic, change: -0.653

Spelling comes out right (fountains with brick look, brick style water features, masonry fountains, faux brick fountains), and codes survive (Ines shag rug w003096809 area rug carpet). For malachi sled, a product line, the LLM guessed a sofa: it wrote Malachi sled sofa couch settee loveseat, while the exact products are malachi sled end table; malachi 44 '' console table; malachi sled coffee table.

On the train queries, the rewrite lowers nDCG@10 for both retrievers: BM25 from 0.695 to 0.646, and semantic retrieval from 0.772 to 0.759. Judged@10 falls for both, from 0.930 to 0.868 and from 0.964 to 0.924, so the rewrites surface products nobody judged, and part of each fall in nDCG@10 may come from them.

The losses aren't one bad query: the numbers of queries that score better and worse are 23, 24 for BM25 and 18, 25 for semantic retrieval. BM25's largest loss is malachi sled, the product line the LLM turned into a sofa, at -0.644, and semantic retrieval's is wall design shelf, at -0.653.

These train numbers shape expectations and prove nothing. The third and fourth of the chapter's 5 planned comparisons were fixed in advance, so they go to the test queries whatever the train queries show, and each counts as established if p is below 0.01. Before the test numbers arrive, pick one: does the rewrite help BM25, the weaker retriever, hurt semantic retrieval, the stronger one, or do neither?

llm-rewrites-on-test.py Python 3.12.15 · bm25s 0.3.16 · numpy 2.5.5 · pandas 3.1.1 · scikit-learn 1.9.1 · scipy 1.18.1 通过
"""Scores BM25 and semantic retrieval on the test queries, as typed and rewritten, as the chapter's planned comparisons,then repeats both on condensed runs as an exploratory check. The prose's point rests on the "established at 0.01" lines."""from lib.lexical import LexicalSearchfrom lib.metrics import evaluatefrom lib.queries import THRESHOLD, condensed, report_planned, rewrite_runsfrom lib.semantic import SemanticSearchfrom lib.significance import comparefrom lib.store import load_products, load_queries, qrelsqueries, judgments = load_queries("test"), qrels("test")             # scored on the human judgments# for each retriever, its run on the rewrite, then its run on the query as typedruns = rewrite_runs(queries, LexicalSearch(load_products()), SemanticSearch())print(f"the planned test comparisons, threshold {THRESHOLD:.2f}:")for retriever, (rewritten, typed) in runs.items():    name = f"llm rewrite, {retriever}"    after, before = evaluate(rewritten, judgments), evaluate(typed, judgments)    for form, frame in [("typed", before), ("rewrite", after)]:        print(f"{name}, test ndcg@10 {form}: {frame['ndcg@10'].mean():.3f}")        print(f"{name}, test judged@10 {form}: {frame['judged@10'].mean():.3f}")    report_planned(name, after["ndcg@10"], before["ndcg@10"])    # Exploratory, never established: the same comparison on condensed runs, which drop unjudged products    # rewritten comes first in each pair, as rewrite_runs returns them; the rewrite is "after"    after, before = (evaluate(condensed(run, judgments), judgments)["ndcg@10"] for run in (rewritten, typed))    # confidence = 1 - THRESHOLD: a 99% interval, at the level the planned comparisons use    result = compare(after, before, confidence=1 - THRESHOLD)    print(f"{name}, condensed ndcg@10 typed: {before.mean():.3f}")    print(f"{name}, condensed ndcg@10 rewrite: {after.mean():.3f}")    print(f"{name}, condensed, exploratory mean difference: {result['mean difference']:.3f}")    print(f"{name}, condensed, exploratory permutation p: {result['permutation p']:.3f}")    low, high = result["ci low"], result["ci high"]    print(f"{name}, condensed, exploratory {1 - THRESHOLD:.0%} interval: {low:.3f} to {high:.3f}")
输出python llm-rewrites-on-test.py
the planned test comparisons, threshold 0.01:
llm rewrite, bm25, test ndcg@10 typed: 0.725
llm rewrite, bm25, test judged@10 typed: 0.927
llm rewrite, bm25, test ndcg@10 rewrite: 0.669
llm rewrite, bm25, test judged@10 rewrite: 0.903
llm rewrite, bm25, mean difference: -0.056
llm rewrite, bm25, wins, losses, ties: 13, 15, 2
llm rewrite, bm25, permutation p: 0.060
llm rewrite, bm25, established at 0.01: no
llm rewrite, bm25, t-test p: 0.060
llm rewrite, bm25, t-test agrees: yes
llm rewrite, bm25, 99% interval: -0.143 to 0.005
llm rewrite, bm25, interval agrees: yes
llm rewrite, bm25, condensed ndcg@10 typed: 0.778
llm rewrite, bm25, condensed ndcg@10 rewrite: 0.725
llm rewrite, bm25, condensed, exploratory mean difference: -0.053
llm rewrite, bm25, condensed, exploratory permutation p: 0.044
llm rewrite, bm25, condensed, exploratory 99% interval: -0.150 to -0.002
llm rewrite, semantic, test ndcg@10 typed: 0.830
llm rewrite, semantic, test judged@10 typed: 0.990
llm rewrite, semantic, test ndcg@10 rewrite: 0.749
llm rewrite, semantic, test judged@10 rewrite: 0.947
llm rewrite, semantic, mean difference: -0.081
llm rewrite, semantic, wins, losses, ties: 7, 19, 4
llm rewrite, semantic, permutation p: 0.000
llm rewrite, semantic, established at 0.01: yes
llm rewrite, semantic, t-test p: 0.000
llm rewrite, semantic, t-test agrees: yes
llm rewrite, semantic, 99% interval: -0.132 to -0.035
llm rewrite, semantic, interval agrees: yes
llm rewrite, semantic, condensed ndcg@10 typed: 0.834
llm rewrite, semantic, condensed ndcg@10 rewrite: 0.791
llm rewrite, semantic, condensed, exploratory mean difference: -0.043
llm rewrite, semantic, condensed, exploratory permutation p: 0.002
llm rewrite, semantic, condensed, exploratory 99% interval: -0.079 to -0.014

For BM25, nDCG@10 falls from 0.725 to 0.669, a difference that isn't established (planned comparisons). Judged@10 falls too, from 0.927 to 0.903.

Semantic retrieval shows an established loss: nDCG@10 falls from 0.830 to 0.749, with a permutation p-value that rounds to 0.000. Judged@10 falls as well, from 0.990 to 0.947.

Because judged@10 fell, we repeat both comparisons on condensed lists, which drop unjudged products before scoring. That check was chosen after seeing the drop, so it is exploratory and can't establish anything.

For semantic retrieval, condensed nDCG@10 still falls, from 0.834 to 0.791, a difference of -0.043, a permutation p of 0.002, and a 99% interval from -0.079 to -0.014. So the loss doesn't rest on unjudged products alone.

For BM25, condensed nDCG@10 goes from 0.778 to 0.725, with a permutation p of 0.044.

The rewrite didn't help BM25: it moved the other way from what Weller et al. and query2doc report, though not by an established amount. The rewrite's harm to semantic retrieval is what Weller et al. found for strong retrievers. Both results hold for this model, this prompt and these queries.

The queries that move most show what a rewrite does. Read them with the number of unjudged products in each rewritten top 10, which the metric counts as irrelevant:

rewrite-largest-changes.py Python 3.12.15 · bm25s 0.3.16 · numpy 2.5.5 · pandas 3.1.1 · scikit-learn 1.9.1 · scipy 1.18.1 通过
"""For each retriever, finds the test queries whose nDCG@10 moves most under the rewrite, and counts the unjudgedproducts in each rewritten top 10. The prose's point rests on the largest-loss and largest-gain lines, and on thespelling-only check for BM25's largest gain."""from bm25s.stopwords import STOPWORDS_ENfrom lib.lexical import LexicalSearchfrom lib.metrics import evaluate, ndcg_at_kfrom lib.queries import rewrite_runsfrom lib.recorded import llm_rewrites, probesfrom lib.semantic import SemanticSearchfrom lib.store import load_products, load_queries, qrelsqueries, judgments = load_queries("test"), qrels("test")bm25 = LexicalSearch(load_products())rewrites = llm_rewrites()[0].set_index("query_id")["rewrite"]largest = {}for retriever, (rewritten, typed) in rewrite_runs(queries, bm25, SemanticSearch()).items():    change = evaluate(rewritten, judgments)["ndcg@10"] - evaluate(typed, judgments)["ndcg@10"]    for kind, query_id in [("loss", change.idxmin()), ("gain", change.idxmax())]:        label, largest[(retriever, kind)] = f"{retriever}, largest {kind}", query_id        print(f"{label}: {queries.at[query_id, 'query']}")        print(f"{label}, change: {change[query_id]:+.3f}")        print(f"{label}, rewrite: {rewrites[query_id]}")        unjudged = sum(product_id not in judgments[query_id] for product_id in rewritten[query_id][:10])        print(f"{label}, unjudged products in the rewrite's top 10: {unjudged}")# BM25's largest gain: score the query with only its spelling fixed (chapter 6's probe), apart from the added wordsquery_id = largest[("bm25", "gain")]# the spelling probes, keyed by the test query each one correctsspelled = probes()[0].dropna(subset=["of_query_id"]).set_index("of_query_id")["query"]if query_id in spelled.index:    for name, text in [("typed", queries.at[query_id, "query"]), ("spelling fixed only", spelled[query_id])]:        ranking = list(bm25.search(text, k=10).index)        print(f"bm25, largest gain, ndcg@10 {name}: {ndcg_at_k(ranking, judgments[query_id], 10):.3f}")    # bm25s's English stop words, which it drops before matching    print("bm25 stop words among an and and:", ", ".join(word for word in ["an", "and"] if word in STOPWORDS_EN))
输出python rewrite-largest-changes.py
bm25, largest loss: fold in half table
bm25, largest loss, change: -0.471
bm25, largest loss, rewrite: folding table, collapsible table, drop leaf table, gateleg table
bm25, largest loss, unjudged products in the rewrite's top 10: 3
bm25, largest gain: refrigerator with ice an water in door
bm25, largest gain, change: +0.159
bm25, largest gain, rewrite: refrigerator with ice and water in door, fridge with door dispenser, refrigerator with external water and ice dispenser
bm25, largest gain, unjudged products in the rewrite's top 10: 1
semantic, largest loss: pull out sleeper loveseat
semantic, largest loss, change: -0.304
semantic, largest loss, rewrite: pull out sleeper loveseat sofa couch convertible sofa sofa bed
semantic, largest loss, unjudged products in the rewrite's top 10: 4
semantic, largest gain: nautical platters
semantic, largest gain, change: +0.055
semantic, largest gain, rewrite: nautical platters serving trays seafood platters maritime serving boards
semantic, largest gain, unjudged products in the rewrite's top 10: 0
bm25, largest gain, ndcg@10 typed: 0.501
bm25, largest gain, ndcg@10 spelling fixed only: 0.501
bm25 stop words among an and and: an, and

Both largest losses come from a rewrite that drops or broadens what the shopper asked for. BM25's largest loss, -0.471 for fold in half table, comes from folding table, collapsible table, drop leaf table, gateleg table, which drops in half and adds other kinds of table; 3 of its top 10 are unjudged. Semantic retrieval's largest loss, -0.304 for pull out sleeper loveseat, comes from pull out sleeper loveseat sofa couch convertible sofa sofa bed, which widens a loveseat to any sofa; 4 of its top 10 are unjudged.

BM25's largest gain, +0.159 for refrigerator with ice an water in door, comes from refrigerator with ice and water in door, fridge with door dispenser, refrigerator with external water and ice dispenser. The rewrite fixes an, but with only that fix BM25 scores 0.501, the same as the query as typed, 0.501, because both spellings are stop words. The gain comes from the words the rewrite adds. Semantic retrieval's largest gain is +0.055, for nautical platters, rewritten as nautical platters serving trays seafood platters maritime serving boards.

Working rule: Don't put an LLM rewrite in front of a strong retriever without a planned comparison on the test queries. Read the per-query losses for dropped requirements and broadened products, which the prompt's "keep every requirement" doesn't prevent. The store's semantic retrieval, the ranker to beat, keeps searching the query as typed.

用假想商品检索(HyDE)

最近一次修改见于版次 2026.11.10

查询改写是请 LLM 给出更好的查询,而 HyDE(hypothetical document embeddings,即假想文档嵌入)则请它写出它设想中能够回答该查询的文档。编码器对这份假想文档(由 LLM 为回答查询而写)做嵌入,所得向量会落在语料嵌入空间中真实文档所在的邻域(Gao et al., 2023)。编造的细节不必为真:作者认为编码器的稠密瓶颈会滤掉幻觉出来的细节,但这是他们的论证,而不是测量结果。他们的主要结果使用 InstructGPT(text-davinci-003),采样温度为 0.7。在段落排序基准 TREC DL19 上,HyDE 使 Contriever(一种未经相关性标注训练的嵌入模型)的 nDCG@10 在 0 到 100 的刻度上由 44.5 提高到 61.3(Gao et al., Table 1)。对于用标注微调过的 Contriever,结果取决于所用的 LLM。较小的 LLM 使 nDCG@10 略有下降,原因是较弱的指令模型可能损害微调后检索器的整体表现;InstructGPT 则使之提高,在 DL19 上提高最多,由 62.1 升至 67.4(Table 4)。

有几种扩展方法会在混合中保留原始查询。HyDE 把各份假想文档的嵌入与查询自身的嵌入一起取平均(Gao et al., section 3.2)。相关性反馈根据对首批结果的判断来改进查询,Rocchio 的做法是使查询向量向被判为相关的结果移动,同时在求和中保留原始向量(Manning et al., 2008)。对于 BM25,query2doc 在生成的段落之前把查询重复五次,使原始词项保持其权重(Wang et al., 2023)。

HyDE 的记录中,每个查询对应一个假想商品,由商店的 LLM 仅根据查询写成:包括名称、类别和描述,并按商品的方式嵌入。这里的 hyde 函数把它与查询的嵌入取平均,再把结果缩放回长度 1,这正是第 6 章在任何平均之后给出的建议。这两个嵌入有一处不同:查询的嵌入带有嵌入模型的查询提示,而商品的嵌入没有,所有商品都是如此。

hyde(query) = (embedding(hypothetical product) + embedding(query)) / 2,  then scaled to length 1

在训练查询上,我们还尝试了单独使用假想商品的嵌入;对于 BM25,则尝试单独使用假想商品的文本,以及按 query2doc 的加权方式,在其前面重复五次查询后使用。然后我们运行本章计划内比较中的最后一项:

hyde-train-and-test.py Python 3.12.15 · bm25s 0.3.16 · numpy 2.5.5 · pandas 3.1.1 · scikit-learn 1.9.1 · scipy 1.18.1 通过
"""Scores HyDE, its parts and their combinations with BM25 and semantic retrieval on the train queries, then runs HyDEagainst semantic retrieval as the chapter's last planned comparison on the test queries. The prose's point rests onthe train lines for the hypothetical product alone and on the planned comparison's lines."""import numpy as npfrom lib.lexical import LexicalSearchfrom lib.metrics import evaluatefrom lib.queries import THRESHOLD, report_plannedfrom lib.recorded import llm_rewritesfrom lib.semantic import SemanticSearchfrom lib.store import load_products, load_queries, qrels# the hypothetical products' embeddings come in the table's row order; zip pairs each with its query_idtable, _, hypothetical_vectors = llm_rewrites()hypothetical, vectors = table.set_index("query_id")["hypothetical"], dict(zip(table["query_id"], hypothetical_vectors))bm25, semantic = LexicalSearch(load_products()), SemanticSearch()def hyde(query_id):    """Gao et al.'s average of the hypothetical product's embedding and the query's, scaled back to length 1."""    average = (vectors[query_id] + semantic.query_vectors[query_id]) / 2    return average / np.linalg.norm(average)def runs(queries):    ids, typed = queries.index, queries["query"]    return {"semantic": semantic.run(queries),            "hypothetical alone": {query_id: list(semantic.search_vector(vectors[query_id]).index) for query_id in ids},            "hyde": {query_id: list(semantic.search_vector(hyde(query_id)).index) for query_id in ids},            "bm25": bm25.run(queries),            "bm25, hypothetical alone": bm25.run(queries.assign(query=ids.map(hypothetical))),            "bm25, query five times plus hypothetical": bm25.run(queries.assign(           # query2doc's weighting                query=[" ".join([typed[query_id]] * 5 + [hypothetical[query_id]]) for query_id in ids]))}train = {name: evaluate(run, qrels("train")) for name, run in runs(load_queries("train")).items()}for name, frame in train.items():    print(f"train ndcg@10 {name}: {frame['ndcg@10'].mean():.3f}")    print(f"train judged@10 {name}: {frame['judged@10'].mean():.3f}")change = (train["hypothetical alone"]["ndcg@10"] - train["semantic"]["ndcg@10"]).round(3)    # for choosing, not evidenceprint(f"train queries better, worse, hypothetical alone: {(change > 0).sum()}, {(change < 0).sum()}")print(f"\nthe planned test comparison, threshold {THRESHOLD:.2f}:")test = {name: evaluate(run, qrels("test")) for name, run in runs(load_queries("test")).items()        if name in ["semantic", "hyde"]}for name, frame in test.items():    print(f"test judged@10 {name}: {frame['judged@10'].mean():.3f}")report_planned("hyde, semantic", test["hyde"]["ndcg@10"], test["semantic"]["ndcg@10"])
输出python hyde-train-and-test.py
train ndcg@10 semantic: 0.772
train judged@10 semantic: 0.964
train ndcg@10 hypothetical alone: 0.704
train judged@10 hypothetical alone: 0.890
train ndcg@10 hyde: 0.769
train judged@10 hyde: 0.946
train ndcg@10 bm25: 0.695
train judged@10 bm25: 0.930
train ndcg@10 bm25, hypothetical alone: 0.644
train judged@10 bm25, hypothetical alone: 0.854
train ndcg@10 bm25, query five times plus hypothetical: 0.694
train judged@10 bm25, query five times plus hypothetical: 0.912
train queries better, worse, hypothetical alone: 13, 29

the planned test comparison, threshold 0.01:
test judged@10 semantic: 0.990
test judged@10 hyde: 0.970
hyde, semantic, mean difference: -0.031
hyde, semantic, wins, losses, ties: 8, 14, 8
hyde, semantic, permutation p: 0.108
hyde, semantic, established at 0.01: no
hyde, semantic, t-test p: 0.121
hyde, semantic, t-test agrees: yes
hyde, semantic, 99% interval: -0.115 to 0.001
hyde, semantic, interval agrees: yes

在训练查询上,单独使用假想商品使语义检索的 nDCG@10 由 0.772 变为 0.704,得分更好和更差的查询数为 13, 29。Judged@10 也随之下降,由 0.964 降到 0.890。

与查询取平均后,得分为 0.769,judged@10 为 0.946。BM25 的表现相同:只用查询为 0.695,只用假想文本为 0.644,在其前面重复五次查询为 0.694,judged@10 分别为 0.930、0.854 和 0.912。

保留查询可以挽回假想商品造成的损失;而在训练查询上,对任一检索器而言,这种组合相比只用查询都没有增益。训练数据的数字用来指导设计,只有计划内的测试比较才算证据。

在测试查询上,judged@10 由 0.990 变为 0.970,HyDE 与语义检索之间没有确立的差异(计划内比较)。这一结果对应于每个查询一个假想商品,由商店的 120 亿参数模型 gemma4:12b-it-q4_K_M 在温度 0 下生成。Gao et al. 采样自一个大得多的 LLM,而且只有那一个提高了他们的微调检索器。

工作准则: 尝试 HyDE 时,务必把查询自身的嵌入平均进去。预期收益最大的是未经标注训练的编码器;对于微调过的编码器,Gao et al. 只在其最大的 LLM 上看到了收益,而商店使用自己的模型、提示和编码器,没有看到任何收益。对于像商店这样的编码器,它在这些查询上已经胜过 BM25,因此在为每个查询支付一次 LLM 调用的费用之前,应先测试 HyDE。商店不使用它。

将后续追问改写为独立查询

最近一次修改见于版次 2026.11.10

本章开头提出的最后一个缺口来自对话:顾客的下一条消息往往依赖上一条,例如 in white 是对书桌搜索的进一步限定。检索系统只看到一段文本,因此后续追问必须变成独立查询,即不依赖前面的轮次也能读懂,并且保留顾客仍然想要的内容。对话式查询改写把依赖上下文的问题转换为自成一体且答案相同的问题,CANARD 数据集收录了 40,527 条这样的改写(Elgohary et al., 2019)。在对话式搜索评测 TREC CAsT 2019 中,人工改写比最好的自动系统在相对值上高出 35%(Dalton et al., 2020)。后来使用提示式大语言模型(LLM)的研究发现,“informative query rewrites can yield substantially improved retrieval performance compared to human rewrites, especially with sparse retrievers”(Ye et al., 2023);在 CAsT 的后继评测 TREC iKAT 2025 中,大多数运行在流水线中使用了 LLM,进行单次或多次查询改写(TREC iKAT 2025 overview)。

WANDS 没有对话,所以我们为本书编写了商店的对话:12 条后续追问,每条用一个属性(颜色、形状、尺寸或表面处理)限定一个测试查询。商店的 LLM 只看到这两轮内容,并把每条追问改写为独立查询。例如,在 coat rack for bedroom 之后是 something in black,改写为 black coat rack for bedroom。相关性判断是针对第一条查询给产品评级的,而不是针对限定后的查询,因此 nDCG 无法为后续追问打分。我们转而在每个前 10 名结果中统计三类产品:保留限定条件的产品,即其 attributes 或文本包含该值,例如 black(示例还会打印仅按 attributes 统计的数量);属于正确品类的产品,即评判者针对第一条查询评为完全匹配或部分匹配的产品;以及两者兼具的产品。我们把单独的后续追问、两轮拼接后的文本和独立查询相互对照,各自都作为查询进行嵌入,并以第一条查询作为基线。我们还在独立查询的前 100 名结果上加一个过滤器,只保留明确写有该值的产品,该值取自记录下来的对话;实际运行的系统则必须从追问中提取它。

follow-up-refinements.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 · scikit-learn 1.9.1 · scipy 1.18.1 通过
"""Counts, in each conversation's top 10, the products that keep the refinement and the products of the right kind, forthe follow-up alone, the joined turns, the standalone query, and the standalone query filtered by the value. The prose'spoint rests on the "refinement in text or attributes" and "right kind with the refinement" lines."""import numpy as npfrom lib.queries import mentionsfrom lib.recorded import llm_conversationsfrom lib.semantic import SemanticSearchfrom lib.store import load_products, load_queries, product_text, qrelsproducts, judgments, semantic = load_products(), qrels("test"), SemanticSearch()attributes, text = products["attributes"], product_text(products)        # the embeddings read only the textconversations, vectors = llm_conversations()                 # follow-ups we wrote for the book, not shoppers'print("conversations:", len(conversations))# row 4, the fifth conversation, which the example printsexample = conversations.iloc[4]print(f"first query: {load_queries().at[example['query_id'], 'query']}")print(f"follow-up: {example['follow_up']}")print(f"standalone: {example['standalone']}")shares, filtered_lengths = {}, []                            # shares: (top 10, measure) -> one share per conversationfor row, (first, value) in enumerate(zip(conversations["query_id"], conversations["value"])):    # the standalone query's top 100, the pool the filter draws from    standalone = semantic.search_vector(vectors["standalone"][row], k=100).index    stated = mentions(standalone, value, attributes) | mentions(standalone, value, text)    tops = {"first query": semantic.search(first, k=10).index,            "follow-up alone": semantic.search_vector(vectors["follow_up"][row], k=10).index,            "turns joined": semantic.search_vector(vectors["joined"][row], k=10).index,            "standalone": standalone[:10],            "standalone, filtered": standalone[stated][:10]}         # given the value, which a live system extracts    filtered_lengths.append(len(tops["standalone, filtered"]))    for name, top in tops.items():        in_attributes = mentions(top, value, attributes)        in_either = in_attributes | mentions(top, value, text)        # the first query's grade 1 (partial) or 2 (exact); a product with no judgment counts as 0        right_kind = np.array([judgments[first].get(product_id, 0) >= 1 for product_id in top], dtype=bool)        for measure, hits in [("refinement in attributes", in_attributes), ("refinement in text or attributes", in_either),                              ("right kind", right_kind), ("right kind with the refinement", in_either & right_kind)]:            shares.setdefault((name, measure), []).append(hits.sum() / 10)     # empty places count as missesfor (name, measure), values in shares.items():    print(f"{name}, {measure}: {np.mean(values):.3f}")kept = np.subtract(shares["standalone", "refinement in text or attributes"], shares["turns joined", "refinement in text or attributes"])print(f"standalone against turns joined, more, fewer, same: {(kept > 0).sum()}, {(kept < 0).sum()}, {(kept == 0).sum()}")print("conversations the filter leaves with fewer than 10 products:", sum(length < 10 for length in filtered_lengths))
输出python follow-up-refinements.py
conversations: 12
first query: coat rack for bedroom
follow-up: something in black
standalone: black coat rack for bedroom
first query, refinement in attributes: 0.125
first query, refinement in text or attributes: 0.200
first query, right kind: 0.958
first query, right kind with the refinement: 0.175
follow-up alone, refinement in attributes: 0.433
follow-up alone, refinement in text or attributes: 0.950
follow-up alone, right kind: 0.058
follow-up alone, right kind with the refinement: 0.058
turns joined, refinement in attributes: 0.208
turns joined, refinement in text or attributes: 0.383
turns joined, right kind: 0.933
turns joined, right kind with the refinement: 0.325
standalone, refinement in attributes: 0.200
standalone, refinement in text or attributes: 0.392
standalone, right kind: 0.958
standalone, right kind with the refinement: 0.350
standalone, filtered, refinement in attributes: 0.583
standalone, filtered, refinement in text or attributes: 0.900
standalone, filtered, right kind: 0.675
standalone, filtered, right kind with the refinement: 0.675
standalone against turns joined, more, fewer, same: 5, 4, 3
conversations the filter leaves with fewer than 10 products: 3

单独的后续追问在其前 10 名中有 0.950 保留了限定条件,但只有 0.058 属于正确品类。任何带有第一条查询的形式都能找到正确品类的产品:拼接两轮为 0.933,独立查询为 0.958。

在这些对话上,拼接两轮和独立查询保留限定条件的频率都高于第一条查询,后者从未提出该要求:分别为 0.383 和 0.392,而第一条查询为 0.200。

仅按 attributes 统计,占比较低,分别为 0.208、0.200 和 0.125,因为许多产品只在文本中写明该值,而嵌入读取的正是文本。下面的结论基于“文本或属性”的统计,这种统计可能偏高,因为提到 round legs 的产品也会被算作圆形。

LLM 的改写并不比拼接两轮更好:与拼接两轮相比,独立查询在 5, 4, 3 条对话中分别保留了更多、更少或同样数量的限定产品;既属于正确品类又保留限定条件的产品,占独立查询前 10 名的 0.350,而拼接两轮为 0.325。

过滤器把这一占比提高到 0.675,但会使 3 条对话的产品少于 10 个。

这 12 条对话是我们自己编写的,不是顾客的,因此这些计数只是诊断,而不是检验。

**工作准则:**绝不要单独搜索后续追问。至少要把它与前面的轮次拼接,或者把它改写为独立查询。当限定条件指明了目录中写明的某个值时,提取该值并据此提升排名或过滤,若剩余产品太少,就退回未过滤的列表。在信任其中任何一种做法之前,先用你自己日志中的对话进行测量。

生产环境中会出什么问题

最近一次修改见于版次 2026.11.8

**每一次 LLM 改写,都是查询路径上的一次模型调用。**示例读取的是一次性离线记录下来的改写结果,而线上的搜索框会在购物者等待时,对每个查询都调用模型。query2doc 的作者指出,该方法需要 LLM 推理,速度可能慢得多(Wang et al., 2023),而且更长的查询也会拖慢 BM25 检索。一条经验法则(并非测量结果)是:对高频查询离线改写并缓存结果,只对经测试证明有增益的查询才在线调用模型。

**一条改写只属于一个模型和一个提示词。**换用另一个版本的模型,或者改动提示词的措辞,都可能改变任何一条改写。存储中各份记录的清单保存了模型的摘要和每个提示词的原文,二者之一发生变化,就意味着要重新测量。

**词典必须跟随商品目录。**用上个月的商品目录构建的纠错器,可能把本月的品牌名当作拼写错误。请随索引一起重建词典,并把纠错器改动过的每个查询与原查询一并记入日志,这样才能发现诸如 tye → the 这样的过度纠错。在机器翻译的查询中,拼写错误会改变检索返回的商品(Hanneman et al., 2024),因此,纠错器漏掉的错误,会表现为同一查询的结果不稳定。

**分类器取决于其分类体系。**商品类别被重命名、拆分或合并时,要重新训练分类器。如果某个查询类别没有对应任何商品类别,例如 Crock Pots & Slow Cookers,它就永远不可能被预测出来;对于 7qt slow cooker,分类器预测出的是 Slow Cookers。

**改写后的查询会带出未经判定的商品。**本章中凡是用 nDCG@10 评分的、基于 LLM 的改动,都降低了 judged@10。在采用某种改写之前,应当让人对其前 10 名中未经判定的商品做出判定,或者像第 2 章那样检查压缩后的分数。

**工作准则:**记录用户输入的查询和改写后的查询,缓存 LLM 改写结果,让模型和提示词的版本随输出一起记录,并且每当商品目录变化时,重建词典和分类器。

本章的决定

最近一次修改见于版次 2026.11.8

下面的每项设置都是在训练查询上选定的。表中列出本章在测试查询上的 5 项计划比较,每列一项:每项都是把只改动一处的检索器,与按原样输入查询的同一检索器相比较。

指标 BM25,同义词 语义检索,类别加权 BM25,LLM 改写 语义检索,LLM 改写 语义检索,HyDE
平均差值 0.000 0.005 -0.056 -0.081 -0.031
胜、负、平 0, 0, 30 7, 4, 19 13, 15, 2 7, 19, 4 8, 14, 8
置换检验 p 值 1.000 0.174 0.060 0.000 0.108
是否成立 no no no yes no
t 检验 p 值 1.000 0.164 0.060 0.000 0.121
99% 区间 0.000 to 0.000 -0.002 to 0.017 -0.143 to 0.005 -0.132 to -0.035 -0.115 to 0.001
t 检验是否一致 yes yes yes yes yes
区间是否一致 yes yes yes yes yes
  • 拼写:只纠正目录中不使用的词。 纠错器能修正商店训练查询中的拼写错误,但拼写错误的查询太少,无法对它做测试,而且它需要上下文才能处理 tye dye 或 ice an water。商店目前还没有启用它。
  • 单位:商品和查询采用同一种书写形式,取自目录自身的写法, 小数保持完整。尺寸类查询太少,无法对此做测试,因此基线 BM25 保持不变。
  • 同义词:针对训练查询中发现的缺口建立一份列表, 只用于 BM25。它改变了 1 条测试查询,且没有显示出成立的差异,因此基线不做扩展。
  • 查询类别:用商品文本训练朴素贝叶斯, 因为样本中每个类别至多只有一条查询。预测出的类别只作为提示。过滤在训练集上表现更差,最佳加权在测试集上也不成立,所以语义检索保持不加权。
  • LLM 改写:不采用。 它对语义检索造成的损失是成立的,而对 BM25 的改变并不成立。
  • HyDE:不采用。 与查询取平均后,没有显示出成立的差异,而单独使用假设商品在训练集上表现更差。
  • 后续追问:绝不单独检索。 把各轮对话连接起来,效果与 LLM 给出的独立查询大致相当。下一步是对细化后的值做过滤,这需要提取出该值,并准备一个回退方案。
  • 检索器按输入的原样接收查询: 语义检索仍是要超越的排序器,未做词干提取的 BM25 仍是基线。
  • 本章表明: 在训练查询上看起来正确的改动,必须通过测试查询上的计划比较,而这里测试的改动没有一项作为增益通过。LLM 的改写是唯一成立的差异,而且对语义检索而言是一种损失。

本章,在本版次中

版次 2026.11.10示例于 2026年11月19日 通过本章 12个示例 个示例中有 12 个通过

译自英文版图书,版次 2026.11.10。

引用本节

格式

每种格式均引用英文原书,本译本以其为准。

每种格式都会注明该版次。其链接会打开当前版次,节标题处会说明该节文本最后一次修改的时间。访问日期和检索日期即为您打开此页的当天。

本页内容

第 8 章,共 11 章: 理解查询

  1. 依据目录中的词语纠正拼写
  2. Normalize units the way the catalog writes them
  3. Expand queries with a synonym list
  4. 基于商品文本训练查询分类器
  5. 提升预测类别的得分,而不是按类别过滤
  6. Rewrite queries with an LLM
  7. 用假想商品检索(HyDE)
  8. 将后续追问改写为独立查询
  9. 生产环境中会出什么问题
  10. 本章的决定

返回顶部全部章节