理解查詢
閱讀約需 90 分鐘 分鐘,10 節 中共有 12 個範例。
讀完本章後,你可以在檢索之前修改購物者的查詢:修正拼字、統一單位的寫法、以同義詞擴充查詢、預測商品類別、以大型語言模型改寫查詢、以該模型想像出的商品進行搜尋,以及把後續追問轉成獨立的查詢。你也可以根據人工判斷,決定每一項修改是否對它所接上的檢索器有幫助。
購物者在 the store 搜尋框中輸入的一些簡短查詢,無法比對到他們想要的商品。ligth bulb 把一個字拼錯了。7qt slow cooker 把容量寫成一個詞,而含有該詞的商品有 0 個。以 midcentury tv unit 為例,第 7 章統計了 25 個完全符合的商品,其中 0 個含有 midcentury,0 個含有 unit。
購物者在搜尋書桌之後輸入 in white,指的是白色的書桌,但光看這幾個字並看不出來。查詢理解(query understanding)是搜尋系統在檢索之前對查詢所做的一切處理,目的是彌補這類落差:包括修正、正規化、擴充、分類與改寫。第 5 章到第 7 章改變的是檢索器。本章改變的是查詢,並讓檢索器維持不變:一個是 BM25,即本書在 lib.lexical 中未做詞幹還原的基準;另一個是語意檢索,它把查詢的嵌入向量與每項商品的嵌入向量相比較,兩者皆由 Snowflake/snowflake-arctic-embed-m-v1.5 事先算好。
其中三項技術使用本店的 LLM,也就是 gemma4:12b-it-q4_K_M,這是 Google 以 Apache 2.0 授權發布的開放模型(模型卡)。範例中沒有任何地方會呼叫它。recordings/llm_rewrites.py 與 recordings/llm_conversations.py 這兩個腳本以溫度 0 和固定的種子執行過一次,並把輸出儲存為錄製檔(recordings),範例透過 lib.recorded 讀取這些錄製檔。每份錄製檔的 manifest.json 都記有完整的提示詞。這些錄製檔使用的開放模型與設定,和第 4 章相同。LLM 只看得到購物者的用語,看不到任何商品。本章的每一項分數都使用人工判斷。
每一項設定都是在 50 筆訓練查詢上選定的,30 筆測試查詢則保留下來用於回報結果。本章在讀取任何測試結果之前,就在測試查詢上確定了 5 項計畫中的比較。每一項比較都是只做一項改變的檢索器,並與使用原樣查詢的同一個檢索器相比:
- 使用同義詞清單的 BM25
- 使用預測類別加權的語意檢索
- 以 LLM 的改寫進行的 BM25
- 以 LLM 的改寫進行的語意檢索
- 使用 HyDE 的語意檢索,它把查詢的嵌入向量與 LLM 所想像出的商品之嵌入向量取平均
每一項計畫中的比較都必須通過本書的規則:在成對的排列檢定上,p 值須低於 0.01,也就是 0.05 除以計畫中比較的數目。成對 t 檢定與 99% 的自助法區間則作為輔助並列。拼字修正、單位正規化與後續追問改寫各自涉及的查詢都太少,無法進行檢定,因此這幾節屬於探索性質。
依據型錄用字修正拼字
第一個缺口是打字錯誤,例如 ligth bulb。在 BM25 中,拼錯的字只會比對到有相同拼錯寫法的商品,而且它可能使查詢的嵌入向量偏離其原本想找的商品,如第 6 章所示。搜尋查詢中含有許多這類錯誤。一項針對網頁搜尋的研究,Cucerzan 與 Brill(2004)指出,送到搜尋引擎的查詢中,大約有 10% 至 15% 含有錯誤。經典的修正器採用雜訊通道(noisy channel)模型:對於輸入的字 t,選出使 P(c) × P(t | c) 最大的修正字 c(Kernighan et al., 1990)。P(c) 表示候選字有多常見。P(t | c) 是錯誤模型,表示原本想打 c 的購物者打成 t 的可能性有多高。Kernighan 等人以與錯字相差一次編輯的字作為候選:刪除、插入、替換,或是交換相鄰的兩個字母。將一個字串轉成另一個字串所需的插入、刪除與替換次數,稱為編輯距離(edit distance,Manning et al., 2008),而這裡的修正器也把一次交換算作一次編輯。教科書以簡明的話陳述這兩項原則:在正確的拼法中,選擇最接近的;在差距相近時,選擇較常見的(Manning et al., 2008)。Norvig 只用寥寥數行就寫出這樣的修正器(Norvig)。
correction(t) = argmax over candidates c of P(c) × P(t | c)
詞彙表(lexicon)是被視為正確的字所構成的清單,決定了哪些字會被修正。網頁搜尋引擎從查詢紀錄建立詞彙表,因此它們選出的修正,是其他使用者的查詢中最常見的那一個(Manning et al.)。商店沒有紀錄,所以詞彙表取自型錄的文字。詞彙表的來源對品牌名稱很重要,因為商品搜尋中的修正器必須分辨真正的拼錯,以及拼法不循常規的新品牌(Guo et al., 2024)。
這裡的修正器採用最粗略的錯誤模型:詞彙表中的字予以保留,一次編輯優於兩次編輯,而在編輯次數相同的候選之間,以 P(c)(各候選字有多常見)決定高下。對全部 80 筆查詢執行它,看看哪些查詢被更動:
1"""A spelling corrector built from the catalog's word counts, run on every query and checked against2lib.queries.correct_query. The prose's point rests on the "queries changed" line and on the line that says whether both correctors agree."""3import re4from collections import Counter56from lib.queries import correct_query7from lib.store import load_products, load_queries, product_text89texts = product_text(load_products()).str.lower()10counts = Counter(re.findall(r"[a-z]+", " ".join(texts))) # the lexicon, and how common each word is111213def edits1(word):14 """Every string one edit away: a deletion, a swap of two neighbouring letters, a replacement or an insertion."""15 splits = [(word[:cut], word[cut:]) for cut in range(len(word) + 1)]16 letters = "abcdefghijklmnopqrstuvwxyz"17 return ({left + right[1:] for left, right in splits if right}18 | {left + right[1] + right[0] + right[2:] for left, right in splits if len(right) > 1}19 | {left + letter + right[1:] for left, right in splits if right for letter in letters}20 | {left + letter + right for left, right in splits for letter in letters})212223def correct(word):24 """Keep a word the catalog uses. Otherwise take the commonest catalog word one edit away, else two."""25 if not word.isalpha() or word in counts:26 return word27 candidates = {edit for edit in edits1(word) if edit in counts} or {28 two for edit in edits1(word) for two in edits1(edit) if two in counts}29 # the most common candidate; on a tie in counts, the alphabetically first one30 return max(sorted(candidates), key=counts.get) if candidates else word313233queries = load_queries()34corrected = queries["query"].map(lambda query: " ".join(correct(word) for word in query.lower().split()))35changed = queries[corrected != queries["query"].str.lower()]36print(f"catalog words: {len(counts):,}")37print("queries changed:", len(changed))38print("of them, train queries:", (changed["split"] == "train").sum())39for query_id, query in changed["query"].items():40 print(f" {query} ({changed.at[query_id, 'split']}), corrected: {corrected[query_id]}")41same = corrected.equals(queries["query"].map(lambda query: correct_query(query, counts)))42print("same as lib.queries.correct_query:", same)43for word in ["tye", "tie", "the", "kohler", "whitehaven", "malachi", "luau"]: # P(c), as counts44 print(f"catalog count of {word}:", counts[word])
python catalog-spelling-corrector.pycatalog words: 19,680
queries changed: 4
of them, train queries: 3
foutains with brick look (train), corrected: fountains with brick look
twin over full bunk beds cool desins (train), corrected: twin over full bunk beds cool designs
ligth bulb (train), corrected: light bulb
tye dye duvet cover (test), corrected: the dye duvet cover
same as lib.queries.correct_query: True
catalog count of tye: 0
catalog count of tie: 45
catalog count of the: 20389
catalog count of kohler: 71
catalog count of whitehaven: 60
catalog count of malachi: 5
catalog count of luau: 4
型錄使用了 19,680 個不同的字,修正器更動了 4 筆查詢。其中 3 筆是訓練查詢。共用函式 lib.queries.correct_query 會做出相同的修正,而修正器對每筆訓練查詢的拼寫方式,與第 6 章的拼字探測相同:也就是第 6 章為本書所寫的、這些打字錯誤查詢的拼寫正確版本。
修正器不會動品牌名稱,因為型錄有使用它們:kohler 出現 71 次,whitehaven 出現 60 次。它保留的字中有些很罕見,例如 malachi 使用了 5 次,luau 使用了 4 次。若詞彙表排除罕見的字,這兩個字都會被當成錯字。
修正器的失誤出現在測試查詢中。tye dye duvet cover 變成 the dye duvet cover:tye 與 tie(在型錄中出現 45 次)以及 the(20389)都只差一次編輯,而 P(c) 選了較常見的字。另外,refrigerator with ice an water in door 保留了它的 an。那個 an 是真詞錯誤(real-word error),也就是每個字都拼對的打字錯誤,如 "flew form Heathrow"(Manning et al., 2008)。兩者都需要上下文。對於 tye 與 an,計算與它相差一次編輯的每個型錄字,緊接在查詢下一個字(dye 或 water)之前出現的次數:
1"""For the misspelled word in two test queries, `tye` and the real-word error `an`, counts how often each nearby catalog word comes before the query's next2word, and shows the context-free corrector's choice. The prose's point rests on the "catalog count of ... the3corrector's choice" lines."""4import re5from collections import Counter67from lib.queries import correct_word, edits1, word_counts8from lib.store import load_products, product_text910texts = product_text(load_products()).str.lower()11counts = word_counts(texts) # the corrector's lexicon12words_per_product = [re.findall(r"[a-z]+", text) for text in texts]13pairs = Counter(pair for words in words_per_product for pair in zip(words, words[1:])) # neighbouring words1415# tye in "tye dye duvet cover", an in "refrigerator with ice an water in door": each word, the word after it16for word, after in [("tye", "dye"), ("an", "water")]:17 # the typed word and each catalog word one edit away, ranked by how often each comes before the next word18 options = sorted(((pairs[(edit, after)], edit) for edit in edits1(word) | {word} if edit in counts), reverse=True)19 for count, edit in options[:3]:20 print(f"catalog count of {edit} {after}:", count)21 choice = correct_word(word, counts) # what the corrector without context picks22 print(f"the corrector's choice for {word}: {choice}")23 print(f"catalog count of {choice} {after}, the corrector's choice:", pairs[(choice, after)])
python spelling-context-counts.pycatalog count of tie dye: 18
catalog count of type dye: 0
catalog count of tyke dye: 0
the corrector's choice for tye: the
catalog count of the dye, the corrector's choice: 0
catalog count of and water: 59
catalog count of a water: 20
catalog count of in water: 9
the corrector's choice for an: an
catalog count of an water, the corrector's choice: 0
在 dye 之前,型錄文字中 tie 出現 18 次,而修正器的選擇 the 則為 0。在 water 之前,and 出現 59 次,領先 a 的 20 次,而修正器的選擇 an 只有 0。對這兩筆查詢而言,若修正器以鄰近的字替每個候選評分,就會選出 tie 與 and。現在可以用各查詢自己的判斷來為每筆修正後的查詢評分,藉此看出這項更動是否有幫助:
1"""Scores each typo query, and the variant spelling tye dye, with BM25, and with recorded embeddings where the corrected2text has one, as typed and as corrected. The prose's point rests on the nDCG@10 lines for queries 20 and 224 (train) and the corrected BM25 line for 459 (test)."""3from lib.lexical import LexicalSearch4from lib.metrics import ndcg_at_k5from lib.queries import correct_query, word_counts6from lib.recorded import probes7from lib.semantic import SemanticSearch8from lib.store import load_products, load_queries, product_text, qrels910products, queries, judgments = load_products(), load_queries(), qrels()11counts = word_counts(product_text(products))12bm25, semantic = LexicalSearch(products), SemanticSearch()13probe_queries, probe_vectors = probes() # recorded embeddings of the four typo queries, spelled correctly14spelling = probe_queries[probe_queries["kind"] == "spelling"]15# the probes' embeddings, keyed by the correctly spelled text, so a corrected query can find its own16spelled = {row["query"]: probe_vectors[index] for index, row in spelling.iterrows()}17typo_queries = spelling["of_query_id"].astype(int).tolist()18print("typo queries:", len(typo_queries))19print("of them, test queries:", (queries.loc[typo_queries, "split"] == "test").sum())2021for query_id in typo_queries + [459]: # the typo queries, then the one variant spelling, tye dye22 typed = queries.at[query_id, "query"]23 corrected = correct_query(typed, counts)24 label = f"{query_id} ({queries.at[query_id, 'split']})"25 print(f"{label}: {typed} -> {corrected}")26 for name, text in [("typed", typed), ("corrected", corrected)]:27 ranking = list(bm25.search(text, k=10).index)28 print(f"{label}, bm25 ndcg@10 {name}: {ndcg_at_k(ranking, judgments[query_id], 10):.3f}")29 if corrected == typed.lower():30 print(f"{label}: the corrector changes nothing")31 elif corrected in spelled:32 for name, vector in [("typed", semantic.query_vectors[query_id]), ("corrected", spelled[corrected])]:33 ranking = list(semantic.search_vector(vector, k=10).index)34 print(f"{label}, semantic ndcg@10 {name}: {ndcg_at_k(ranking, judgments[query_id], 10):.3f}")35 else:36 print(f"{label}: no recorded embedding of the corrected text")
python measure-spelling-corrections.pytypo queries: 4
of them, test queries: 1
305 (train): ligth bulb -> light bulb
305 (train), bm25 ndcg@10 typed: 0.945
305 (train), bm25 ndcg@10 corrected: 0.945
305 (train), semantic ndcg@10 typed: 1.000
305 (train), semantic ndcg@10 corrected: 1.000
20 (train): foutains with brick look -> fountains with brick look
20 (train), bm25 ndcg@10 typed: 0.356
20 (train), bm25 ndcg@10 corrected: 0.507
20 (train), semantic ndcg@10 typed: 0.353
20 (train), semantic ndcg@10 corrected: 0.645
224 (train): twin over full bunk beds cool desins -> twin over full bunk beds cool designs
224 (train), bm25 ndcg@10 typed: 0.488
224 (train), bm25 ndcg@10 corrected: 0.488
224 (train), semantic ndcg@10 typed: 0.565
224 (train), semantic ndcg@10 corrected: 0.616
406 (test): refrigerator with ice an water in door -> refrigerator with ice an water in door
406 (test), bm25 ndcg@10 typed: 0.501
406 (test), bm25 ndcg@10 corrected: 0.501
406 (test): the corrector changes nothing
459 (test): tye dye duvet cover -> the dye duvet cover
459 (test), bm25 ndcg@10 typed: 0.953
459 (test), bm25 ndcg@10 corrected: 0.953
459 (test): no recorded embedding of the corrected text
修正對 foutains with brick look(訓練)最重要。語意檢索從 0.353 升到 0.645,BM25 則從 0.356 升到 0.507。
另外兩筆訓練用的錯字查詢中,twin over full bunk beds cool desins 只影響語意檢索,從 0.565 變成 0.616,而 ligth bulb 對兩種檢索器都沒有影響。對 refrigerator with ice an water in door 而言,修正器沒有更動任何東西,即使修正正確,也無助於 BM25。對於輸入的 an 與正確的 and,bm25s 都把 an, and 當作停用詞丟棄,而第 6 章中以 and 拼寫的查詢副本得分為 0.501,與照原樣輸入的查詢相同。
對於 tye dye duvet cover,BM25 在兩種情況下都得到 0.953,因為 the 是停用詞,而 tye 不符合任何商品,所以兩個版本搜尋的詞項相同。語意檢索無法用修正後的文字評分,因為錄製資料中沒有它的嵌入向量。商店有 4 筆打字錯誤查詢,其中 1 筆是測試查詢,如此少量的查詢所能給出的最小雙尾排列檢定 p 值為 0.125。
實用原則: 以你的型錄建立詞彙表,等有查詢紀錄後也納入查詢紀錄,並且只修正詞彙表中沒有的字,這樣品牌名稱才能保留下來。修正一個字之前,先檢查它的鄰近字:沒有上下文的修正器會把變體拼法改成常見的字,也會漏掉真詞錯誤。以打字錯誤查詢自己的判斷為每項修正評分,並在測試修正器之前,蒐集比商店現有數量多得多的打字錯誤查詢。
此部分尚未翻譯,因此以英文呈現。
Normalize units the way the catalog writes them
A typo has one right spelling; a size, as in 7qt slow cooker, often has several. Normalization maps the different ways of writing one thing to a single form, on the products and on the query alike. Sizes in this catalog need it. Chapter 5 found that 7qt and 7 qt never match each other, because the tokenizer drops the one-character 7. A normalizer that joins each number to its unit, written one way, turns both into one term, 7qt. It has to run on the product text when you build the index, and on every query. It also has to produce terms the tokenizer keeps whole. The tokenizer splits words at a decimal point, so 3.7qt would become 7qt. To prevent that, the shared function lib.queries.normalize_units writes the point as an underscore, which the tokenizer keeps inside a word. It also leaves in alone when a number or one follows, as in 3 in 1.
A rule written from the queries would cover qt, in, inch and ". Before you trust it, count how the catalog writes inches. We count them first, print the terms the tokenizer makes of a few normalized texts, and build BM25 three times: over the text as written, normalized by that first rule, and normalized by normalize_units, which also handles the catalog's two apostrophes, '', for inches. We score every query, train or test, that gives a number with a unit:
1"""Counts how the catalog writes inches, prints the terms the tokenizer makes of normalized texts, and scores BM25 on2every query with a unit under three rules. The prose's point rests on the "bm25 ndcg@10" lines for queries 62 and 482."""3from lib.lexical import LexicalSearch4from lib.lexical_by_hand import tokenize5from lib.metrics import ndcg_at_k6from lib.queries import INCH_SPELLINGS, normalize_units, unit_rules7from lib.store import load_products, load_queries, product_text, qrels89products, queries, judgments = load_products(), load_queries(), qrels()10text = product_text(products).str.lower()11# the catalog's spellings of inches after a number; -? also takes a hyphen, as in 48-inch12for name, unit in [("''", r"''"), ('"', r"\""), ("inch", r"-?inch")]:13 print(f"products that write a number then {name}:", text.str.contains(rf"\d\s*{unit}").sum())1415# the first rule, written from the queries' units: lib's inch spellings (inches, inch, in, ") without the two apostrophes; quarts are always included16first_try = unit_rules([spelling for spelling in INCH_SPELLINGS if spelling != "''"]) # written from queries17# None leaves the text as written, with no normalization18rules = {"as written": None, "first try": first_try, "normalized": unit_rules()}19checks = [("3.7-quart slow cooker", "normalized"), ("3 in 1 sofa bed", "normalized"), ("1 in twin", "normalized"),20 ("48 '' range hood", "first try"), ("48 '' range hood", "normalized")]21for example, name in checks: # what the tokenizer reads22 print(f"terms of {example}, {name}:", " ".join(tokenize(normalize_units(example, rules[name]))))2324searches = {}25for name, rule in rules.items():26 rewritten = products.copy()27 if rule is not None:28 for column in ["name", "product_class", "category", "description"]: # the columns product_text joins29 rewritten[column] = rewritten[column].map(lambda value: normalize_units(value, rule))30 searches[name] = LexicalSearch(rewritten)3132# queries with a number and a unit: qt, in, inch or a double quote33with_units = queries[queries["query"].str.contains(r"\d\s*(?:qt|in\b|inch|\")")]34print("queries with a unit:", len(with_units))35print("of them, test queries:", (with_units["split"] == "test").sum())36for query_id, query in with_units["query"].items():37 label = f"{query_id} ({queries.at[query_id, 'split']})"38 print(f"{label}: {query} -> {normalize_units(query)}")39 for name, rule in rules.items():40 typed = query if rule is None else normalize_units(query, rule)41 ranking = list(searches[name].search(typed, k=10).index)42 print(f"{label}, bm25 ndcg@10 {name}: {ndcg_at_k(ranking, judgments[query_id], 10):.3f}")
python normalize-units-both-sides.pyproducts that write a number then '': 3014
products that write a number then ": 65
products that write a number then inch: 215
terms of 3.7-quart slow cooker, normalized: 3_7qt slow cooker
terms of 3 in 1 sofa bed, normalized: sofa bed
terms of 1 in twin, normalized: 1in twin
terms of 48 '' range hood, first try: 48 range hood
terms of 48 '' range hood, normalized: 48in range hood
queries with a unit: 4
of them, test queries: 2
62 (train): 7qt slow cooker -> 7qt slow cooker
62 (train), bm25 ndcg@10 as written: 0.554
62 (train), bm25 ndcg@10 first try: 0.930
62 (train), bm25 ndcg@10 normalized: 0.930
101 (train): 48 in entry table with side by side drawer -> 48in entry table with side by side drawer
101 (train), bm25 ndcg@10 as written: 0.569
101 (train), bm25 ndcg@10 first try: 0.569
101 (train), bm25 ndcg@10 normalized: 0.569
285 (test): 48" sliding single track , barn door for laundry -> 48in sliding single track , barn door for laundry
285 (test), bm25 ndcg@10 as written: 0.592
285 (test), bm25 ndcg@10 first try: 0.592
285 (test), bm25 ndcg@10 normalized: 0.592
482 (test): 48 inch kitchen hood -> 48in kitchen hood
482 (test), bm25 ndcg@10 as written: 1.000
482 (test), bm25 ndcg@10 first try: 0.824
482 (test), bm25 ndcg@10 normalized: 1.000
The catalog's usual way to write inches is the one a rule written from the queries misses: two apostrophes after a number, in 3014 products, against 65 for " and 215 for inch.
The tokenizer reads 3.7-quart slow cooker, normalized, as 3_7qt slow cooker, and 3 in 1 sofa bed as sofa bed. normalize_units still misreads 1 in twin as 1in twin, which is why the terms are worth printing.
Normalizing lifts the train query 7qt slow cooker: BM25's nDCG@10 goes from 0.554 to 0.930, and the first rule gives 0.930. The test query 48 inch kitchen hood shows what the first rule loses by missing the two apostrophes. It scores 1.000 as written and 0.824 under the first rule. The query becomes 48in, while the first rule leaves a hood written 48 '' as 48 range hood, and the full rule makes it 48in range hood. With the full rule, the hood query scores 1.000.
The other train query, 48 in entry table with side by side drawer, scores 0.569 as written and 0.569 normalized, and the test query about a sliding barn door scores 0.592 and 0.592. That makes 4 queries with a unit, 2 of them test queries: too few to test.
Working rule: Write normalization rules from the catalog's text, not from the queries, by counting every way it writes each unit. Print the terms the tokenizer makes of the normalized text, because a rule that looks right can produce terms that match the wrong products. Apply the same rules to products and queries in one release, and rebuild the index. When the catalog keeps sizes in fields, such as the width in attributes, match the size against the field instead, as chapter 6 suggests.
此部分尚未翻譯,因此以英文呈現。
Expand queries with a synonym list
Some gaps are neither typos nor units but different words for one thing, as when the exact products for midcentury tv unit say stand. Query expansion adds terms to a query so that it can match products that use other words for the same thing. Expansion generally increases recall, the share of the relevant products that a search finds (Manning et al., 2008). It can also significantly decrease precision, the share of found products that are relevant, particularly when the query contains ambiguous terms (Manning et al., 2008). A word that expansion adds can match products the shopper didn't ask for, and move the query away from its meaning, which is called query drift. Expansion terms can come from a hand-built list of synonyms, or from the results. Pseudo-relevance feedback assumes the top results are relevant and adds their terms to the query, which makes drift its known risk (Manning et al., 2008).
Here lib.queries.SYNONYMS holds 14 groups of interchangeable words, such as stand and unit, written by reading the train queries whose exact products miss a query word. When a query contains a word or phrase from a group, expand appends the group's other members, and BM25 searches the result. Measure it on the train queries, then make the first of the chapter's planned comparisons on the test queries:
1"""Expands each query with the synonym list, scores BM25 on the train and test queries as typed and expanded, and2runs the list's planned test comparison. The prose's point rests on the "synonyms, bm25" lines of that comparison."""3from lib.lexical import LexicalSearch4from lib.metrics import evaluate5from lib.queries import PLANNED_TEST_COMPARISONS, SYNONYMS, THRESHOLD, expand, report_planned6from lib.store import load_products, load_queries, qrels78products, judgments = load_products(), qrels()9bm25 = LexicalSearch(products)10print("synonym groups:", len(SYNONYMS))11scores = {}12for split in ["train", "test"]:13 queries = load_queries(split)14 expanded = queries.assign(query=queries["query"].map(expand))15 changed = queries.index[expanded["query"] != queries["query"]]16 print(f"{split} queries the list changes:", len(changed))17 for name, frame in [("typed", queries), ("expanded", expanded)]:18 scores[split, name] = evaluate(bm25.run(frame), qrels(split))19 print(f"{split} ndcg@10 {name}: {scores[split, name]['ndcg@10'].mean():.3f}")20 print(f"{split} judged@10 {name}: {scores[split, name]['judged@10'].mean():.3f}")21for query in queries.loc[changed, "query"]: # the test queries the list changes22 print(f"test query changed: {query}")23 # expand returns the query, a space, then the added words, so this slice keeps only the added words24 print(f"words the list adds to it: {expand(query)[len(query) + 1:]}")2526change = (scores["train", "expanded"]["ndcg@10"] - scores["train", "typed"]["ndcg@10"]).round(3)27print(f"train queries better, worse: {(change > 0).sum()}, {(change < 0).sum()}") # for choosing, not evidence28print(f"share of the net train gain from the largest gain: {change.max() / change.sum():.3f}")29for kind, query_id in [("gain", change.idxmax()), ("loss", change.idxmin())]:30 query = load_queries().at[query_id, "query"]31 # the same slice: everything after the query and the space that expand puts between them32 added = expand(query)[len(query) + 1:]33 print(f"largest train {kind}: {query}")34 print(f"largest train {kind}, words added: {added}")35 print(f"largest train {kind}, change: {change[query_id]:+.3f}")36named = [product_id for product_id in bm25.search(expand(query), k=10).index if added in products.at[product_id, "name"]]37print(f"largest train loss, top 10 names with {added}:", len(named)) # query and added: the largest loss's38print("largest train loss, of those, unjudged:", sum(product_id not in judgments[query_id] for product_id in named))3940print("\nplanned test comparisons:", len(PLANNED_TEST_COMPARISONS))41print(f"p-value threshold: {THRESHOLD:.2f}")42report_planned("synonyms, bm25", scores["test", "expanded"]["ndcg@10"], scores["test", "typed"]["ndcg@10"])
python synonym-expansion-train-and-test.pysynonym groups: 14
train queries the list changes: 14
train ndcg@10 typed: 0.695
train judged@10 typed: 0.930
train ndcg@10 expanded: 0.722
train judged@10 expanded: 0.944
test queries the list changes: 1
test ndcg@10 typed: 0.725
test judged@10 typed: 0.927
test ndcg@10 expanded: 0.725
test judged@10 expanded: 0.927
test query changed: indoor multi tiered plant stand
words the list adds to it: unit
train queries better, worse: 6, 5
share of the net train gain from the largest gain: 0.473
largest train gain: midcentury tv unit
largest train gain, words added: stand mid century
largest train gain, change: +0.632
largest train loss: wire basket with dividers
largest train loss, words added: hamper
largest train loss, change: -0.176
largest train loss, top 10 names with hamper: 4
largest train loss, of those, unjudged: 4
planned test comparisons: 5
p-value threshold: 0.01
synonyms, bm25, mean difference: 0.000
synonyms, bm25, wins, losses, ties: 0, 0, 30
synonyms, bm25, permutation p: 1.000
synonyms, bm25, established at 0.01: no
synonyms, bm25, t-test p: 1.000
synonyms, bm25, t-test agrees: yes
synonyms, bm25, 99% interval: 0.000 to 0.000
synonyms, bm25, interval agrees: yes
The list changes 14 train queries. Mean nDCG@10 goes from 0.695 to 0.722, and judged@10, the share of the top 10 with a judgment, rises too, from 0.930 to 0.944.
Of the changed train queries, 6, 5 score better and worse, respectively, and one query, midcentury tv unit, accounts for 0.473 of the net gain, gaining +0.632 with stand mid century added. The list was written from these queries, so even these numbers flatter it.
The largest loss, -0.176, shows drift: wire basket with dividers gets hamper, and 4 products whose names contain hamper enter its top 10. Of those, 4 are unjudged, so the metric counts them as irrelevant, which for hampers is a fair guess.
On the test queries, the list changes 1 query, indoor multi tiered plant stand, by adding unit. Judged@10 stays at 0.927, and the difference isn't established (planned comparisons). On the queries it was written from, the list helped about as many queries as it hurt, and queries it wasn't written for rarely contain its words.
Working rule: Keep a synonym list for vocabulary gaps you have seen, write each entry from a failing query, and read the products each entry adds before you keep it. Never report a list's gain on the queries you wrote it from, and keep an entry only if a planned comparison on the test queries shows a gain. Test any expansion of semantic retrieval first: the LLM's rewrites, which add synonyms and also broaden and drop words, hurt it. When the gaps are too many to list, try stemming or learned sparse retrieval instead.
以商品文字訓練查詢分類器
系統除了改寫查詢的用字,也可以預測查詢要找的是哪一類商品。查詢分類(query classification)會把查詢歸入商店分類體系中的某個類別,例如Range Hoods。在商品搜尋中,「將查詢對應到預先定義的商品分類體系,是電子商務查詢理解流程中的重要步驟」(Skinner and Kallumadi, 2019):預測出的類別可以引導檢索前往商品目錄中正確的部分。訓練分類器通常需要已標註的查詢。Skinner 與 Kallumadi 指出,既有的商品分類對應可以改善查詢分類,並減少對標註資料的需求。他們先以類別已知的商品標題進行預訓練,再用較少的已標註查詢進行微調。
這間商店更進一步,完全不使用已標註的查詢,因為它根本沒有。WANDS 為每個查詢提供一個query_class,即「查詢所屬的類別」(WANDS),但第 2 章已說明它是一種預測,而且有些值是錯的。此外,本書的 WANDS 樣本中每個類別至多只有一個查詢(examples/data/wands/README.md),因此沒有任何類別有兩個查詢可供學習。所以我們改用帶有product_class的商品來訓練,再把分類器套用到查詢上。
單純貝氏(naive Bayes)分類器只需幾行程式,就能從商品文字學到類別。用文字來說明:當某個類別在商品中很常見,而且查詢的字詞在該類別的商品文字中也很常見時,單純貝氏就會認為這個類別很可能是該查詢的類別。它把各個字詞視為彼此獨立,這就是「單純」(naive)的部分。每個計數都加 1,使得在某類別中從未出現過的字詞不會直接排除該類別:
predicted class = argmax over classes c of log P(c) + sum over the query's words w of log P(w | c)
P(w | c) = (count of w in class c's text + 1) / (number of words in class c's text + vocabulary size)
我們根據商品文字的字詞計數親手建構它,以 scikit-learn 1.9.1 版的MultinomialNB檢查其結果,再把它的預測與query_class以及各查詢的完全相符商品所屬的類別相比較:
1"""A naive Bayes query classifier built by hand from product text, checked against scikit-learn's MultinomialNB and2compared with each query's exact products. The prose's point rests on the two "queries where most exact products3have" lines and on the agreement line with MultinomialNB."""4import numpy as np56from lib.queries import class_rows, train_classifier7from lib.store import load_labels, load_products, load_queries, product_text89products, queries, labels = load_products(), load_queries(), load_labels()10rows = class_rows(products) # one row per (product, class) for products with a class11vectorizer, model = train_classifier(products) # the same, by scikit-learn's MultinomialNB12words = vectorizer.transform(product_text(rows)) # word counts: one row per (product, class), one column per word13classes = np.array(sorted(rows["label"].unique()))14print("product classes:", len(classes))15print("products without a class:", (products["product_class"] == "").sum())16print("products with several classes:", products["product_class"].str.contains("|", regex=False).sum())17# query classes whose name matches no product class18missing = queries["query_class"].dropna()[lambda names: ~names.isin(classes)]19print("query classes that no product has:", len(missing))20print("query classes that no product has, names:", "; ".join(missing))2122# Naive Bayes: log P(class) + sum over the query's words of log P(word | class), each word count smoothed by 123word_counts = np.vstack([np.asarray(words[(rows["label"] == name).to_numpy()].sum(axis=0)) for name in classes])24log_prior = np.log(rows["label"].value_counts()[classes].to_numpy() / len(rows))25log_likelihood = np.log((word_counts + 1) / (word_counts.sum(axis=1, keepdims=True) + word_counts.shape[1]))26query_words = vectorizer.transform(queries["query"])27predicted = classes[np.asarray(query_words @ log_likelihood.T + log_prior).argmax(axis=1)]28print("same predictions as MultinomialNB:", (predicted == model.predict(query_words)).sum(), "of", len(queries))2930queries = queries.assign(predicted=predicted)31print("predicted class equals query_class:", (queries["predicted"] == queries["query_class"]).sum())32# judgment grade 2 is exact; a product's classes are joined by |, so they're split before matching33exact = labels[labels["grade"] == 2].join(products["product_class"].str.split("|"), on="product_id")34for column, name in [("predicted", "predicted class"), ("query_class", "query_class")]:35 share = [exact.loc[exact["query_id"] == query_id, "product_class"].apply(lambda names: label in names).mean()36 for query_id, label in queries[column].items()]37 print(f"queries where most exact products have the {name}:", (np.array(share) > 0.5).sum())38# outdoor light fixtures, desk and chair set, and pool floats39for query_id in [133, 255, 410]:40 query, split, label = queries.loc[query_id, ["query", "split", "predicted"]]41 print(f"{query_id} ({split}): {query} -> {label}"42 f" (query_class: {queries.at[query_id, 'query_class']})")43print("predicted class for 7qt slow cooker:", queries.at[62, "predicted"]) # its query_class has no products
python query-classes-by-hand.pyproduct classes: 410
products without a class: 800
products with several classes: 620
query classes that no product has: 1
query classes that no product has, names: Crock Pots & Slow Cookers
same predictions as MultinomialNB: 80 of 80
predicted class equals query_class: 60
queries where most exact products have the predicted class: 63
queries where most exact products have the query_class: 63
133 (test): outdoor light fixtures -> Light Bulbs (query_class: Outdoor Wall Lights)
255 (test): desk and chair set -> Desks (query_class: Office Chairs)
410 (train): pool floats -> Deck Boxes (query_class: Lawn Games)
predicted class for 7qt slow cooker: Slow Cookers
商品共使用 410 個類別。有 800 件商品沒有類別,因此被排除;有 620 件商品有多個類別,所以每個類別各自成為一列訓練資料。
手工建構的分類器與MultinomialNB對 80 of 80 個查詢做出相同的預測。它的預測等於query_class的有 60 個,但query_class同樣只是一種猜測。對照相關性判斷,兩種猜測不分高下:有 63 個查詢的多數完全相符商品帶有預測類別,有 63 個查詢的多數完全相符商品帶有query_class。以下三個預測顯示它如何僅憑字詞來解讀查詢:outdoor light fixtures被判為Light Bulbs,desk and chair set被判為Desks,pool floats被判為Deck Boxes。
樣本的設計限制了這項檢查所能呈現的結果。每個類別至多只由一個查詢來測試,因此無法衡量各類別的準確率,而且 80 個查詢只涵蓋 410 個類別的一部分,所以這些計數無法說明其餘類別的情形。這些查詢完全無法用來訓練查詢分類器,兩個分類器也只能比較各自答對這少數查詢的情形。此外,有 1 個查詢類別(Crock Pots & Slow Cookers)完全沒有對應任何商品類別,因此以商品訓練的分類器不可能預測出它。
**實用原則:**在沒有已標註查詢時,以商品的類別來訓練查詢分類器,並以經過判斷的完全相符商品所屬的類別來檢查它,而不是拿另一個分類器的猜測來檢查。一旦記錄檔提供了已標註的查詢,就改用這些查詢訓練。請把預測類別視為排序的提示,而不是標籤。
以加權提升預測類別,而非過濾
這裡的預測類別來自以商品文字訓練的單純貝氏分類器。預測類別可以用兩種方式作用於檢索。過濾只保留屬於預測類別的商品。加權提升則是為預測類別中商品的分數加上固定的量,使這些商品排名上移,同時不移除任何其他商品。過濾的效果取決於分類器的好壞:凡是落在預測類別之外的相關商品,都會消失。
商店的訓練判斷顯示了過濾會付出多少代價:就訓練查詢而言,有 501 of 1937 個完全相符的商品落在其查詢的預測類別之外。我們在語意檢索的相似度上加入加權提升,語意檢索是商店中表現最好的排序器,因此也是必須超越的排序器。我們在訓練查詢上試驗五種提升權重與一種過濾。接著把最佳的加權提升帶到測試查詢,作為本章規劃比較中的第二項:
1"""Counts the exact train products outside each query's predicted class, scores a class filter and several boosts on2the train queries, and takes the best boost to the test queries as a planned comparison. The prose's point rests on3the "exact train products outside the predicted class" line and on the planned comparison's lines."""4from lib.metrics import evaluate5from lib.queries import THRESHOLD, class_boost_run, in_class, report_planned, train_classifier6from lib.semantic import SemanticSearch7from lib.store import load_products, load_queries, qrels89products, queries = load_products(), load_queries()10vectorizer, model = train_classifier(products) # trained on product text, never on queries11predicted = dict(zip(queries.index, model.predict(vectorizer.transform(queries["query"]))))12semantic = SemanticSearch()13train, train_judgments = load_queries("train"), qrels("train")1415exact, outside = 0, 016for query_id, judged in train_judgments.items():17 member = in_class(products, predicted[query_id])18 exact += sum(grade == 2 for grade in judged.values())19 outside += sum(grade == 2 and not member[product_id] for product_id, grade in judged.items())20print(f"exact train products outside the predicted class: {outside} of {exact}")2122base = evaluate(semantic.run(train), train_judgments)23print(f"train ndcg@10 semantic: {base['ndcg@10'].mean():.3f}")24print(f"train judged@10 semantic: {base['judged@10'].mean():.3f}")25train_scores = {}26for weight in [None, 0.01, 0.02, 0.05, 0.1, 0.2]: # None: keep only the predicted class27 train_scores[weight] = evaluate(class_boost_run(semantic, train.index, predicted, products, weight), train_judgments)28 name = "filter" if weight is None else f"boost {weight}"29 print(f"train ndcg@10 {name}: {train_scores[weight]['ndcg@10'].mean():.3f}")30# every weight except None, the filter31boosts = [weight for weight in train_scores if weight is not None]32best = max(boosts, key=lambda weight: train_scores[weight]["ndcg@10"].mean())33for name, weight in [("filter", None), ("best boost", best)]:34 print(f"train judged@10 {name}: {train_scores[weight]['judged@10'].mean():.3f}")35print("best boost on train:", best)3637print(f"\nthe planned test comparison, threshold {THRESHOLD:.2f}:")38test, test_judgments = load_queries("test"), qrels("test")39boosted = evaluate(class_boost_run(semantic, test.index, predicted, products, best), test_judgments)40plain = evaluate(semantic.run(test), test_judgments)41print(f"test judged@10 semantic: {plain['judged@10'].mean():.3f}")42print(f"test judged@10 boosted: {boosted['judged@10'].mean():.3f}")43report_planned("class boost, semantic", boosted["ndcg@10"], plain["ndcg@10"])
python class-boost-train-and-test.pyexact train products outside the predicted class: 501 of 1937
train ndcg@10 semantic: 0.772
train judged@10 semantic: 0.964
train ndcg@10 filter: 0.675
train ndcg@10 boost 0.01: 0.773
train ndcg@10 boost 0.02: 0.774
train ndcg@10 boost 0.05: 0.768
train ndcg@10 boost 0.1: 0.742
train ndcg@10 boost 0.2: 0.692
train judged@10 filter: 0.866
train judged@10 best boost: 0.966
best boost on train: 0.02
the planned test comparison, threshold 0.01:
test judged@10 semantic: 0.990
test judged@10 boosted: 0.993
class boost, semantic, mean difference: 0.005
class boost, semantic, wins, losses, ties: 7, 4, 19
class boost, semantic, permutation p: 0.174
class boost, semantic, established at 0.01: no
class boost, semantic, t-test p: 0.164
class boost, semantic, t-test agrees: yes
class boost, semantic, 99% interval: -0.002 to 0.017
class boost, semantic, interval agrees: yes
在訓練查詢上,過濾的表現比單純的排序器差:nDCG@10 從 0.772 降到 0.675,judged@10 從 0.964 降到 0.866,所以它的前 10 名中也混入了無人判斷過的商品。
小幅的加權提升幾乎不改變訓練分數,大幅的加權提升則會使分數變差:加權提升 0.01 時 nDCG@10 為 0.773,0.2 時為 0.692。訓練上最佳的加權提升為 0.02,此時 nDCG@10 為 0.774,judged@10 為 0.966。
在測試查詢上,未加權提升時 judged@10 為 0.990,加權提升後為 0.993,而加權提升的增益並未成立(規劃比較)。
實用原則: 不要以預測類別進行過濾。應以在訓練查詢上調整過的權重對它加權提升,且只有在其規劃比較成立時才保留加權提升。商店的比較並未成立,因此其語意檢索維持不加權提升。我們只建議在購物者自行選擇類別時才使用類別過濾,例如點選某個分類。
此部分尚未翻譯,因此以英文呈現。
Rewrite queries with an LLM
Query rewriting replaces the query with a text written to retrieve better. Where the corrector, the unit rules and the synonym list each fix one kind of gap, an LLM can fix spelling, expand abbreviations and add synonyms in one step, with no list to maintain. In rewrite-retrieve-read, the rewritten query goes to the retriever in place of the original (Ma et al., 2023). Studies report gains. With query2doc, which appends an LLM-written passage, a pseudo-document, to the query, BM25 improved by 3% to 15% on ad-hoc IR datasets, with text-davinci-003 writing the passages for MS MARCO and TREC DL queries (Wang et al., 2023). Other studies report failures. A comparison of 11 expansion methods, 12 datasets and 24 retrievers found that expansion helps weaker models and generally harms stronger ones (Weller et al., 2024). Another study found that LLM-based expansion can significantly degrade retrieval when the LLM lacks the knowledge or the query is highly ambiguous (Abe et al., 2025).
The rewrites that the examples read were recorded with the store's LLM, the open model named in the chapter's opening, run locally on each query alone. The prompt's instruction is: "Rewrite this search query from an online furniture and home-goods store so that a search engine finds what the shopper wants. Fix spelling, expand abbreviations and add common synonyms for the product type, but keep every requirement the shopper stated and add none of your own. Return a single line." The rest of the prompt gives the query and asks for JSON, and the recording's manifest holds it in full. The recording script embedded each rewrite as a query, with the same embedding model as the products. BM25 searches the rewrite's text, and semantic retrieval its embedding. The train queries come first, to see what the rewrites change before any test:
1"""Prints the store's recorded LLM rewrites for a few train queries, then scores BM25 and semantic retrieval on the2train queries, typed and rewritten. The prose's point rests on the nDCG@10 lines for both retrievers."""3from lib.lexical import LexicalSearch4from lib.metrics import evaluate5from lib.recorded import llm_rewrites, manifest6from lib.semantic import SemanticSearch7from lib.store import load_products, load_queries, qrels89recording = manifest("llm-rewrites")10print("model:", recording["model"])11print("embedding model:", recording["embedding_model"])12print("rewrite prompt, first line:", recording["prompts"]["rewrite"].split("\n")[0])13table, rewrite_vectors, _ = llm_rewrites()14rewrites = table.set_index("query_id")["rewrite"]15# the embeddings come in the table's row order, not query order: row maps each query_id to its row16row = {query_id: position for position, query_id in enumerate(table["query_id"])}17# a spelling fix, a product code, and malachi sled18for query_id in [20, 205, 313]:19 print(f"rewrite of {load_queries().at[query_id, 'query']}: {rewrites[query_id]}")2021queries, judgments = load_queries("train"), qrels("train")22products = load_products()23# judgment grade 2 is exact24exact = [product_id for product_id, grade in judgments[313].items() if grade == 2]25print("exact products for malachi sled:", "; ".join(products.loc[exact, "name"]))26bm25, semantic = LexicalSearch(products), SemanticSearch()27runs = {"bm25, typed": bm25.run(queries),28 "bm25, rewrite": bm25.run(queries.assign(query=rewrites[queries.index].to_numpy())),29 "semantic, typed": semantic.run(queries),30 "semantic, rewrite": {query_id: list(semantic.search_vector(rewrite_vectors[row[query_id]]).index)31 for query_id in queries.index}}32scores = {name: evaluate(run, judgments) for name, run in runs.items()}33for name, frame in scores.items():34 print(f"train ndcg@10 {name}: {frame['ndcg@10'].mean():.3f}")35 print(f"train judged@10 {name}: {frame['judged@10'].mean():.3f}")36for retriever in ["bm25", "semantic"]: # for choosing, not evidence37 change = (scores[f"{retriever}, rewrite"]["ndcg@10"] - scores[f"{retriever}, typed"]["ndcg@10"]).round(3)38 print(f"train queries better, worse, {retriever}: {(change > 0).sum()}, {(change < 0).sum()}")39 print(f"largest train loss, {retriever}: {queries.at[change.idxmin(), 'query']}")40 print(f"largest train loss, {retriever}, change: {change.min():+.3f}")
python llm-rewrites-on-train.pymodel: gemma4:12b-it-q4_K_M
embedding model: Snowflake/snowflake-arctic-embed-m-v1.5
rewrite prompt, first line: Rewrite this search query from an online furniture and home-goods store so that a search engine finds what the shopper wants. Fix spelling, expand abbreviations and add common synonyms for the product type, but keep every requirement the shopper stated and add none of your own. Return a single line.
rewrite of foutains with brick look: fountains with brick look, brick style water features, masonry fountains, faux brick fountains
rewrite of ines shag rug w003096809: Ines shag rug w003096809 area rug carpet
rewrite of malachi sled: Malachi sled sofa couch settee loveseat
exact products for malachi sled: malachi sled end table; malachi 44 '' console table; malachi sled coffee table
train ndcg@10 bm25, typed: 0.695
train judged@10 bm25, typed: 0.930
train ndcg@10 bm25, rewrite: 0.646
train judged@10 bm25, rewrite: 0.868
train ndcg@10 semantic, typed: 0.772
train judged@10 semantic, typed: 0.964
train ndcg@10 semantic, rewrite: 0.759
train judged@10 semantic, rewrite: 0.924
train queries better, worse, bm25: 23, 24
largest train loss, bm25: malachi sled
largest train loss, bm25, change: -0.644
train queries better, worse, semantic: 18, 25
largest train loss, semantic: wall design shelf
largest train loss, semantic, change: -0.653
Spelling comes out right (fountains with brick look, brick style water features, masonry fountains, faux brick fountains), and codes survive (Ines shag rug w003096809 area rug carpet). For malachi sled, a product line, the LLM guessed a sofa: it wrote Malachi sled sofa couch settee loveseat, while the exact products are malachi sled end table; malachi 44 '' console table; malachi sled coffee table.
On the train queries, the rewrite lowers nDCG@10 for both retrievers: BM25 from 0.695 to 0.646, and semantic retrieval from 0.772 to 0.759. Judged@10 falls for both, from 0.930 to 0.868 and from 0.964 to 0.924, so the rewrites surface products nobody judged, and part of each fall in nDCG@10 may come from them.
The losses aren't one bad query: the numbers of queries that score better and worse are 23, 24 for BM25 and 18, 25 for semantic retrieval. BM25's largest loss is malachi sled, the product line the LLM turned into a sofa, at -0.644, and semantic retrieval's is wall design shelf, at -0.653.
These train numbers shape expectations and prove nothing. The third and fourth of the chapter's 5 planned comparisons were fixed in advance, so they go to the test queries whatever the train queries show, and each counts as established if p is below 0.01. Before the test numbers arrive, pick one: does the rewrite help BM25, the weaker retriever, hurt semantic retrieval, the stronger one, or do neither?
1"""Scores BM25 and semantic retrieval on the test queries, as typed and rewritten, as the chapter's planned comparisons,2then repeats both on condensed runs as an exploratory check. The prose's point rests on the "established at 0.01" lines."""3from lib.lexical import LexicalSearch4from lib.metrics import evaluate5from lib.queries import THRESHOLD, condensed, report_planned, rewrite_runs6from lib.semantic import SemanticSearch7from lib.significance import compare8from lib.store import load_products, load_queries, qrels910queries, judgments = load_queries("test"), qrels("test") # scored on the human judgments11# for each retriever, its run on the rewrite, then its run on the query as typed12runs = rewrite_runs(queries, LexicalSearch(load_products()), SemanticSearch())1314print(f"the planned test comparisons, threshold {THRESHOLD:.2f}:")15for retriever, (rewritten, typed) in runs.items():16 name = f"llm rewrite, {retriever}"17 after, before = evaluate(rewritten, judgments), evaluate(typed, judgments)18 for form, frame in [("typed", before), ("rewrite", after)]:19 print(f"{name}, test ndcg@10 {form}: {frame['ndcg@10'].mean():.3f}")20 print(f"{name}, test judged@10 {form}: {frame['judged@10'].mean():.3f}")21 report_planned(name, after["ndcg@10"], before["ndcg@10"])2223 # Exploratory, never established: the same comparison on condensed runs, which drop unjudged products24 # rewritten comes first in each pair, as rewrite_runs returns them; the rewrite is "after"25 after, before = (evaluate(condensed(run, judgments), judgments)["ndcg@10"] for run in (rewritten, typed))26 # confidence = 1 - THRESHOLD: a 99% interval, at the level the planned comparisons use27 result = compare(after, before, confidence=1 - THRESHOLD)28 print(f"{name}, condensed ndcg@10 typed: {before.mean():.3f}")29 print(f"{name}, condensed ndcg@10 rewrite: {after.mean():.3f}")30 print(f"{name}, condensed, exploratory mean difference: {result['mean difference']:.3f}")31 print(f"{name}, condensed, exploratory permutation p: {result['permutation p']:.3f}")32 low, high = result["ci low"], result["ci high"]33 print(f"{name}, condensed, exploratory {1 - THRESHOLD:.0%} interval: {low:.3f} to {high:.3f}")
python llm-rewrites-on-test.pythe planned test comparisons, threshold 0.01:
llm rewrite, bm25, test ndcg@10 typed: 0.725
llm rewrite, bm25, test judged@10 typed: 0.927
llm rewrite, bm25, test ndcg@10 rewrite: 0.669
llm rewrite, bm25, test judged@10 rewrite: 0.903
llm rewrite, bm25, mean difference: -0.056
llm rewrite, bm25, wins, losses, ties: 13, 15, 2
llm rewrite, bm25, permutation p: 0.060
llm rewrite, bm25, established at 0.01: no
llm rewrite, bm25, t-test p: 0.060
llm rewrite, bm25, t-test agrees: yes
llm rewrite, bm25, 99% interval: -0.143 to 0.005
llm rewrite, bm25, interval agrees: yes
llm rewrite, bm25, condensed ndcg@10 typed: 0.778
llm rewrite, bm25, condensed ndcg@10 rewrite: 0.725
llm rewrite, bm25, condensed, exploratory mean difference: -0.053
llm rewrite, bm25, condensed, exploratory permutation p: 0.044
llm rewrite, bm25, condensed, exploratory 99% interval: -0.150 to -0.002
llm rewrite, semantic, test ndcg@10 typed: 0.830
llm rewrite, semantic, test judged@10 typed: 0.990
llm rewrite, semantic, test ndcg@10 rewrite: 0.749
llm rewrite, semantic, test judged@10 rewrite: 0.947
llm rewrite, semantic, mean difference: -0.081
llm rewrite, semantic, wins, losses, ties: 7, 19, 4
llm rewrite, semantic, permutation p: 0.000
llm rewrite, semantic, established at 0.01: yes
llm rewrite, semantic, t-test p: 0.000
llm rewrite, semantic, t-test agrees: yes
llm rewrite, semantic, 99% interval: -0.132 to -0.035
llm rewrite, semantic, interval agrees: yes
llm rewrite, semantic, condensed ndcg@10 typed: 0.834
llm rewrite, semantic, condensed ndcg@10 rewrite: 0.791
llm rewrite, semantic, condensed, exploratory mean difference: -0.043
llm rewrite, semantic, condensed, exploratory permutation p: 0.002
llm rewrite, semantic, condensed, exploratory 99% interval: -0.079 to -0.014
For BM25, nDCG@10 falls from 0.725 to 0.669, a difference that isn't established (planned comparisons). Judged@10 falls too, from 0.927 to 0.903.
Semantic retrieval shows an established loss: nDCG@10 falls from 0.830 to 0.749, with a permutation p-value that rounds to 0.000. Judged@10 falls as well, from 0.990 to 0.947.
Because judged@10 fell, we repeat both comparisons on condensed lists, which drop unjudged products before scoring. That check was chosen after seeing the drop, so it is exploratory and can't establish anything.
For semantic retrieval, condensed nDCG@10 still falls, from 0.834 to 0.791, a difference of -0.043, a permutation p of 0.002, and a 99% interval from -0.079 to -0.014. So the loss doesn't rest on unjudged products alone.
For BM25, condensed nDCG@10 goes from 0.778 to 0.725, with a permutation p of 0.044.
The rewrite didn't help BM25: it moved the other way from what Weller et al. and query2doc report, though not by an established amount. The rewrite's harm to semantic retrieval is what Weller et al. found for strong retrievers. Both results hold for this model, this prompt and these queries.
The queries that move most show what a rewrite does. Read them with the number of unjudged products in each rewritten top 10, which the metric counts as irrelevant:
1"""For each retriever, finds the test queries whose nDCG@10 moves most under the rewrite, and counts the unjudged2products in each rewritten top 10. The prose's point rests on the largest-loss and largest-gain lines, and on the3spelling-only check for BM25's largest gain."""4from bm25s.stopwords import STOPWORDS_EN56from lib.lexical import LexicalSearch7from lib.metrics import evaluate, ndcg_at_k8from lib.queries import rewrite_runs9from lib.recorded import llm_rewrites, probes10from lib.semantic import SemanticSearch11from lib.store import load_products, load_queries, qrels1213queries, judgments = load_queries("test"), qrels("test")14bm25 = LexicalSearch(load_products())15rewrites = llm_rewrites()[0].set_index("query_id")["rewrite"]16largest = {}17for retriever, (rewritten, typed) in rewrite_runs(queries, bm25, SemanticSearch()).items():18 change = evaluate(rewritten, judgments)["ndcg@10"] - evaluate(typed, judgments)["ndcg@10"]19 for kind, query_id in [("loss", change.idxmin()), ("gain", change.idxmax())]:20 label, largest[(retriever, kind)] = f"{retriever}, largest {kind}", query_id21 print(f"{label}: {queries.at[query_id, 'query']}")22 print(f"{label}, change: {change[query_id]:+.3f}")23 print(f"{label}, rewrite: {rewrites[query_id]}")24 unjudged = sum(product_id not in judgments[query_id] for product_id in rewritten[query_id][:10])25 print(f"{label}, unjudged products in the rewrite's top 10: {unjudged}")2627# BM25's largest gain: score the query with only its spelling fixed (chapter 6's probe), apart from the added words28query_id = largest[("bm25", "gain")]29# the spelling probes, keyed by the test query each one corrects30spelled = probes()[0].dropna(subset=["of_query_id"]).set_index("of_query_id")["query"]31if query_id in spelled.index:32 for name, text in [("typed", queries.at[query_id, "query"]), ("spelling fixed only", spelled[query_id])]:33 ranking = list(bm25.search(text, k=10).index)34 print(f"bm25, largest gain, ndcg@10 {name}: {ndcg_at_k(ranking, judgments[query_id], 10):.3f}")35 # bm25s's English stop words, which it drops before matching36 print("bm25 stop words among an and and:", ", ".join(word for word in ["an", "and"] if word in STOPWORDS_EN))
python rewrite-largest-changes.pybm25, largest loss: fold in half table
bm25, largest loss, change: -0.471
bm25, largest loss, rewrite: folding table, collapsible table, drop leaf table, gateleg table
bm25, largest loss, unjudged products in the rewrite's top 10: 3
bm25, largest gain: refrigerator with ice an water in door
bm25, largest gain, change: +0.159
bm25, largest gain, rewrite: refrigerator with ice and water in door, fridge with door dispenser, refrigerator with external water and ice dispenser
bm25, largest gain, unjudged products in the rewrite's top 10: 1
semantic, largest loss: pull out sleeper loveseat
semantic, largest loss, change: -0.304
semantic, largest loss, rewrite: pull out sleeper loveseat sofa couch convertible sofa sofa bed
semantic, largest loss, unjudged products in the rewrite's top 10: 4
semantic, largest gain: nautical platters
semantic, largest gain, change: +0.055
semantic, largest gain, rewrite: nautical platters serving trays seafood platters maritime serving boards
semantic, largest gain, unjudged products in the rewrite's top 10: 0
bm25, largest gain, ndcg@10 typed: 0.501
bm25, largest gain, ndcg@10 spelling fixed only: 0.501
bm25 stop words among an and and: an, and
Both largest losses come from a rewrite that drops or broadens what the shopper asked for. BM25's largest loss, -0.471 for fold in half table, comes from folding table, collapsible table, drop leaf table, gateleg table, which drops in half and adds other kinds of table; 3 of its top 10 are unjudged. Semantic retrieval's largest loss, -0.304 for pull out sleeper loveseat, comes from pull out sleeper loveseat sofa couch convertible sofa sofa bed, which widens a loveseat to any sofa; 4 of its top 10 are unjudged.
BM25's largest gain, +0.159 for refrigerator with ice an water in door, comes from refrigerator with ice and water in door, fridge with door dispenser, refrigerator with external water and ice dispenser. The rewrite fixes an, but with only that fix BM25 scores 0.501, the same as the query as typed, 0.501, because both spellings are stop words. The gain comes from the words the rewrite adds. Semantic retrieval's largest gain is +0.055, for nautical platters, rewritten as nautical platters serving trays seafood platters maritime serving boards.
Working rule: Don't put an LLM rewrite in front of a strong retriever without a planned comparison on the test queries. Read the per-query losses for dropped requirements and broadened products, which the prompt's "keep every requirement" doesn't prevent. The store's semantic retrieval, the ranker to beat, keeps searching the query as typed.
以假設性商品搜尋(HyDE)
查詢改寫是請 LLM 提供更好的查詢,而 HyDE(hypothetical document embeddings 的縮寫)則是請 LLM 提供它設想中能回答該查詢的文件。編碼器會嵌入這份由 LLM 為回答查詢而撰寫的假設性文件,其向量會落在嵌入空間中真實文件所在的鄰近區域(Gao et al., 2023)。虛構的細節不必為真:作者主張編碼器的稠密瓶頸會濾除幻覺產生的細節,但這是他們的論點,而不是量測結果。他們的主要結果使用 InstructGPT(text-davinci-003),以溫度 0.7 取樣。在段落排序基準 TREC DL19 上,HyDE 將 Contriever(一個未使用相關性標註訓練的嵌入模型)的 nDCG@10 在 0 到 100 的尺度上由 44.5 提高到 61.3(Gao et al., Table 1)。對於以標註微調過的 Contriever,結果則取決於所用的 LLM。較小的模型使 nDCG@10 略為下降,因為較弱的指令模型可能損害微調檢索器的整體表現;InstructGPT 則使其提升,在 DL19 上最多,由 62.1 提高到 67.4(Table 4)。
有數種擴展方法會把原始查詢保留在混合之中。HyDE 將假設性文件的嵌入與查詢本身的嵌入一併取平均(Gao et al., section 3.2)。相關性回饋(relevance feedback)根據對最初結果的判斷來改進查詢,而 Rocchio 的版本會把查詢向量移向被判定為相關的結果,同時在加總中保留原始向量(Manning et al., 2008)。對於 BM25,query2doc 會先將查詢重複五次,再接上生成的段落,使原始詞彙保有其權重(Wang et al., 2023)。
HyDE 的錄製資料中,每個查詢有一件假設性商品,由商店的 LLM 僅根據查詢寫成:包含名稱、類別與描述,並以商品的形式嵌入。此處的 hyde 函式將它與查詢的嵌入取平均,並把結果縮放回長度 1,一如第 6 章建議在任何平均之後所做的。這兩個嵌入有一點不同:查詢的嵌入帶有嵌入模型的查詢提示,而商品的嵌入則沒有,與所有商品相同。
hyde(query) = (embedding(hypothetical product) + embedding(query)) / 2, then scaled to length 1
在訓練查詢上,我們也試了只用假設性商品的嵌入;對 BM25,則試了只用假設性商品的文字,以及依 query2doc 的加權方式,將查詢重複五次後再接上該文字。接著我們進行本章最後一項預先規劃的比較:
1"""Scores HyDE, its parts and their combinations with BM25 and semantic retrieval on the train queries, then runs HyDE2against semantic retrieval as the chapter's last planned comparison on the test queries. The prose's point rests on3the train lines for the hypothetical product alone and on the planned comparison's lines."""4import numpy as np56from lib.lexical import LexicalSearch7from lib.metrics import evaluate8from lib.queries import THRESHOLD, report_planned9from lib.recorded import llm_rewrites10from lib.semantic import SemanticSearch11from lib.store import load_products, load_queries, qrels1213# the hypothetical products' embeddings come in the table's row order; zip pairs each with its query_id14table, _, hypothetical_vectors = llm_rewrites()15hypothetical, vectors = table.set_index("query_id")["hypothetical"], dict(zip(table["query_id"], hypothetical_vectors))16bm25, semantic = LexicalSearch(load_products()), SemanticSearch()171819def hyde(query_id):20 """Gao et al.'s average of the hypothetical product's embedding and the query's, scaled back to length 1."""21 average = (vectors[query_id] + semantic.query_vectors[query_id]) / 222 return average / np.linalg.norm(average)232425def runs(queries):26 ids, typed = queries.index, queries["query"]27 return {"semantic": semantic.run(queries),28 "hypothetical alone": {query_id: list(semantic.search_vector(vectors[query_id]).index) for query_id in ids},29 "hyde": {query_id: list(semantic.search_vector(hyde(query_id)).index) for query_id in ids},30 "bm25": bm25.run(queries),31 "bm25, hypothetical alone": bm25.run(queries.assign(query=ids.map(hypothetical))),32 "bm25, query five times plus hypothetical": bm25.run(queries.assign( # query2doc's weighting33 query=[" ".join([typed[query_id]] * 5 + [hypothetical[query_id]]) for query_id in ids]))}343536train = {name: evaluate(run, qrels("train")) for name, run in runs(load_queries("train")).items()}37for name, frame in train.items():38 print(f"train ndcg@10 {name}: {frame['ndcg@10'].mean():.3f}")39 print(f"train judged@10 {name}: {frame['judged@10'].mean():.3f}")40change = (train["hypothetical alone"]["ndcg@10"] - train["semantic"]["ndcg@10"]).round(3) # for choosing, not evidence41print(f"train queries better, worse, hypothetical alone: {(change > 0).sum()}, {(change < 0).sum()}")4243print(f"\nthe planned test comparison, threshold {THRESHOLD:.2f}:")44test = {name: evaluate(run, qrels("test")) for name, run in runs(load_queries("test")).items()45 if name in ["semantic", "hyde"]}46for name, frame in test.items():47 print(f"test judged@10 {name}: {frame['judged@10'].mean():.3f}")48report_planned("hyde, semantic", test["hyde"]["ndcg@10"], test["semantic"]["ndcg@10"])
python hyde-train-and-test.pytrain ndcg@10 semantic: 0.772
train judged@10 semantic: 0.964
train ndcg@10 hypothetical alone: 0.704
train judged@10 hypothetical alone: 0.890
train ndcg@10 hyde: 0.769
train judged@10 hyde: 0.946
train ndcg@10 bm25: 0.695
train judged@10 bm25: 0.930
train ndcg@10 bm25, hypothetical alone: 0.644
train judged@10 bm25, hypothetical alone: 0.854
train ndcg@10 bm25, query five times plus hypothetical: 0.694
train judged@10 bm25, query five times plus hypothetical: 0.912
train queries better, worse, hypothetical alone: 13, 29
the planned test comparison, threshold 0.01:
test judged@10 semantic: 0.990
test judged@10 hyde: 0.970
hyde, semantic, mean difference: -0.031
hyde, semantic, wins, losses, ties: 8, 14, 8
hyde, semantic, permutation p: 0.108
hyde, semantic, established at 0.01: no
hyde, semantic, t-test p: 0.121
hyde, semantic, t-test agrees: yes
hyde, semantic, 99% interval: -0.115 to 0.001
hyde, semantic, interval agrees: yes
在訓練查詢上,單獨使用假設性商品,使語意檢索的 nDCG@10 從 0.772 降到 0.704,表現較好與較差的查詢數分別為 13, 29。Judged@10 也隨之下降,從 0.964 降到 0.890。
與查詢取平均後,得分為 0.769,judged@10 為 0.946。BM25 的表現也是如此:僅用查詢為 0.695,僅用假設性文字為 0.644,將查詢重複五次再接上假設性文字則為 0.694;judged@10 依序為 0.930、0.854 與 0.912。
保留查詢可以補回假設性商品所損失的部分,而在訓練查詢上,無論哪一種檢索器,這種組合都沒有比單用查詢更好。訓練集的數字用來引導設計;只有預先規劃的測試比較才算證據。
在測試查詢上,judged@10 由 0.990 變為 0.970,HyDE 與語意檢索之間沒有確立的差異(預先規劃的比較)。這個結果僅屬於每個查詢一件假設性商品的情形,這些商品來自商店的 120 億參數模型 gemma4:12b-it-q4_K_M,並以溫度 0 執行。Gao et al. 取樣自大得多的 LLM,而且只有該 LLM 提升了他們微調過的檢索器。
實用原則: 嘗試 HyDE 時,一律把查詢本身的嵌入平均進去。預期在未使用標註訓練的編碼器上會有最大的增益;對於微調過的編碼器,Gao et al. 只在其最大的 LLM 上看到增益,而商店以其模型、提示與編碼器,則沒有看到任何增益。對於像商店這樣在這些查詢上已經勝過 BM25的編碼器,在為每個查詢付出一次 LLM 呼叫的成本之前,請先測試 HyDE。商店並未採用它。
將後續提問轉成獨立查詢
本章開頭提到的最後一項缺口來自對話:購物者的下一則訊息常常依賴上一則,例如 in white 是在細化對書桌的搜尋。檢索系統只看到一段文字,因此後續提問必須變成獨立查詢(standalone query),也就是不需要先前的對話輪次也能讀懂,並保留購物者仍然想要的條件。對話式查詢改寫(conversational query rewriting)把依賴上下文的問題轉成自足且答案相同的問題,CANARD 資料集收錄了 40,527 筆這類改寫(Elgohary et al., 2019)。在對話式搜尋評測 TREC CAsT 2019 中,人工改寫的效果以相對值計比最佳自動系統高出 35%(Dalton et al., 2020)。後來以提示詞驅動 LLM 的研究發現,「資訊豐富的查詢改寫可以讓檢索效能比人工改寫大幅提升,特別是搭配稀疏檢索器時」(Ye et al., 2023);在 CAsT 的後繼評測 TREC iKAT 2025 中,多數參賽執行結果在流程中使用 LLM,進行單一或多重查詢改寫(TREC iKAT 2025 overview)。
WANDS 沒有對話,所以我們為本書撰寫了商店的對話:12 則後續提問,每則以一個屬性細化測試查詢,屬性包括顏色、形狀、尺寸或表面處理。商店的 LLM 只看到這兩輪對話,並把每則改寫成獨立查詢。例如在 coat rack for bedroom 之後接著 something in black,改寫後為 black coat rack for bedroom。判斷資料是針對第一則查詢而非細化後的查詢對產品分級,因此 nDCG 無法為後續提問評分。我們改為在每個前 10 名中計算三類產品:保留細化條件的產品,也就是其 attributes 或文字含有該值,例如 black(範例也會印出僅由 attributes 計得的數量);屬於正確種類的產品,也就是評審針對第一則查詢評為完全相符或部分相符者;以及兩者兼具的產品。我們把單獨的後續提問,與兩輪對話串接後的查詢及獨立查詢互相比較,每一種都以查詢方式嵌入,並以第一則查詢作為基準。我們也在獨立查詢的前 100 名上加一道篩選,只保留明確載有該值的產品,該值取自記錄下來的對話;實際運作的系統則必須從後續提問中自行擷取。
1"""Counts, in each conversation's top 10, the products that keep the refinement and the products of the right kind, for2the follow-up alone, the joined turns, the standalone query, and the standalone query filtered by the value. The prose's3point rests on the "refinement in text or attributes" and "right kind with the refinement" lines."""4import numpy as np56from lib.queries import mentions7from lib.recorded import llm_conversations8from lib.semantic import SemanticSearch9from lib.store import load_products, load_queries, product_text, qrels1011products, judgments, semantic = load_products(), qrels("test"), SemanticSearch()12attributes, text = products["attributes"], product_text(products) # the embeddings read only the text13conversations, vectors = llm_conversations() # follow-ups we wrote for the book, not shoppers'14print("conversations:", len(conversations))15# row 4, the fifth conversation, which the example prints16example = conversations.iloc[4]17print(f"first query: {load_queries().at[example['query_id'], 'query']}")18print(f"follow-up: {example['follow_up']}")19print(f"standalone: {example['standalone']}")2021shares, filtered_lengths = {}, [] # shares: (top 10, measure) -> one share per conversation22for row, (first, value) in enumerate(zip(conversations["query_id"], conversations["value"])):23 # the standalone query's top 100, the pool the filter draws from24 standalone = semantic.search_vector(vectors["standalone"][row], k=100).index25 stated = mentions(standalone, value, attributes) | mentions(standalone, value, text)26 tops = {"first query": semantic.search(first, k=10).index,27 "follow-up alone": semantic.search_vector(vectors["follow_up"][row], k=10).index,28 "turns joined": semantic.search_vector(vectors["joined"][row], k=10).index,29 "standalone": standalone[:10],30 "standalone, filtered": standalone[stated][:10]} # given the value, which a live system extracts31 filtered_lengths.append(len(tops["standalone, filtered"]))32 for name, top in tops.items():33 in_attributes = mentions(top, value, attributes)34 in_either = in_attributes | mentions(top, value, text)35 # the first query's grade 1 (partial) or 2 (exact); a product with no judgment counts as 036 right_kind = np.array([judgments[first].get(product_id, 0) >= 1 for product_id in top], dtype=bool)37 for measure, hits in [("refinement in attributes", in_attributes), ("refinement in text or attributes", in_either),38 ("right kind", right_kind), ("right kind with the refinement", in_either & right_kind)]:39 shares.setdefault((name, measure), []).append(hits.sum() / 10) # empty places count as misses40for (name, measure), values in shares.items():41 print(f"{name}, {measure}: {np.mean(values):.3f}")4243kept = np.subtract(shares["standalone", "refinement in text or attributes"], shares["turns joined", "refinement in text or attributes"])44print(f"standalone against turns joined, more, fewer, same: {(kept > 0).sum()}, {(kept < 0).sum()}, {(kept == 0).sum()}")45print("conversations the filter leaves with fewer than 10 products:", sum(length < 10 for length in filtered_lengths))
python follow-up-refinements.pyconversations: 12
first query: coat rack for bedroom
follow-up: something in black
standalone: black coat rack for bedroom
first query, refinement in attributes: 0.125
first query, refinement in text or attributes: 0.200
first query, right kind: 0.958
first query, right kind with the refinement: 0.175
follow-up alone, refinement in attributes: 0.433
follow-up alone, refinement in text or attributes: 0.950
follow-up alone, right kind: 0.058
follow-up alone, right kind with the refinement: 0.058
turns joined, refinement in attributes: 0.208
turns joined, refinement in text or attributes: 0.383
turns joined, right kind: 0.933
turns joined, right kind with the refinement: 0.325
standalone, refinement in attributes: 0.200
standalone, refinement in text or attributes: 0.392
standalone, right kind: 0.958
standalone, right kind with the refinement: 0.350
standalone, filtered, refinement in attributes: 0.583
standalone, filtered, refinement in text or attributes: 0.900
standalone, filtered, right kind: 0.675
standalone, filtered, right kind with the refinement: 0.675
standalone against turns joined, more, fewer, same: 5, 4, 3
conversations the filter leaves with fewer than 10 products: 3
單獨的後續提問在前 10 名中有 0.950 保留了細化條件,但只有 0.058 是正確種類的產品。任何帶有第一則查詢的形式都能找到正確種類的產品,串接對話為 0.933,獨立查詢為 0.958。
在這些對話上,串接對話與獨立查詢保留細化條件的頻率都高於從未要求該條件的第一則查詢:分別為 0.383 與 0.392,而第一則查詢為 0.200。
若只以 attributes 計算,比例較低,分別為 0.208、0.200 與 0.125,因為許多產品只在文字中載明該值,而嵌入只讀取文字。以下結論根據「文字或屬性」的計數,這種計法可能高估,因為提到 round legs 的產品也會被算作圓形。
LLM 的改寫並未比串接對話更好:與串接對話相比,獨立查詢在 5, 4, 3 則對話中保留的細化產品分別是較多、較少或相同;而同時屬於正確種類且保留細化條件的產品,在獨立查詢的前 10 名中占 0.350,串接對話則為 0.325。
篩選把這個比例提高到 0.675,但使 3 則對話的產品少於 10 個。
這 12 則對話是我們自己撰寫的,並非購物者的對話,所以這些計數只是診斷,不是檢驗。
實用原則: 絕不要單獨搜尋後續提問。至少要把它與先前的對話輪次串接,或改寫成獨立查詢。當細化條件指出目錄中載明的某個值時,擷取該值並據此加權或篩選,剩餘產品過少時則退回未篩選的清單。在信任上述任何做法之前,先以自己日誌中的對話進行衡量。
正式環境中會出什麼問題
**每一次 LLM 查詢改寫,都是查詢路徑上的一次模型呼叫。**範例讀取的改寫結果是離線一次記錄下來的,但實際運作的搜尋框會在每次查詢時呼叫模型,購物者則必須等待。query2doc 的作者指出,該方法需要 LLM 推論,速度可能慢上許多(Wang et al., 2023),而且較長的查詢也會拖慢 BM25 搜尋。以下是經驗法則,而非實測結果:對高頻查詢,應離線改寫並快取結果,只有在測試顯示確有成效的查詢上,才在線上呼叫模型。
**一則改寫只對應特定的一個模型與一個提示詞。**換用另一個模型版本,或改動提示詞的措辭,都可能改變任何一筆改寫。商店錄製資料的資訊清單中,保存了模型的摘要值與每一個提示詞的原文,只要其中任一項有變動,就必須重新評量。
**詞彙表必須跟著商品目錄更新。**以上個月的商品目錄建立的拼字校正器,可能把這個月的品牌名稱當成拼字錯誤。請隨索引一併重建詞彙表,並記錄校正器更動過的每一筆查詢及其原始查詢,以便找出過度校正的情形,例如 tye → the。在機器翻譯產生的查詢中,拼字錯誤會改變搜尋回傳的商品(Hanneman et al., 2024),因此校正器漏掉的錯誤,預期會表現為單一查詢的結果不穩定。
**分類器隨其分類體系而變。**商品類別被改名、拆分或合併時,必須重新訓練分類器。若某個查詢類別沒有對應到任何商品類別,例如 Crock Pots & Slow Cookers,就永遠不可能被預測出來;對於 7qt slow cooker,分類器會改為預測 Slow Cookers。
**改寫後的查詢會帶出未經判定的商品。**本章中每一項以 nDCG@10 評分的 LLM 改動,都降低了 judged@10。在採用某種改寫之前,應請人判定其前 10 名中尚未判定的商品,或像第 2 章那樣檢查濃縮(condensed)分數。
**實用原則:**記錄使用者輸入的原始查詢與改寫後的查詢,快取 LLM 改寫結果,將模型與提示詞連同輸出一併版本化,並在商品目錄每次變動時重建詞彙表與分類器。
本章做出的決定
下列每項設定都是依據訓練查詢選定的。表中列出本章在測試查詢上進行的 5 項預先規劃的比較,每欄一項:一個只改動一處的檢索器,對照同一個以使用者原始輸入查詢的檢索器。
| 指標 | BM25、同義詞 | 語意檢索、類別加權 | BM25、LLM 改寫 | 語意檢索、LLM 改寫 | 語意檢索、HyDE |
|---|---|---|---|---|---|
| 平均差異 | 0.000 | 0.005 | -0.056 | -0.081 | -0.031 |
| 勝、負、平手 | 0, 0, 30 | 7, 4, 19 | 13, 15, 2 | 7, 19, 4 | 8, 14, 8 |
| 排列檢定 p 值 | 1.000 | 0.174 | 0.060 | 0.000 | 0.108 |
| 是否確立 | no | no | no | yes | no |
| t 檢定 p 值 | 1.000 | 0.164 | 0.060 | 0.000 | 0.121 |
| 99% 區間 | 0.000 to 0.000 | -0.002 to 0.017 | -0.143 to 0.005 | -0.132 to -0.035 | -0.115 to 0.001 |
| t 檢定是否一致 | yes | yes | yes | yes | yes |
| 區間是否一致 | yes | yes | yes | yes | yes |
- 拼字:只修正商品目錄未使用的字詞。 拼字校正器能修正商店訓練查詢中的錯字,但錯字查詢數量太少,無法加以測試,而且它需要上下文才能處理
tye dye或ice an water。商店目前尚未啟用它。 - 單位:商品與查詢採用同一種書寫形式,取自商品目錄本身的拼法, 並完整保留小數。尺寸查詢太少,無法測試,因此基準 BM25 維持原樣。
- 同義詞:針對訓練查詢中發現的缺口建立一份清單, 僅用於 BM25。它改變了 1 個測試查詢,且未顯示出已確立的差異,因此基準不做擴展。
- 查詢類別:以商品文字訓練的樸素貝氏分類器, 因為樣本中每個類別至多只有一個查詢。預測出的類別只是一項提示。篩選器在訓練查詢上表現較差,而最佳的加權在測試查詢上並未確立,因此語意檢索維持不加權。
- LLM 改寫:不採用。 對語意檢索而言,退步已經確立,而 BM25 的變化則未確立。
- HyDE:不採用。 與查詢取平均後,沒有顯示出已確立的差異,而單獨使用假設商品則在訓練查詢上表現較差。
- 後續提問:絕不單獨搜尋。 將各輪對話串接起來,效果與 LLM 產生的獨立查詢相當。針對精煉後的值設定篩選器是下一步,這需要有擷取出的值與備援做法。
- 檢索器以使用者原始輸入的查詢進行檢索: 語意檢索仍是要超越的排序器,未做詞幹還原的 BM25 仍是基準。
- 本章所呈現的: 在訓練查詢上看起來正確的改動,必須通過測試查詢上預先規劃的比較,而本章測試的改動中,沒有一項以進步的形式通過。LLM 的改寫是唯一已確立的差異,而且對語意檢索而言是退步。
本章,於本版次中
版次 2026.11.10範例於 2026年11 月 19 通過測試本章 12 個範例 個範例中有 12 個通過
譯自英文書,版次 2026.11.10。