Understanding queries

By the end of this chapter you can change a shopper's query before retrieval: correct its spelling, write its units one way, expand it with synonyms, predict its product class, rewrite it with a large language model, search with a product that model imagines, and turn a follow-up into a standalone query. You can also decide, from human judgments, whether each change helps the retriever it feeds.

The store whose search this book follows, an online seller of furniture and home goods (chapter 1), gets short queries, and some of them can't match the products the shopper wants. ligth bulb misspells a word. 7qt slow cooker writes its size as a term that 0 products contain. For midcentury tv unit, chapter 7 counts 25 exact products, of which 0 contain midcentury and 0 contain unit. A shopper who types in white after a search for a desk means a white desk, but the words alone don't say so. Query understanding is everything a search system does to a query before retrieval to close gaps like these: it corrects, normalizes, expands, classifies and rewrites. Chapters 5 to 7 changed the retrievers. This chapter changes the query and keeps the retrievers fixed: BM25, the book's unstemmed baseline in lib.lexical, and semantic retrieval, which compares the query's embedding with each product's, both computed in advance by Snowflake/snowflake-arctic-embed-m-v1.5.

Three of the techniques use a large language model (LLM), a model trained on large amounts of text to generate text in response to a prompt. The store's LLM is gemma4:12b-it-q4_K_M, an open model that Google publishes under the Apache 2.0 licence (model card). The examples never call it. Two scripts, recordings/llm_rewrites.py and recordings/llm_conversations.py, ran it once, at temperature 0 with a fixed seed, and saved its outputs as recordings, which the examples read through lib.recorded. Each recording's manifest.json holds the full prompts. We chose the model because anyone can download it and run it on their own machine, so anyone can rerun the recordings. The LLM sees only the shopper's words, never a product. Every score in the chapter uses the human judgments.

Every setting is chosen on the 50 train queries, and the 30 test queries are kept for reporting. Before reading any test result, the chapter fixes 5 planned comparisons on the test queries, each against the same retriever on the query as typed:

1. BM25 with a synonym list
2. semantic retrieval with a boost for the predicted class
3. BM25 on the LLM's rewrite
4. semantic retrieval on the LLM's rewrite
5. semantic retrieval with HyDE, which averages the query's embedding with that of a product the LLM imagines

By the book's rule, a difference is established when its paired permutation test gives a p-value below 0.01, 0.05 divided by the number of planned comparisons. The paired t-test and a 99% bootstrap interval stand beside it. Spelling correction, unit normalization and follow-up rewriting each touch too few queries to test, so their sections are diagnoses, not tests.

Correct spelling against the catalog's words 

The first gap is a typo, as in ligth bulb. In BM25, a misspelled word matches only products with the same misspelling, and it can move the query's embedding away from the products it means, as chapter 6 shows. Search queries hold many such errors: one study of web search reports that "Roughly 10-15% of the queries sent to search engines contain errors" (Cucerzan and Brill, 2004). The classic corrector is the noisy channel model: for a typed word t, choose the correction c that maximizes P(c) × P(t | c) (Kernighan et al., 1990). P(c) says how common the candidate word is. P(t | c), the error model, says how likely a shopper who meant c is to type t. Kernighan et al. take as candidates the words one edit from the typo: a deletion, an insertion, a replacement or a swap of two neighbouring letters. The number of insertions, deletions and replacements that turn one string into another is their edit distance (Manning et al., 2008), and the corrector here also counts a swap as one edit. A textbook states the two principles in plain terms: of the correct spellings, choose the nearest, and between near ties, choose the more common (Manning et al., 2008). Norvig writes such a corrector in a few lines (Norvig).

correction(t) = argmax over candidates c of  P(c) × P(t | c)

The lexicon, the list of words that count as correct, decides what gets corrected. Web search engines take it from their query logs, choosing "the correction that is most common among queries typed in by other users" (Manning et al.). The store has no log, so its lexicon comes from the catalog's text. Where the lexicon comes from matters for brand names: in product search, a corrector "must distinguish genuine misspellings from novel brand names that use unconventional spelling" (Guo et al., 2024).

In the store, build the corrector with the crudest error model: a word in the lexicon stays, one edit beats two, and P(c), how common each candidate is, breaks ties between candidates the same number of edits away. Run it on all 80 queries:

1import re2from collections import Counter34from lib.queries import correct_query5from lib.store import load_products, load_queries, product_text67texts = product_text(load_products()).str.lower()8counts = Counter(re.findall(r"[a-z]+", " ".join(texts)))       # the lexicon, and how common each word is91011def edits1(word):12    """Every string one edit away: a deletion, a swap of two neighbouring letters, a replacement or an insertion."""13    splits = [(word[:cut], word[cut:]) for cut in range(len(word) + 1)]14    letters = "abcdefghijklmnopqrstuvwxyz"15    return ({left + right[1:] for left, right in splits if right}16            | {left + right[1] + right[0] + right[2:] for left, right in splits if len(right) > 1}17            | {left + letter + right[1:] for left, right in splits if right for letter in letters}18            | {left + letter + right for left, right in splits for letter in letters})192021def correct(word):22    """Keep a word the catalog uses. Otherwise take the commonest catalog word one edit away, else two."""23    if not word.isalpha() or word in counts:24        return word25    candidates = {edit for edit in edits1(word) if edit in counts} or {26        two for edit in edits1(word) for two in edits1(edit) if two in counts}27    return max(sorted(candidates), key=counts.get) if candidates else word282930queries = load_queries()31corrected = queries["query"].map(lambda query: " ".join(correct(word) for word in query.lower().split()))32changed = queries[corrected != queries["query"].str.lower()]33print(f"catalog words: {len(counts):,}")34print("queries changed:", len(changed))35print("of them, train queries:", (changed["split"] == "train").sum())36for query_id, query in changed["query"].items():37    print(f"  {query} ({changed.at[query_id, 'split']}), corrected: {corrected[query_id]}")38same = corrected.equals(queries["query"].map(lambda query: correct_query(query, counts)))39print("same as lib.queries.correct_query:", same)40for word in ["tye", "tie", "the", "kohler", "whitehaven", "malachi", "luau"]:      # P(c), as counts41    print(f"catalog count of {word}:", counts[word])

catalog words: 19,680
queries changed: 4
of them, train queries: 3
  foutains with brick look (train), corrected: fountains with brick look
  twin over full bunk beds cool desins (train), corrected: twin over full bunk beds cool designs
  ligth bulb (train), corrected: light bulb
  tye dye duvet cover (test), corrected: the dye duvet cover
same as lib.queries.correct_query: True
catalog count of tye: 0
catalog count of tie: 45
catalog count of the: 20389
catalog count of kohler: 71
catalog count of whitehaven: 60
catalog count of malachi: 5
catalog count of luau: 4

The catalog uses 19,680 distinct words. The corrector changes 4 queries, and the shared function lib.queries.correct_query makes the same corrections. Of those, 3 are train queries, and the corrector spells each one as chapter 6's spelling probes do: the correctly spelled copies of the typo queries that chapter 6 wrote for the book. The corrector leaves the brands alone, because the catalog uses them: kohler 71 times and whitehaven 60. Some of the words it keeps are rare, such as malachi, used 5 times, and luau, used 4. A lexicon that left out rare words would treat both as typos.

The corrector's failures are in test queries. tye dye duvet cover becomes the dye duvet cover: tye is one edit from tie (45 times in the catalog) and from the (20389), and P(c) picks the common word. And refrigerator with ice an water in door keeps its an. That is a real-word error: every word is spelled right, as in "flew form Heathrow" (Manning et al., 2008). Both need context. For tye and for an, count how often each catalog word within one edit of it comes right before the next word of the query, dye or water:

1import re2from collections import Counter34from lib.queries import correct_word, edits1, word_counts5from lib.store import load_products, product_text67texts = product_text(load_products()).str.lower()8counts = word_counts(texts)                                   # the corrector's lexicon9words_per_product = [re.findall(r"[a-z]+", text) for text in texts]10pairs = Counter(pair for words in words_per_product for pair in zip(words, words[1:]))      # neighbouring words1112# tye in "tye dye duvet cover", an in "refrigerator with ice an water in door": each word, the word after it13for word, after in [("tye", "dye"), ("an", "water")]:14    options = sorted(((pairs[(edit, after)], edit) for edit in edits1(word) | {word} if edit in counts), reverse=True)15    for count, edit in options[:3]:16        print(f"catalog count of {edit} {after}:", count)17    choice = correct_word(word, counts)                        # what the corrector without context picks18    print(f"the corrector's choice for {word}: {choice}")19    print(f"catalog count of {choice} {after}, the corrector's choice:", pairs[(choice, after)])

catalog count of tie dye: 18
catalog count of type dye: 0
catalog count of tyke dye: 0
the corrector's choice for tye: the
catalog count of the dye, the corrector's choice: 0
catalog count of and water: 59
catalog count of a water: 20
catalog count of in water: 9
the corrector's choice for an: an
catalog count of an water, the corrector's choice: 0

Before dye, the catalog's text has tie 18 times, and the corrector's choice, the, 0. Before water, it has and 59 times, ahead of a at 20, and an 0. For these two queries, a corrector that scored each candidate with its neighbouring word would pick tie and and. Now measure each corrected query on its own judgments:

1from lib.lexical import LexicalSearch2from lib.metrics import ndcg_at_k3from lib.queries import correct_query, word_counts4from lib.recorded import probes5from lib.semantic import SemanticSearch6from lib.store import load_products, load_queries, product_text, qrels78products, queries, judgments = load_products(), load_queries(), qrels()9counts = word_counts(product_text(products))10bm25, semantic = LexicalSearch(products), SemanticSearch()11probe_queries, probe_vectors = probes()      # recorded embeddings of the four typo queries, spelled correctly12spelling = probe_queries[probe_queries["kind"] == "spelling"]13spelled = {row["query"]: probe_vectors[index] for index, row in spelling.iterrows()}14typo_queries = spelling["of_query_id"].astype(int).tolist()15print("typo queries:", len(typo_queries))16print("of them, test queries:", (queries.loc[typo_queries, "split"] == "test").sum())1718for query_id in typo_queries + [459]:        # the typo queries, then the one variant spelling, tye dye19    typed = queries.at[query_id, "query"]20    corrected = correct_query(typed, counts)21    label = f"{query_id} ({queries.at[query_id, 'split']})"22    print(f"{label}: {typed} -> {corrected}")23    for name, text in [("typed", typed), ("corrected", corrected)]:24        ranking = list(bm25.search(text, k=10).index)25        print(f"{label}, bm25 ndcg@10 {name}: {ndcg_at_k(ranking, judgments[query_id], 10):.3f}")26    if corrected == typed.lower():27        print(f"{label}: the corrector changes nothing")28    elif corrected in spelled:29        for name, vector in [("typed", semantic.query_vectors[query_id]), ("corrected", spelled[corrected])]:30            ranking = list(semantic.search_vector(vector, k=10).index)31            print(f"{label}, semantic ndcg@10 {name}: {ndcg_at_k(ranking, judgments[query_id], 10):.3f}")32    else:33        print(f"{label}: no recorded embedding of the corrected text")

typo queries: 4
of them, test queries: 1
305 (train): ligth bulb -> light bulb
305 (train), bm25 ndcg@10 typed: 0.945
305 (train), bm25 ndcg@10 corrected: 0.945
305 (train), semantic ndcg@10 typed: 1.000
305 (train), semantic ndcg@10 corrected: 1.000
20 (train): foutains with brick look -> fountains with brick look
20 (train), bm25 ndcg@10 typed: 0.356
20 (train), bm25 ndcg@10 corrected: 0.507
20 (train), semantic ndcg@10 typed: 0.353
20 (train), semantic ndcg@10 corrected: 0.645
224 (train): twin over full bunk beds cool desins -> twin over full bunk beds cool designs
224 (train), bm25 ndcg@10 typed: 0.488
224 (train), bm25 ndcg@10 corrected: 0.488
224 (train), semantic ndcg@10 typed: 0.565
224 (train), semantic ndcg@10 corrected: 0.616
406 (test): refrigerator with ice an water in door -> refrigerator with ice an water in door
406 (test), bm25 ndcg@10 typed: 0.501
406 (test), bm25 ndcg@10 corrected: 0.501
406 (test): the corrector changes nothing
459 (test): tye dye duvet cover -> the dye duvet cover
459 (test), bm25 ndcg@10 typed: 0.953
459 (test), bm25 ndcg@10 corrected: 0.953
459 (test): no recorded embedding of the corrected text

For the train query foutains with brick look, BM25's nDCG@10 goes from 0.356 to 0.507 and semantic retrieval from 0.353 to 0.645. For twin over full bunk beds cool desins, only semantic retrieval moves, from 0.565 to 0.616, and for ligth bulb neither does. For refrigerator with ice an water in door, the corrector changes nothing, and a correct fix couldn't help BM25 either. Of the typed an and the correct and, bm25s drops an, and as stop words, and chapter 6's copy of the query spelled with and scores 0.501, the same as the query as typed. For tye dye duvet cover, BM25 scores 0.953 either way, because the is a stop word and tye matches no product, so both versions search the same terms. Semantic retrieval can't be scored on the corrected text, because the recordings hold no embedding of it. The store has 4 typo queries, 1 of them a test query, and the smallest two-sided permutation p-value that so few could give is 0.125.

What to do: build the lexicon from your catalog, and from your query log once you have one, and correct only words it doesn't hold, so that brand names survive. Before you correct a word, check its neighbours: a corrector without context turns variant spellings into common words and misses real-word errors. Score each correction on the typo queries' own judgments, and collect many more typo queries than the store has before you test a corrector.

Normalize units the way the catalog writes them 

A typo has one right spelling; a size, as in 7qt slow cooker, often has several. Normalization maps the different ways of writing one thing to a single form, on the products and on the query alike. In the store, sizes need it. Chapter 5 found that 7qt and 7 qt never match each other, because the tokenizer drops the one-character 7. A normalizer that joins each number to its unit, written one way, turns both into one term, 7qt. It has to run on the product text when you build the index, and on every query. It also has to produce terms the tokenizer keeps whole. The tokenizer splits words at a decimal point, so 3.7qt would become 7qt. To prevent that, the shared function lib.queries.normalize_units writes the point as an underscore, which the tokenizer keeps inside a word. It also leaves in alone when a number or one follows, as in 3 in 1.

In the store, a rule written from the queries would cover qt, in, inch and ". Before you trust it, count how the catalog writes inches. The example counts them, prints the terms that the tokenizer makes of a few normalized texts, then builds BM25 three times: over the text as written, normalized by that first rule, and normalized by normalize_units, which also handles the catalog's two apostrophes, '', for inches. It scores every query, train or test, that gives a number with a unit:

1from lib.lexical import LexicalSearch2from lib.lexical_by_hand import tokenize3from lib.metrics import ndcg_at_k4from lib.queries import INCH_SPELLINGS, normalize_units, unit_rules5from lib.store import load_products, load_queries, product_text, qrels67products, queries, judgments = load_products(), load_queries(), qrels()8text = product_text(products).str.lower()9for name, unit in [("''", r"''"), ('"', r"\""), ("inch", r"-?inch")]:10    print(f"products that write a number then {name}:", text.str.contains(rf"\d\s*{unit}").sum())1112first_try = unit_rules([spelling for spelling in INCH_SPELLINGS if spelling != "''"])   # written from queries13rules = {"as written": None, "first try": first_try, "normalized": unit_rules()}14checks = [("3.7-quart slow cooker", "normalized"), ("3 in 1 sofa bed", "normalized"), ("1 in twin", "normalized"),15          ("48 '' range hood", "first try"), ("48 '' range hood", "normalized")]16for example, name in checks:                                                   # what the tokenizer reads17    print(f"terms of {example}, {name}:", " ".join(tokenize(normalize_units(example, rules[name]))))1819searches = {}20for name, rule in rules.items():21    rewritten = products.copy()22    if rule is not None:23        for column in ["name", "product_class", "category", "description"]:   # the columns product_text joins24            rewritten[column] = rewritten[column].map(lambda value: normalize_units(value, rule))25    searches[name] = LexicalSearch(rewritten)2627with_units = queries[queries["query"].str.contains(r"\d\s*(?:qt|in\b|inch|\")")]28print("queries with a unit:", len(with_units))29print("of them, test queries:", (with_units["split"] == "test").sum())30for query_id, query in with_units["query"].items():31    label = f"{query_id} ({queries.at[query_id, 'split']})"32    print(f"{label}: {query} -> {normalize_units(query)}")33    for name, rule in rules.items():34        typed = query if rule is None else normalize_units(query, rule)35        ranking = list(searches[name].search(typed, k=10).index)36        print(f"{label}, bm25 ndcg@10 {name}: {ndcg_at_k(ranking, judgments[query_id], 10):.3f}")

products that write a number then '': 3014
products that write a number then ": 65
products that write a number then inch: 215
terms of 3.7-quart slow cooker, normalized: 3_7qt slow cooker
terms of 3 in 1 sofa bed, normalized: sofa bed
terms of 1 in twin, normalized: 1in twin
terms of 48 '' range hood, first try: 48 range hood
terms of 48 '' range hood, normalized: 48in range hood
queries with a unit: 4
of them, test queries: 2
62 (train): 7qt slow cooker -> 7qt slow cooker
62 (train), bm25 ndcg@10 as written: 0.554
62 (train), bm25 ndcg@10 first try: 0.930
62 (train), bm25 ndcg@10 normalized: 0.930
101 (train): 48 in entry table with side by side drawer -> 48in entry table with side by side drawer
101 (train), bm25 ndcg@10 as written: 0.569
101 (train), bm25 ndcg@10 first try: 0.569
101 (train), bm25 ndcg@10 normalized: 0.569
285 (test): 48" sliding single track , barn door for laundry -> 48in sliding single track , barn door for laundry
285 (test), bm25 ndcg@10 as written: 0.592
285 (test), bm25 ndcg@10 first try: 0.592
285 (test), bm25 ndcg@10 normalized: 0.592
482 (test): 48 inch kitchen hood -> 48in kitchen hood
482 (test), bm25 ndcg@10 as written: 1.000
482 (test), bm25 ndcg@10 first try: 0.824
482 (test), bm25 ndcg@10 normalized: 1.000

The catalog writes a number followed by two apostrophes in 3014 products, by " in 65 and by inch in 215. The tokenizer reads 3.7-quart slow cooker, normalized, as 3_7qt slow cooker, and 3 in 1 sofa bed as sofa bed. normalize_units still misreads 1 in twin as 1in twin, which is why the terms are worth printing.

For the train query 7qt slow cooker, normalizing takes BM25's nDCG@10 from 0.554 to 0.930, and the first rule gives 0.930. The test query 48 inch kitchen hood shows what the first rule loses by missing the two apostrophes: it scores 1.000 as written and 0.824 under the first rule. The query becomes 48in, while the first rule leaves a hood written 48 '' as 48 range hood, and the full rule makes it 48in range hood. With the full rule, the hood query scores 1.000. The other train query, 48 in entry table with side by side drawer, scores 0.569 as written and 0.569 normalized, and the test query about a barn door 0.592 and 0.592. That makes 4 queries with a unit, 2 of them test queries: too few for a test, so this is a diagnosis.

What to do: write normalization rules from the catalog's text, not from the queries, by counting every way it writes each unit. Print the terms the tokenizer makes of the normalized text, because a rule that looks right can produce terms that match the wrong products. Apply the same rules to products and queries in one release, and rebuild the index. When the catalog keeps sizes in fields, such as the width in attributes, match the size against the field instead, as chapter 6 suggests.

Expand queries with a synonym list 

Some gaps are neither typos nor units but different words for one thing, as when the exact products for midcentury tv unit say stand. Query expansion adds terms to a query so that it can match products that use other words for the same thing. "Use of query expansion generally increases recall" (Manning et al., 2008), the share of the relevant products that a search finds. It can cost precision, the share of found products that are relevant: "query expansion may also significantly decrease precision, particularly when the query contains ambiguous terms" (Manning et al., 2008). An added word can match products the shopper didn't ask for, and move the query away from its meaning, which is called query drift. Expansion terms can come from a hand-built list of synonyms, or from the results. Pseudo-relevance feedback assumes the top results are relevant and adds their terms to the query, which makes drift its known risk (Manning et al., 2008).

In the store, lib.queries.SYNONYMS holds 14 groups of interchangeable words, such as stand and unit, written by reading the train queries whose exact products miss a query word. When a query contains a word or phrase from a group, expand appends the group's other members, and BM25 searches the result. Measure it on the train queries, then make the first of the chapter's five planned comparisons on the test queries, which counts as established if p is below 0.01:

1from lib.lexical import LexicalSearch2from lib.metrics import evaluate3from lib.queries import PLANNED_TEST_COMPARISONS, SYNONYMS, THRESHOLD, expand, report_planned4from lib.store import load_products, load_queries, qrels56products, judgments = load_products(), qrels()7bm25 = LexicalSearch(products)8print("synonym groups:", len(SYNONYMS))9scores = {}10for split in ["train", "test"]:11    queries = load_queries(split)12    expanded = queries.assign(query=queries["query"].map(expand))13    changed = queries.index[expanded["query"] != queries["query"]]14    print(f"{split} queries the list changes:", len(changed))15    for name, frame in [("typed", queries), ("expanded", expanded)]:16        scores[split, name] = evaluate(bm25.run(frame), qrels(split))17        print(f"{split} ndcg@10 {name}: {scores[split, name]['ndcg@10'].mean():.3f}")18        print(f"{split} judged@10 {name}: {scores[split, name]['judged@10'].mean():.3f}")19for query in queries.loc[changed, "query"]:                   # the test queries the list changes20    print(f"test query changed: {query}")21    print(f"words the list adds to it: {expand(query)[len(query) + 1:]}")2223change = (scores["train", "expanded"]["ndcg@10"] - scores["train", "typed"]["ndcg@10"]).round(3)24print(f"train queries better, worse: {(change > 0).sum()}, {(change < 0).sum()}")       # for choosing, not evidence25print(f"share of the net train gain from the largest gain: {change.max() / change.sum():.3f}")26for kind, query_id in [("gain", change.idxmax()), ("loss", change.idxmin())]:27    query = load_queries().at[query_id, "query"]28    added = expand(query)[len(query) + 1:]29    print(f"largest train {kind}: {query}")30    print(f"largest train {kind}, words added: {added}")31    print(f"largest train {kind}, change: {change[query_id]:+.3f}")32named = [product_id for product_id in bm25.search(expand(query), k=10).index if added in products.at[product_id, "name"]]33print(f"largest train loss, top 10 names with {added}:", len(named))      # query and added: the largest loss's34print("largest train loss, of those, unjudged:", sum(product_id not in judgments[query_id] for product_id in named))3536print("\nplanned test comparisons:", len(PLANNED_TEST_COMPARISONS))37print(f"p-value threshold: {THRESHOLD:.2f}")38report_planned("synonyms, bm25", scores["test", "expanded"]["ndcg@10"], scores["test", "typed"]["ndcg@10"])

synonym groups: 14
train queries the list changes: 14
train ndcg@10 typed: 0.695
train judged@10 typed: 0.930
train ndcg@10 expanded: 0.722
train judged@10 expanded: 0.944
test queries the list changes: 1
test ndcg@10 typed: 0.725
test judged@10 typed: 0.927
test ndcg@10 expanded: 0.725
test judged@10 expanded: 0.927
test query changed: indoor multi tiered plant stand
words the list adds to it: unit
train queries better, worse: 6, 5
share of the net train gain from the largest gain: 0.473
largest train gain: midcentury tv unit
largest train gain, words added: stand mid century
largest train gain, change: +0.632
largest train loss: wire basket with dividers
largest train loss, words added: hamper
largest train loss, change: -0.176
largest train loss, top 10 names with hamper: 4
largest train loss, of those, unjudged: 4

planned test comparisons: 5
p-value threshold: 0.01
synonyms, bm25, mean difference: 0.000
synonyms, bm25, wins, losses, ties: 0, 0, 30
synonyms, bm25, permutation p: 1.000
synonyms, bm25, established at 0.01: no
synonyms, bm25, t-test p: 1.000
synonyms, bm25, t-test agrees: yes
synonyms, bm25, 99% interval: 0.000 to 0.000
synonyms, bm25, interval agrees: yes

The list changes 14 train queries. Mean nDCG@10 goes from 0.695 to 0.722, and judged@10, the share of the top 10 with a judgment, from 0.930 to 0.944. The numbers of changed queries that score better and worse are 6, 5, and one query carries a share of 0.473 of the net gain: midcentury tv unit, which gains +0.632 with stand mid century added. The list was written from these queries, so even these numbers flatter it. The largest loss, -0.176, shows drift: wire basket with dividers gets hamper, and 4 products whose names contain hamper enter its top 10. 4 of them are unjudged, so the metric counts them as irrelevant, which for hampers is a fair guess.

On the test queries, the list changes 1 query, indoor multi tiered plant stand, by adding unit. Judged@10 stays at 0.927, and the difference isn't established (planned comparisons). On the queries it was written from, the list helped about as many queries as it hurt, and queries it wasn't written for rarely contain its words.

What to do: keep a synonym list for vocabulary gaps you have seen, write each entry from a failing query, and read the products each entry adds before you keep it. Never report a list's gain on the queries you wrote it from, and keep an entry only if a test on held-out queries shows a gain. Test any expansion of semantic retrieval first: the LLM's rewrites, which add synonyms and also broaden and drop words, hurt it. When the gaps are too many to list, try stemming or learned sparse retrieval instead.

Train a query classifier on product text 

Instead of changing a query's words, a system can predict what kind of product the query asks for. Query classification assigns a query to a class in the store's taxonomy, such as Range Hoods. In product search, "mapping a query to a pre-defined product taxonomy is an important step in e-commerce query understanding pipeline" (Skinner and Kallumadi, 2019): a predicted class can steer retrieval toward the right part of the catalog. Training a classifier usually takes labelled queries. Skinner and Kallumadi report that "an existing product-taxonomy mapping can improve query classification, and reduce the need for labeled data": they pretrain on product titles, whose classes are known, then fine-tune on fewer labelled queries.

The store goes further and uses no labelled queries, because it has none. WANDS gives each query a query_class, "category to which the query falls under" (WANDS), but chapter 2 shows that it is a prediction, and some values are wrong. And the book's sample of WANDS holds at most one query per class (examples/data/wands/README.md), so no class has two queries to learn from. So train on the products, which carry product_class, and apply the classifier to the queries.

A naive Bayes classifier can learn classes from product text in a few lines. In words: a class is likely for a query when the class is common among products, and when the query's words are common in that class's product text. It treats words as independent, which is the "naive" part. Each count gets 1 added, so that a word never seen in a class doesn't rule the class out:

predicted class = argmax over classes c of  log P(c) + sum over the query's words w of  log P(w | c)
P(w | c)        = (count of w in class c's text + 1) / (number of words in class c's text + vocabulary size)

In the store, build it by hand from word counts of the product text, check it against scikit-learn's MultinomialNB at 1.9.1, then compare its predictions with query_class and with the classes of each query's exact products:

1import numpy as np23from lib.queries import class_rows, train_classifier4from lib.store import load_labels, load_products, load_queries, product_text56products, queries, labels = load_products(), load_queries(), load_labels()7rows = class_rows(products)                     # one row per (product, class) for products with a class8vectorizer, model = train_classifier(products)  # the same, by scikit-learn's MultinomialNB9words = vectorizer.transform(product_text(rows))  # word counts: one row per (product, class), one column per word10classes = np.array(sorted(rows["label"].unique()))11print("product classes:", len(classes))12print("products without a class:", (products["product_class"] == "").sum())13print("products with several classes:", products["product_class"].str.contains("|", regex=False).sum())14missing = queries["query_class"].dropna()[lambda names: ~names.isin(classes)]15print("query classes that no product has:", len(missing))16print("query classes that no product has, names:", "; ".join(missing))1718# Naive Bayes: log P(class) + sum over the query's words of log P(word | class), each word count smoothed by 119word_counts = np.vstack([np.asarray(words[(rows["label"] == name).to_numpy()].sum(axis=0)) for name in classes])20log_prior = np.log(rows["label"].value_counts()[classes].to_numpy() / len(rows))21log_likelihood = np.log((word_counts + 1) / (word_counts.sum(axis=1, keepdims=True) + word_counts.shape[1]))22query_words = vectorizer.transform(queries["query"])23predicted = classes[np.asarray(query_words @ log_likelihood.T + log_prior).argmax(axis=1)]24print("same predictions as MultinomialNB:", (predicted == model.predict(query_words)).sum(), "of", len(queries))2526queries = queries.assign(predicted=predicted)27print("predicted class equals query_class:", (queries["predicted"] == queries["query_class"]).sum())28exact = labels[labels["grade"] == 2].join(products["product_class"].str.split("|"), on="product_id")29for column, name in [("predicted", "predicted class"), ("query_class", "query_class")]:30    share = [exact.loc[exact["query_id"] == query_id, "product_class"].apply(lambda names: label in names).mean()31             for query_id, label in queries[column].items()]32    print(f"queries where most exact products have the {name}:", (np.array(share) > 0.5).sum())33for query_id in [133, 255, 410]:34    query, split, label = queries.loc[query_id, ["query", "split", "predicted"]]35    print(f"{query_id} ({split}): {query} -> {label}"36          f" (query_class: {queries.at[query_id, 'query_class']})")37print("predicted class for 7qt slow cooker:", queries.at[62, "predicted"])       # its query_class has no products

product classes: 410
products without a class: 800
products with several classes: 620
query classes that no product has: 1
query classes that no product has, names: Crock Pots & Slow Cookers
same predictions as MultinomialNB: 80 of 80
predicted class equals query_class: 60
queries where most exact products have the predicted class: 63
queries where most exact products have the query_class: 63
133 (test): outdoor light fixtures -> Light Bulbs (query_class: Outdoor Wall Lights)
255 (test): desk and chair set -> Desks (query_class: Office Chairs)
410 (train): pool floats -> Deck Boxes (query_class: Lawn Games)
predicted class for 7qt slow cooker: Slow Cookers

The products use 410 classes. 800 products have none and are left out, and 620 have several, so each of those is one training row per class. The hand-built classifier makes the same prediction as MultinomialNB for 80 of 80 queries. Its prediction equals query_class for 60, but query_class is a guess too. Against the judgments, the two guesses tie: for 63 queries most exact products carry the predicted class, and for 63 they carry query_class. Three of its predictions show how it reads a query by its words alone: outdoor light fixtures becomes Light Bulbs, desk and chair set becomes Desks, and pool floats becomes Deck Boxes.

The sample's design limits what this check can show. Each class is tested by at most one query, so no per-class accuracy can be measured, and 80 queries cover only part of the 410 classes, so the counts say nothing about the rest. A query classifier can't be trained on these queries at all, and two classifiers can only be compared on whether each gets these few queries right. And 1 query class, Crock Pots & Slow Cookers, names no product class at all, so no classifier trained on products can predict it.

What to do: when you have no labelled queries, train a query classifier on your products' classes and check it against the classes of judged exact products, not against another classifier's guesses. Once your logs give you labelled queries, train on those. Treat the predicted class as a hint for ranking, never as a label.

Boost the predicted class instead of filtering 

In the store, the predicted class comes from the naive Bayes classifier trained on product text. A predicted class can act on retrieval in two ways. A filter keeps only products in the class. A boost adds a fixed amount to their scores, so that they move up without anything else being removed. A filter is only as good as the classifier: every relevant product outside the predicted class disappears.

The store's train judgments show how much a filter would cost: for the train queries, 501 of 1937 exact products lie outside their query's predicted class. The example adds a boost to the similarities of semantic retrieval, the store's best ranker and so its ranker to beat, and tries five boost weights and a filter on the train queries. It takes the best boost to the test queries for the second of the chapter's five planned comparisons, which counts as established if p is below 0.01:

1from lib.metrics import evaluate2from lib.queries import THRESHOLD, class_boost_run, in_class, report_planned, train_classifier3from lib.semantic import SemanticSearch4from lib.store import load_products, load_queries, qrels56products, queries = load_products(), load_queries()7vectorizer, model = train_classifier(products)               # trained on product text, never on queries8predicted = dict(zip(queries.index, model.predict(vectorizer.transform(queries["query"]))))9semantic = SemanticSearch()10train, train_judgments = load_queries("train"), qrels("train")1112exact, outside = 0, 013for query_id, judged in train_judgments.items():14    member = in_class(products, predicted[query_id])15    exact += sum(grade == 2 for grade in judged.values())16    outside += sum(grade == 2 and not member[product_id] for product_id, grade in judged.items())17print(f"exact train products outside the predicted class: {outside} of {exact}")1819base = evaluate(semantic.run(train), train_judgments)20print(f"train ndcg@10 semantic: {base['ndcg@10'].mean():.3f}")21print(f"train judged@10 semantic: {base['judged@10'].mean():.3f}")22train_scores = {}23for weight in [None, 0.01, 0.02, 0.05, 0.1, 0.2]:                          # None: keep only the predicted class24    train_scores[weight] = evaluate(class_boost_run(semantic, train.index, predicted, products, weight), train_judgments)25    name = "filter" if weight is None else f"boost {weight}"26    print(f"train ndcg@10 {name}: {train_scores[weight]['ndcg@10'].mean():.3f}")27boosts = [weight for weight in train_scores if weight is not None]28best = max(boosts, key=lambda weight: train_scores[weight]["ndcg@10"].mean())29for name, weight in [("filter", None), ("best boost", best)]:30    print(f"train judged@10 {name}: {train_scores[weight]['judged@10'].mean():.3f}")31print("best boost on train:", best)3233print(f"\nthe planned test comparison, threshold {THRESHOLD:.2f}:")34test, test_judgments = load_queries("test"), qrels("test")35boosted = evaluate(class_boost_run(semantic, test.index, predicted, products, best), test_judgments)36plain = evaluate(semantic.run(test), test_judgments)37print(f"test judged@10 semantic: {plain['judged@10'].mean():.3f}")38print(f"test judged@10 boosted: {boosted['judged@10'].mean():.3f}")39report_planned("class boost, semantic", boosted["ndcg@10"], plain["ndcg@10"])

exact train products outside the predicted class: 501 of 1937
train ndcg@10 semantic: 0.772
train judged@10 semantic: 0.964
train ndcg@10 filter: 0.675
train ndcg@10 boost 0.01: 0.773
train ndcg@10 boost 0.02: 0.774
train ndcg@10 boost 0.05: 0.768
train ndcg@10 boost 0.1: 0.742
train ndcg@10 boost 0.2: 0.692
train judged@10 filter: 0.866
train judged@10 best boost: 0.966
best boost on train: 0.02

the planned test comparison, threshold 0.01:
test judged@10 semantic: 0.990
test judged@10 boosted: 0.993
class boost, semantic, mean difference: 0.005
class boost, semantic, wins, losses, ties: 7, 4, 19
class boost, semantic, permutation p: 0.174
class boost, semantic, established at 0.01: no
class boost, semantic, t-test p: 0.164
class boost, semantic, t-test agrees: yes
class boost, semantic, 99% interval: -0.002 to 0.017
class boost, semantic, interval agrees: yes

On the train queries, the filter takes nDCG@10 from 0.772 to 0.675, and judged@10 from 0.964 to 0.866, so its top 10s also fill with products nobody judged. The boosts score from 0.773 at 0.01 and 0.774 at 0.02 down to 0.692 at 0.2, and the output lists the rest. The best boost on train is 0.02, with judged@10 of 0.966.

On the test queries, judged@10 is 0.990 without the boost and 0.993 with it, and the boost's gain isn't established (planned comparisons).

What to do: don't filter on a predicted class. Boost it with a weight tuned on train queries, and keep the boost only if its planned test comparison is established. The store's isn't, so its semantic retrieval stays unboosted. We recommend a class filter only when the shopper chose the class, for example by clicking a category.

Rewrite queries with an LLM 

Query rewriting replaces the query with a text written to retrieve better. Where the corrector, the unit rules and the synonym list each fix one kind of gap, an LLM can fix spelling, expand abbreviations and add synonyms in one step, with no list to maintain. In rewrite-retrieve-read, the rewritten query goes to the retriever in place of the original (Ma et al., 2023). Studies report gains. With query2doc, which appends an LLM-written passage, a pseudo-document, to the query, BM25 improved "by 3% to 15% on ad-hoc IR datasets", with text-davinci-003 writing the passages for MS MARCO and TREC DL queries (Wang et al., 2023). Other studies report failures. A comparison of 11 expansion methods, 12 datasets and 24 retrievers found that "expansion improves scores for weaker models, but generally harms stronger models" (Weller et al., 2024). Another study found that LLM-based expansion "can significantly degrade the retrieval effectiveness when knowledge in the LLM is insufficient or query ambiguity is high" (Abe et al., 2025).

The rewrites that the examples read were recorded with the store's LLM, the open model named in the chapter's opening, run locally on each query alone. The prompt's instruction is: "Rewrite this search query from an online furniture and home-goods store so that a search engine finds what the shopper wants. Fix spelling, expand abbreviations and add common synonyms for the product type, but keep every requirement the shopper stated and add none of your own. Return a single line." The rest of the prompt gives the query and asks for JSON, and the recording's manifest holds it in full. The recording script embedded each rewrite as a query, with the same embedding model as the products. BM25 searches the rewrite's text, and semantic retrieval its embedding. Look at the train queries first:

1from lib.lexical import LexicalSearch2from lib.metrics import evaluate3from lib.recorded import llm_rewrites, manifest4from lib.semantic import SemanticSearch5from lib.store import load_products, load_queries, qrels67recording = manifest("llm-rewrites")8print("model:", recording["model"])9print("embedding model:", recording["embedding_model"])10print("rewrite prompt, first line:", recording["prompts"]["rewrite"].split("\n")[0])11table, rewrite_vectors, _ = llm_rewrites()12rewrites = table.set_index("query_id")["rewrite"]13row = {query_id: position for position, query_id in enumerate(table["query_id"])}14for query_id in [20, 205, 313]:15    print(f"rewrite of {load_queries().at[query_id, 'query']}: {rewrites[query_id]}")1617queries, judgments = load_queries("train"), qrels("train")18products = load_products()19exact = [product_id for product_id, grade in judgments[313].items() if grade == 2]20print("exact products for malachi sled:", "; ".join(products.loc[exact, "name"]))21bm25, semantic = LexicalSearch(products), SemanticSearch()22runs = {"bm25, typed": bm25.run(queries),23        "bm25, rewrite": bm25.run(queries.assign(query=rewrites[queries.index].to_numpy())),24        "semantic, typed": semantic.run(queries),25        "semantic, rewrite": {query_id: list(semantic.search_vector(rewrite_vectors[row[query_id]]).index)26                              for query_id in queries.index}}27scores = {name: evaluate(run, judgments) for name, run in runs.items()}28for name, frame in scores.items():29    print(f"train ndcg@10 {name}: {frame['ndcg@10'].mean():.3f}")30    print(f"train judged@10 {name}: {frame['judged@10'].mean():.3f}")31for retriever in ["bm25", "semantic"]:                       # for choosing, not evidence32    change = (scores[f"{retriever}, rewrite"]["ndcg@10"] - scores[f"{retriever}, typed"]["ndcg@10"]).round(3)33    print(f"train queries better, worse, {retriever}: {(change > 0).sum()}, {(change < 0).sum()}")34    print(f"largest train loss, {retriever}: {queries.at[change.idxmin(), 'query']}")35    print(f"largest train loss, {retriever}, change: {change.min():+.3f}")

model: gemma4:12b-it-q4_K_M
embedding model: Snowflake/snowflake-arctic-embed-m-v1.5
rewrite prompt, first line: Rewrite this search query from an online furniture and home-goods store so that a search engine finds what the shopper wants. Fix spelling, expand abbreviations and add common synonyms for the product type, but keep every requirement the shopper stated and add none of your own. Return a single line.
rewrite of foutains with brick look: fountains with brick look, brick style water features, masonry fountains, faux brick fountains
rewrite of ines shag rug w003096809: Ines shag rug w003096809 area rug carpet
rewrite of malachi sled: Malachi sled sofa couch settee loveseat
exact products for malachi sled: malachi sled end table; malachi 44 '' console table; malachi sled coffee table
train ndcg@10 bm25, typed: 0.695
train judged@10 bm25, typed: 0.930
train ndcg@10 bm25, rewrite: 0.646
train judged@10 bm25, rewrite: 0.868
train ndcg@10 semantic, typed: 0.772
train judged@10 semantic, typed: 0.964
train ndcg@10 semantic, rewrite: 0.759
train judged@10 semantic, rewrite: 0.924
train queries better, worse, bm25: 23, 24
largest train loss, bm25: malachi sled
largest train loss, bm25, change: -0.644
train queries better, worse, semantic: 18, 25
largest train loss, semantic: wall design shelf
largest train loss, semantic, change: -0.653

The rewrites fix spelling, fountains with brick look, brick style water features, masonry fountains, faux brick fountains, and keep codes, Ines shag rug w003096809 area rug carpet. For malachi sled, a product line, the LLM guessed a sofa: it wrote Malachi sled sofa couch settee loveseat, while the exact products are malachi sled end table; malachi 44 '' console table; malachi sled coffee table. On the train queries, BM25's nDCG@10 goes from 0.695 to 0.646, and semantic retrieval's from 0.772 to 0.759. Judged@10 falls for both, from 0.930 to 0.868 and from 0.964 to 0.924, so the rewrites surface products nobody judged, and part of each fall in nDCG@10 may come from them. The numbers of queries that score better and worse are 23, 24 for BM25 and 18, 25 for semantic retrieval. BM25's largest loss is malachi sled, at -0.644, and semantic retrieval's is wall design shelf, at -0.653.

These train numbers shape expectations and prove nothing. The third and fourth of the chapter's five planned comparisons were fixed in advance, so they go to the test queries whatever the train queries show, and each counts as established if p is below 0.01. Before you run them, predict: on the test queries, will the rewrite help BM25, the weaker retriever, hurt semantic retrieval, the stronger one, or neither?

1from lib.lexical import LexicalSearch2from lib.metrics import evaluate3from lib.queries import THRESHOLD, condensed, report_planned, rewrite_runs4from lib.semantic import SemanticSearch5from lib.significance import compare6from lib.store import load_products, load_queries, qrels78queries, judgments = load_queries("test"), qrels("test")             # scored on the human judgments9runs = rewrite_runs(queries, LexicalSearch(load_products()), SemanticSearch())1011print(f"the planned test comparisons, threshold {THRESHOLD:.2f}:")12for retriever, (rewritten, typed) in runs.items():13    name = f"llm rewrite, {retriever}"14    after, before = evaluate(rewritten, judgments), evaluate(typed, judgments)15    for form, frame in [("typed", before), ("rewrite", after)]:16        print(f"{name}, test ndcg@10 {form}: {frame['ndcg@10'].mean():.3f}")17        print(f"{name}, test judged@10 {form}: {frame['judged@10'].mean():.3f}")18    report_planned(name, after["ndcg@10"], before["ndcg@10"])1920    # Exploratory, never established: the same comparison on condensed runs, which drop unjudged products21    after, before = (evaluate(condensed(run, judgments), judgments)["ndcg@10"] for run in (rewritten, typed))22    result = compare(after, before, confidence=1 - THRESHOLD)23    print(f"{name}, condensed ndcg@10 typed: {before.mean():.3f}")24    print(f"{name}, condensed ndcg@10 rewrite: {after.mean():.3f}")25    print(f"{name}, condensed, exploratory mean difference: {result['mean difference']:.3f}")26    print(f"{name}, condensed, exploratory permutation p: {result['permutation p']:.3f}")27    low, high = result["ci low"], result["ci high"]28    print(f"{name}, condensed, exploratory {1 - THRESHOLD:.0%} interval: {low:.3f} to {high:.3f}")

the planned test comparisons, threshold 0.01:
llm rewrite, bm25, test ndcg@10 typed: 0.725
llm rewrite, bm25, test judged@10 typed: 0.927
llm rewrite, bm25, test ndcg@10 rewrite: 0.669
llm rewrite, bm25, test judged@10 rewrite: 0.903
llm rewrite, bm25, mean difference: -0.056
llm rewrite, bm25, wins, losses, ties: 13, 15, 2
llm rewrite, bm25, permutation p: 0.060
llm rewrite, bm25, established at 0.01: no
llm rewrite, bm25, t-test p: 0.060
llm rewrite, bm25, t-test agrees: yes
llm rewrite, bm25, 99% interval: -0.143 to 0.005
llm rewrite, bm25, interval agrees: yes
llm rewrite, bm25, condensed ndcg@10 typed: 0.778
llm rewrite, bm25, condensed ndcg@10 rewrite: 0.725
llm rewrite, bm25, condensed, exploratory mean difference: -0.053
llm rewrite, bm25, condensed, exploratory permutation p: 0.044
llm rewrite, bm25, condensed, exploratory 99% interval: -0.150 to -0.002
llm rewrite, semantic, test ndcg@10 typed: 0.830
llm rewrite, semantic, test judged@10 typed: 0.990
llm rewrite, semantic, test ndcg@10 rewrite: 0.749
llm rewrite, semantic, test judged@10 rewrite: 0.947
llm rewrite, semantic, mean difference: -0.081
llm rewrite, semantic, wins, losses, ties: 7, 19, 4
llm rewrite, semantic, permutation p: 0.000
llm rewrite, semantic, established at 0.01: yes
llm rewrite, semantic, t-test p: 0.000
llm rewrite, semantic, t-test agrees: yes
llm rewrite, semantic, 99% interval: -0.132 to -0.035
llm rewrite, semantic, interval agrees: yes
llm rewrite, semantic, condensed ndcg@10 typed: 0.834
llm rewrite, semantic, condensed ndcg@10 rewrite: 0.791
llm rewrite, semantic, condensed, exploratory mean difference: -0.043
llm rewrite, semantic, condensed, exploratory permutation p: 0.002
llm rewrite, semantic, condensed, exploratory 99% interval: -0.079 to -0.014

For BM25, judged@10 falls from 0.927 to 0.903, and nDCG@10 from 0.725 to 0.669, a difference that isn't established (planned comparisons).

For semantic retrieval, judged@10 falls from 0.990 to 0.947, and nDCG@10 from 0.830 to 0.749, an established loss, with a permutation p-value that rounds to 0.000.

Because judged@10 fell, the example repeats both comparisons on condensed lists, which drop unjudged products before scoring. That check was chosen after seeing the drop, so it is exploratory and can't establish anything. For semantic retrieval, condensed nDCG@10 goes from 0.834 to 0.791, a difference of -0.043 with a permutation p of 0.002 and a 99% interval from -0.079 to -0.014. So the loss doesn't rest on unjudged products alone. For BM25, condensed nDCG@10 goes from 0.778 to 0.725, with a permutation p of 0.044, a question for new queries rather than a finding.

If you predicted that the rewrite would help BM25, the test doesn't show it, on these queries with this LLM and prompt: BM25 moved the other way from what Weller et al. and query2doc report, though not by an established amount. If you predicted harm to semantic retrieval, you predicted what Weller et al. found for strong retrievers, here with this model, this prompt and these queries.

The queries that move most show what a rewrite does. Read them with the number of unjudged products in each rewritten top 10, which the metric counts as irrelevant:

1from bm25s.stopwords import STOPWORDS_EN23from lib.lexical import LexicalSearch4from lib.metrics import evaluate, ndcg_at_k5from lib.queries import rewrite_runs6from lib.recorded import llm_rewrites, probes7from lib.semantic import SemanticSearch8from lib.store import load_products, load_queries, qrels910queries, judgments = load_queries("test"), qrels("test")11bm25 = LexicalSearch(load_products())12rewrites = llm_rewrites()[0].set_index("query_id")["rewrite"]13largest = {}14for retriever, (rewritten, typed) in rewrite_runs(queries, bm25, SemanticSearch()).items():15    change = evaluate(rewritten, judgments)["ndcg@10"] - evaluate(typed, judgments)["ndcg@10"]16    for kind, query_id in [("loss", change.idxmin()), ("gain", change.idxmax())]:17        label, largest[(retriever, kind)] = f"{retriever}, largest {kind}", query_id18        print(f"{label}: {queries.at[query_id, 'query']}")19        print(f"{label}, change: {change[query_id]:+.3f}")20        print(f"{label}, rewrite: {rewrites[query_id]}")21        unjudged = sum(product_id not in judgments[query_id] for product_id in rewritten[query_id][:10])22        print(f"{label}, unjudged products in the rewrite's top 10: {unjudged}")2324# BM25's largest gain: score the query with only its spelling fixed (chapter 6's probe), apart from the added words25query_id = largest[("bm25", "gain")]26spelled = probes()[0].dropna(subset=["of_query_id"]).set_index("of_query_id")["query"]27if query_id in spelled.index:28    for name, text in [("typed", queries.at[query_id, "query"]), ("spelling fixed only", spelled[query_id])]:29        ranking = list(bm25.search(text, k=10).index)30        print(f"bm25, largest gain, ndcg@10 {name}: {ndcg_at_k(ranking, judgments[query_id], 10):.3f}")31    print("bm25 stop words among an and and:", ", ".join(word for word in ["an", "and"] if word in STOPWORDS_EN))

bm25, largest loss: fold in half table
bm25, largest loss, change: -0.471
bm25, largest loss, rewrite: folding table, collapsible table, drop leaf table, gateleg table
bm25, largest loss, unjudged products in the rewrite's top 10: 3
bm25, largest gain: refrigerator with ice an water in door
bm25, largest gain, change: +0.159
bm25, largest gain, rewrite: refrigerator with ice and water in door, fridge with door dispenser, refrigerator with external water and ice dispenser
bm25, largest gain, unjudged products in the rewrite's top 10: 1
semantic, largest loss: pull out sleeper loveseat
semantic, largest loss, change: -0.304
semantic, largest loss, rewrite: pull out sleeper loveseat sofa couch convertible sofa sofa bed
semantic, largest loss, unjudged products in the rewrite's top 10: 4
semantic, largest gain: nautical platters
semantic, largest gain, change: +0.055
semantic, largest gain, rewrite: nautical platters serving trays seafood platters maritime serving boards
semantic, largest gain, unjudged products in the rewrite's top 10: 0
bm25, largest gain, ndcg@10 typed: 0.501
bm25, largest gain, ndcg@10 spelling fixed only: 0.501
bm25 stop words among an and and: an, and

BM25's largest loss, -0.471 for fold in half table, comes from folding table, collapsible table, drop leaf table, gateleg table, which drops in half and adds other kinds of table; 3 of its top 10 are unjudged. Semantic retrieval's largest loss, -0.304 for pull out sleeper loveseat, comes from pull out sleeper loveseat sofa couch convertible sofa sofa bed, which widens a loveseat to any sofa; 4 of its top 10 are unjudged. BM25's largest gain, +0.159 for refrigerator with ice an water in door, comes from refrigerator with ice and water in door, fridge with door dispenser, refrigerator with external water and ice dispenser. The rewrite fixes an, but with only that fix BM25 scores 0.501, the same as the query as typed, 0.501, because both spellings are stop words. The gain comes from the words the rewrite adds. Semantic retrieval's largest gain is +0.055, for nautical platters, rewritten as nautical platters serving trays seafood platters maritime serving boards.

What to do: don't put an LLM rewrite in front of a strong retriever without a planned test on held-out queries. Read the per-query losses for dropped requirements and broadened products, which the prompt's "keep every requirement" doesn't prevent. For the store, semantic retrieval, the ranker to beat, keeps searching the query as typed.

Search with a hypothetical product (HyDE) 

Where a rewrite asks the LLM for a better query, HyDE, short for hypothetical document embeddings, asks it for the document it imagines would answer the query. An encoder embeds that hypothetical document, and "This vector identifies a neighborhood in the corpus embedding space" where real documents lie (Gao et al., 2023). The made-up details needn't be true: the authors argue that the encoder's "dense bottleneck" filters out "the hallucinations", which is their argument, not a measurement. Their main results use InstructGPT (text-davinci-003), sampling at temperature 0.7. On TREC DL19, a passage-ranking benchmark, HyDE raised the nDCG@10 of Contriever, an embedding model trained without relevance labels, from 44.5 to 61.3 on a 0 to 100 scale (Gao et al., Table 1). With a Contriever fine-tuned on labels, the result depended on the LLM. The smaller ones lowered nDCG@10 slightly, since "less powerful instruction LMs can negatively impact the overall performance of the fine-tuned retriever", while InstructGPT raised it, most on DL19, from 62.1 to 67.4 (Table 4).

Several expansion methods keep the original query in the mix. HyDE averages the embeddings of the hypothetical documents together with the query's own (Gao et al., section 3.2). Relevance feedback improves a query from judgments of its first results, and Rocchio's version moves the query vector toward the results judged relevant while keeping the original vector in its sum (Manning et al., 2008). For BM25, query2doc repeats the query five times before the generated passage, so that the original terms keep their weight (Wang et al., 2023).

For the store, the HyDE recording holds one hypothetical product per query, which the store's LLM wrote from the query alone: a name, a class and a description, embedded as a product. The example's hyde function averages it with the query's embedding and scales the result back to length 1, as chapter 6 advises after any averaging. The two embeddings differ in one way: the query's carries the embedding model's query prompt, and the product's has none, as for every product.

hyde(query) = (embedding(hypothetical product) + embedding(query)) / 2,  then scaled to length 1

On the train queries, the example also tries the hypothetical product's embedding alone, and, for BM25, the hypothetical product's text alone and with the query repeated five times before it, as query2doc weights it. Then it makes the last of the chapter's five planned comparisons, which counts as established if p is below 0.01:

1import numpy as np23from lib.lexical import LexicalSearch4from lib.metrics import evaluate5from lib.queries import THRESHOLD, report_planned6from lib.recorded import llm_rewrites7from lib.semantic import SemanticSearch8from lib.store import load_products, load_queries, qrels910table, _, hypothetical_vectors = llm_rewrites()11hypothetical, vectors = table.set_index("query_id")["hypothetical"], dict(zip(table["query_id"], hypothetical_vectors))12bm25, semantic = LexicalSearch(load_products()), SemanticSearch()131415def hyde(query_id):16    """Gao et al.'s average of the hypothetical product's embedding and the query's, scaled back to length 1."""17    average = (vectors[query_id] + semantic.query_vectors[query_id]) / 218    return average / np.linalg.norm(average)192021def runs(queries):22    ids, typed = queries.index, queries["query"]23    return {"semantic": semantic.run(queries),24            "hypothetical alone": {query_id: list(semantic.search_vector(vectors[query_id]).index) for query_id in ids},25            "hyde": {query_id: list(semantic.search_vector(hyde(query_id)).index) for query_id in ids},26            "bm25": bm25.run(queries),27            "bm25, hypothetical alone": bm25.run(queries.assign(query=ids.map(hypothetical))),28            "bm25, query five times plus hypothetical": bm25.run(queries.assign(           # query2doc's weighting29                query=[" ".join([typed[query_id]] * 5 + [hypothetical[query_id]]) for query_id in ids]))}303132train = {name: evaluate(run, qrels("train")) for name, run in runs(load_queries("train")).items()}33for name, frame in train.items():34    print(f"train ndcg@10 {name}: {frame['ndcg@10'].mean():.3f}")35    print(f"train judged@10 {name}: {frame['judged@10'].mean():.3f}")36change = (train["hypothetical alone"]["ndcg@10"] - train["semantic"]["ndcg@10"]).round(3)    # for choosing, not evidence37print(f"train queries better, worse, hypothetical alone: {(change > 0).sum()}, {(change < 0).sum()}")3839print(f"\nthe planned test comparison, threshold {THRESHOLD:.2f}:")40test = {name: evaluate(run, qrels("test")) for name, run in runs(load_queries("test")).items()41        if name in ["semantic", "hyde"]}42for name, frame in test.items():43    print(f"test judged@10 {name}: {frame['judged@10'].mean():.3f}")44report_planned("hyde, semantic", test["hyde"]["ndcg@10"], test["semantic"]["ndcg@10"])

train ndcg@10 semantic: 0.772
train judged@10 semantic: 0.964
train ndcg@10 hypothetical alone: 0.704
train judged@10 hypothetical alone: 0.890
train ndcg@10 hyde: 0.769
train judged@10 hyde: 0.946
train ndcg@10 bm25: 0.695
train judged@10 bm25: 0.930
train ndcg@10 bm25, hypothetical alone: 0.644
train judged@10 bm25, hypothetical alone: 0.854
train ndcg@10 bm25, query five times plus hypothetical: 0.694
train judged@10 bm25, query five times plus hypothetical: 0.912
train queries better, worse, hypothetical alone: 13, 29

the planned test comparison, threshold 0.01:
test judged@10 semantic: 0.990
test judged@10 hyde: 0.970
hyde, semantic, mean difference: -0.031
hyde, semantic, wins, losses, ties: 8, 14, 8
hyde, semantic, permutation p: 0.108
hyde, semantic, established at 0.01: no
hyde, semantic, t-test p: 0.121
hyde, semantic, t-test agrees: yes
hyde, semantic, 99% interval: -0.115 to 0.001
hyde, semantic, interval agrees: yes

On the train queries, the hypothetical product alone takes semantic retrieval's nDCG@10 from 0.772 to 0.704, with judged@10 from 0.964 to 0.890, and the numbers of queries that score better and worse are 13, 29. Averaged with the query, it scores 0.769, with judged@10 of 0.946. BM25 behaves the same way: 0.695 on the query, 0.644 on the hypothetical text alone, and 0.694 with the query repeated five times before it, with judged@10 of 0.930, 0.854 and 0.912. Keeping the query recovers what the hypothetical product loses, and on the train queries, for either retriever, the combination adds nothing over the query alone. Train numbers guide the design; only the planned test comparison is evidence.

On the test queries, judged@10 goes from 0.990 to 0.970, and HyDE shows no established difference from semantic retrieval (planned comparisons). This result belongs to one hypothetical product per query, from a 12-billion-parameter model run at temperature 0. Gao et al. sampled from a far larger LLM, and only that one raised their fine-tuned retriever.

What to do: when you try HyDE, always average the query's own embedding in. Expect the largest gains with an encoder trained without labels; with a fine-tuned encoder, Gao et al. saw gains only from their largest LLM, and the store, with its model, prompt and encoder, saw none. On an encoder like the store's, which already beats BM25 on these queries, test HyDE before you pay for an LLM call per query. The store doesn't use it.

Turn a follow-up into a standalone query 

The last gap from the chapter's opening comes from conversation: a shopper's next message often depends on the last one, as in white refines a search for a desk. Retrieval sees one text, so the follow-up has to become a standalone query, one that keeps what the shopper still wants. Conversational query rewriting is the task of turning "a context-dependent into a self-contained question with the same answer", and the CANARD dataset holds 40,527 such rewrites (Elgohary et al., 2019). In TREC CAsT 2019, a conversational search evaluation, people's rewrites beat the best automatic system, "a 35% relative improvement of manual rewrites over the best automatic system" (Dalton et al., 2020). Later work with prompted LLMs found that "informative query rewrites can yield substantially improved retrieval performance compared to human rewrites, especially with sparse retrievers" (Ye et al., 2023), and in TREC iKAT 2025, CAsT's successor, "Most of the runs leveraged LLMs in their pipelines, for single or multiple query rewriting" (TREC iKAT 2025 overview).

WANDS has no conversations, so we wrote the store's conversations for the book: 12 follow-ups, each refining a test query by one attribute, a color, shape, size or finish. The store's LLM, which saw only the two turns, rewrote each to stand alone. For example, after coat rack for bedroom comes something in black, rewritten as black coat rack for bedroom. The judgments grade products for the first query, not for the refined one, so nDCG can't score the follow-up. Instead, count three kinds of product in each top 10: those that keep the refinement, because their attributes or text contain the value, such as black (the example also prints the count from attributes alone); those of the right kind, which the judges graded exact or partial for the first query; and those that are both. The example compares the follow-up alone, the two turns joined and the standalone query, each embedded as a query, with the first query as a baseline. It adds a filter on the standalone query's top 100 that keeps products stating the value, taking the value from the recorded conversation; a live system would have to extract it from the follow-up.

1import numpy as np23from lib.queries import mentions4from lib.recorded import llm_conversations5from lib.semantic import SemanticSearch6from lib.store import load_products, load_queries, product_text, qrels78products, judgments, semantic = load_products(), qrels("test"), SemanticSearch()9attributes, text = products["attributes"], product_text(products)        # the embeddings read only the text10conversations, vectors = llm_conversations()                 # follow-ups we wrote for the book, not shoppers'11print("conversations:", len(conversations))12example = conversations.iloc[4]13print(f"first query: {load_queries().at[example['query_id'], 'query']}")14print(f"follow-up: {example['follow_up']}")15print(f"standalone: {example['standalone']}")1617shares, filtered_lengths = {}, []                            # shares: (top 10, measure) -> one share per conversation18for row, (first, value) in enumerate(zip(conversations["query_id"], conversations["value"])):19    standalone = semantic.search_vector(vectors["standalone"][row], k=100).index20    stated = mentions(standalone, value, attributes) | mentions(standalone, value, text)21    tops = {"first query": semantic.search(first, k=10).index,22            "follow-up alone": semantic.search_vector(vectors["follow_up"][row], k=10).index,23            "turns joined": semantic.search_vector(vectors["joined"][row], k=10).index,24            "standalone": standalone[:10],25            "standalone, filtered": standalone[stated][:10]}         # given the value, which a live system extracts26    filtered_lengths.append(len(tops["standalone, filtered"]))27    for name, top in tops.items():28        in_attributes = mentions(top, value, attributes)29        in_either = in_attributes | mentions(top, value, text)30        right_kind = np.array([judgments[first].get(product_id, 0) >= 1 for product_id in top], dtype=bool)31        for measure, hits in [("refinement in attributes", in_attributes), ("refinement in text or attributes", in_either),32                              ("right kind", right_kind), ("right kind with the refinement", in_either & right_kind)]:33            shares.setdefault((name, measure), []).append(hits.sum() / 10)     # empty places count as misses34for (name, measure), values in shares.items():35    print(f"{name}, {measure}: {np.mean(values):.3f}")3637kept = np.subtract(shares["standalone", "refinement in text or attributes"], shares["turns joined", "refinement in text or attributes"])38print(f"standalone against turns joined, more, fewer, same: {(kept > 0).sum()}, {(kept < 0).sum()}, {(kept == 0).sum()}")39print("conversations the filter leaves with fewer than 10 products:", sum(length < 10 for length in filtered_lengths))

conversations: 12
first query: coat rack for bedroom
follow-up: something in black
standalone: black coat rack for bedroom
first query, refinement in attributes: 0.125
first query, refinement in text or attributes: 0.200
first query, right kind: 0.958
first query, right kind with the refinement: 0.175
follow-up alone, refinement in attributes: 0.433
follow-up alone, refinement in text or attributes: 0.950
follow-up alone, right kind: 0.058
follow-up alone, right kind with the refinement: 0.058
turns joined, refinement in attributes: 0.208
turns joined, refinement in text or attributes: 0.383
turns joined, right kind: 0.933
turns joined, right kind with the refinement: 0.325
standalone, refinement in attributes: 0.200
standalone, refinement in text or attributes: 0.392
standalone, right kind: 0.958
standalone, right kind with the refinement: 0.350
standalone, filtered, refinement in attributes: 0.583
standalone, filtered, refinement in text or attributes: 0.900
standalone, filtered, right kind: 0.675
standalone, filtered, right kind with the refinement: 0.675
standalone against turns joined, more, fewer, same: 5, 4, 3
conversations the filter leaves with fewer than 10 products: 3

The follow-up alone keeps the refinement in 0.950 of its top 10, but only 0.058 are the right kind of product. Any form that carries the first query finds the right kind of product: 0.933 for the joined turns and 0.958 for the standalone query. On these conversations, the joined turns and the standalone query both keep the refinement more often than the first query, which never asked for it: 0.383 and 0.392, against 0.200. Counted in attributes alone, the shares are lower, 0.208, 0.200 and 0.125, because many products state the value only in their text, which is all the embeddings read. The conclusions below rest on the text-or-attributes count, which can overcount, since a product that mentions round legs counts as round.

The LLM's rewrite does no better than joining the turns: against the joined turns, the standalone query keeps more, fewer or the same number of refined products in 5, 4, 3 conversations, and products of the right kind that also keep the refinement make up 0.350 of the standalone query's top 10s, against 0.325 for the joined turns. The filter raises that share to 0.675, but it leaves 3 conversations with fewer than 10 products. These 12 conversations are ours, not shoppers', so the counts are a diagnosis, not a test.

What to do: never search a follow-up alone. Join it to the earlier turns at least, or rewrite it to stand alone. When the refinement names a value the catalog states, extract it and boost or filter on it, and fall back to the unfiltered list when too few products remain. Measure on conversations from your own logs before you trust any of it.

What goes wrong in production 

Every LLM rewrite is a model call on the query's path. The examples read rewrites recorded once, offline, but a live search box calls the model on every query while the shopper waits. query2doc's authors note that it "requires running inference with LLMs which can be considerably slower" (Wang et al., 2023), and the longer query also slows the BM25 search. A rule of thumb, not a measurement: rewrite frequent queries offline and cache the results, and call the model online only for queries where a test shows a gain.

A rewrite belongs to one model and one prompt. Another model version or a reworded prompt can change any rewrite. The manifests of the store's recordings hold the model's digest and each prompt verbatim, and a change to either means measuring again.

The lexicon must follow the catalog. A corrector built from last month's catalog can treat this month's brand names as typos. Rebuild the lexicon with the index, and log every query the corrector changes next to the original, so that you can find over-corrections such as tye → the. In machine-translated queries, misspellings change which products a search returns (Hanneman et al., 2024), so expect an error the corrector misses to show up as unstable results for one query.

A classifier follows its taxonomy. When product classes are renamed, split or merged, retrain the classifier. A query class that names no product class, such as Crock Pots & Slow Cookers, can never be predicted; for 7qt slow cooker the classifier predicts Slow Cookers instead.

Rewritten queries surface unjudged products. Every LLM-based change in this chapter that was scored with nDCG@10 lowered judged@10. Before you adopt a rewrite, have the unjudged products in its top 10s judged, or check condensed scores as chapter 2 does.

What to do: log the query as typed and as rewritten, cache LLM rewrites, version the model and prompt with the outputs, and rebuild the lexicon and the classifier whenever the catalog changes.

Decisions this chapter made 

Every setting below was chosen on the train queries. The table holds the chapter's 5 planned comparisons on the test queries, one per column: a retriever with one change, against the same retriever on the query as typed. A difference is established when its permutation p-value is below 0.01.

• Spelling: correct only words the catalog doesn't use. The corrector fixes the store's train typos, but the typo queries are too few to test it, and it needs context before it handles tye dye or ice an water. The store doesn't run it yet.
• Units: one written form for products and queries, from the catalog's own spellings, with decimals kept whole. The size queries are too few to test it, so the baseline BM25 stays as it is.
• Synonyms: a list for gaps seen in train queries, for BM25 only. It changed 1 test query and showed no established difference, so the baseline doesn't expand.
• Query classes: naive Bayes trained on product text, because the sample has at most one query per class. The predicted class is a hint. A filter loses on train, and the best boost isn't established on test, so semantic retrieval stays unboosted.
• LLM rewrites: not used. The loss for semantic retrieval is established, and BM25's change isn't.
• HyDE: not used. Averaged with the query, it shows no established difference, and the hypothetical product alone loses on train.
• Follow-ups: never searched alone. Joining the turns does about as well as the LLM's standalone query. A filter on the refined value is the next step, and it needs an extracted value and a fallback.
• The ranker to beat stays semantic retrieval on the query as typed, with BM25, unstemmed, as the baseline.