Rolling Edition

A Primer on Search Relevance

Teaching kit

From Edition 2026.11.4, the book's current edition. It changes when the book does. Use and adapt it freely, with credit to the book.

On this page
  1. Syllabus
  2. Objectives and outlines
  3. Quiz bank

Syllabus

Reading times are for the chapter's text. Students also run its examples, in the Lab or on their own machine.

A chapter a week: 11 weeks

WeekChaptersReadingExamples
11. Search, end to end26 min11
22. What relevance means28 min10
33. Measuring search32 min12
44. Judgments from people and LLMs33 min11
55. Lexical retrieval30 min13
66. Semantic retrieval30 min16
77. Hybrid retrieval32 min12
88. Understanding queries32 min12
99. Ranking32 min14
1010. Retrieval for LLM applications32 min10
1111. Testing changes30 min11

In 5 weeks

WeekChaptersReadingExamples
11. Search, end to end
2. What relevance means
3. Measuring search		86 min33
24. Judgments from people and LLMs
5. Lexical retrieval		63 min24
36. Semantic retrieval
7. Hybrid retrieval		62 min28
48. Understanding queries
9. Ranking		64 min26
510. Retrieval for LLM applications
11. Testing changes		62 min21

Objectives and outlines

Each chapter's objectives are its first sentence. Each outline follows the chapter's sections, with the terms it defines, the predictions it asks for before an example runs, and the rule each section ends on.

1. Search, end to end

About 26 minutes to read, 11 examples.

By the end, students can build a rough but complete search pipeline for a product catalog, from a shopper's query to an ordered list of products, and measure with one number whether a change to that pipeline helps.

Lecture outline: 10 sections
  1. Load the store's data
    • Key terms judgment, grade, relevant
    • Run load-the-store.py
    • Working rule Split your queries into train and test before you change anything. Read test results to understand a ranker, but choose and tune every change on train queries, and report on test. Treat unjudged products as unknown. If you have no judgments yet, collect some before you change the ranking.
  2. Rank products with BM25
    • Run first-bm25-search.py
    • Working rule Start with BM25 at the library's defaults as your baseline. It needs no training data and no model, and when it goes wrong you can see which words caused it. Leave its parameters alone until you can measure a change on train queries. When shoppers and products use different words for the same thing, add semantic retrieval.
  3. Check the results against the judgments
    • Run judge-first-results.py
    • Working rule Before you compute any metric, read the top results of five or ten queries next to their judgments. Look for patterns: accessories, missed modifiers, wrong sizes. When several top results are unjudged, have them judged before you trust a score for that query.
  4. Score one ranking with nDCG@10
    • Run ndcg-for-one-query.py
    • Working rule We recommend nDCG@10 as your headline metric when your judgments are graded and the first page of results matters most, as it does in a store's search. When you have only yes-or-no labels, or when shoppers look at one result only, another metric fits better. Chapter 3 builds the alternatives and says when to choose each.
  5. Score every test query
    • Run bm25-test-scores.py
    • Working rule Report the mean nDCG@10 over the test queries with mean judged@10 beside it, and read the lowest-scoring queries every time you score a run. When one ranker's judged@10 is lower than another's, find out why before you compare their nDCG: the gap may come from missing judgments, not worse results.
  6. Search by meaning with embeddings
    • Key terms embedding
    • Run first-semantic-search.py
    • Run compare-lexical-semantic.py
    • Working rule Add semantic retrieval when shoppers describe what they want in their own words. Don't assume lexical retrieval wins on exact terms such as sizes, brands and model numbers: before you rely on it there, count which retriever wins the train queries that contain them. On this data, semantic retrieval alone scores highest so far, so it is the ranker to beat.
  7. Fuse two rankings with reciprocal rank fusion
    • Ask first Before you read the output, predict the hybrid's mean nDCG@10. BM25 scores 0.725 and semantic retrieval 0.830. Will their fusion score above semantic retrieval, between the two, or below BM25?
    • Run hybrid-rrf.py
    • Working rule Measure each retriever alone before you fuse them. Use RRF with k = 60 as an untuned default, and keep the fused ranking only if it beats the best single retriever on held-out queries. When one retriever is much weaker, try weighting the rankings or tuning k on train queries instead of accepting equal votes. Chapter 7 tries both on the store's data, and none of the tuned blends it reports on the test queries beats semantic retrieval (Report the tuned blends on test queries).
  8. Rerank the top 50 with a cross-encoder
    • Run cross-encoder-rerank.py
    • Run rerank-one-query.py
    • Working rule Rerank a shortlist, here 50 products, because the cost grows with every pair. Adopt a reranker only when it beats the best retrieval alone on held-out queries. This one didn't improve on the hybrid run or on semantic retrieval, so the store's best pipeline so far is semantic retrieval alone. Before you give up on reranking, try a different depth, or a ranker trained on your own judgments. Chapter 9 tries both on the store's data, and neither shows an improvement over semantic retrieval on the test queries (Set the rerank depth from a budget, Report LambdaMART on the test queries).
  9. What goes wrong in production
    • Run tokenizer-drops-terms.py
    • Working rule Check the tokens your queries become, track judged@10, and record the model version next to every vector you store.
  10. Decisions this chapter made

2. What relevance means

About 28 minutes to read, 10 examples.

By the end, students can read a set of relevance judgments critically:

  • say what each grade claims about a product
  • find where the judgments came from and where they have holes
  • decide how far to trust a score computed from them
Lecture outline: 11 sections
  1. Judge relevance against the shopper's need
    • Key terms judgment, grade
    • Ask first Before you run it, predict: of the judged pairs whose text holds every query word, what share did the judges call exact?
    • Run word-match-versus-judgment.py
    • Working rule Treat a judgment as a person's verdict about a need, not a property of the product. Before you blame a ranker for missing an exact product, check whether any field the ranker reads says what the judges saw, and look at the fields it doesn't read, such as attributes. When your judges see more than your ranker does, give the ranker the same evidence where you can, or accept that some exact products will rank low for reasons no change to the ranking can fix.
  2. Expect judges to disagree on some pairs
    • Run conflicting-judgments.py
    • Working rule Keep one grade per pair and write down the rule that chose it. We recommend the lower grade, as the sample does: it errs against the ranker that ranks the product, not in its favour. When judges disagree across two grades, exact against irrelevant, reread the guidelines with them, because they don't share a definition.
  3. Read intent as a product and its modifiers
    • Key terms intent
    • Run predicted-class-check.py
    • Working rule Read each query as an entity plus modifiers, and check that your ranker gets the entity right before you tune it for modifiers. Use query_class as a hint, for a feature or a filter you check against the judgments, and never as a label. When intent is ambiguous, as with cooktop, judge against the guidelines' reading and write that reading down.
  4. Grade relevance instead of yes or no
    • Key terms gain
    • Run binary-labels-hide-partial.py
    • Working rule We recommend graded judgments for product search, so a metric can credit a near miss without mistaking it for a match. Collect grades even if you report a binary metric too: you can always collapse grades to yes or no, but not the reverse. Use binary labels when a near miss is worthless to the user, as when the query is a model number and only that model will do.
  5. Choose the gains on purpose
    • Run gain-choice-changes-ndcg.py
    • Working rule We recommend linear gains, the grades 2, 1 and 0, as a default for product search. Choose a lower partial gain when a near miss is worth little to your shoppers. Whatever you choose, fix it before you compare rankers, report it with every score, and never compare nDCG computed with different gains.
  6. Read the label mix of each query
    • Run count-labels-per-query.py
    • Working rule Print the label mix per query before you average any metric, and look at both ends. Read a low score on a query with few exact products as a different problem from a low score on a broad query. Keep per-query scores next to every mean; chapter 3 shows what averages hide.
  7. Treat clicks as biased evidence
    • Run cascade-clicks.py
    • Working rule Use judgments, not clicks, as the reference for offline evaluation. Use clicks as relative evidence, a clicked product over a skipped one ranked above it, and never read a missing click as a sign that a product is irrelevant. If you train on clicks, check the result against judgments.
  8. Know where the judgments came from
    • Run trundle-daybeds-unjudged.py
    • Working rule Find out where your pool came from before you trust a comparison. When you evaluate a ranker unlike the systems that built it, expect it to surface products nobody judged, and plan to judge them.
  9. Count unjudged products before you compare scores
    • Key terms Judged@k
    • Run unjudged-in-bm25-top-10.py
    • Run condensed-ndcg.py
    • Working rule Report judged@10 next to every nDCG@10, and compute condensed scores as a diagnostic, never as the headline. When two rankers' judged@10 differ, have the unjudged products in their top 10s judged before you compare them. That is the dependable cure, and on this sample it is a small job: 22 judgments for BM25's test run. On a full catalog, expect more.
  10. What goes wrong in production
    • Working rule Date your judgments, track judged@10 in production, and budget for judging again.
  11. Decisions this chapter made
    • Key terms relevant

About 32 minutes to read, 12 examples.

By the end, students can compute the standard ranking metrics from scratch, check them against scikit-learn, choose one before you compare rankers, and tell whether the gap between two rankers on your test queries is established or could be chance.

Lecture outline: 13 sections
  1. Count the relevant products in the top k
    • Run precision-recall-by-hand.py
    • Working rule Use precision@10 when you care how much of the first page is right, and read it next to each query's ceiling. Don't make recall@10 a headline in product search, where a query can have more than a hundred relevant products, as outdoor light fixtures has. We recommend recall at a deep cutoff, such as the top 100, for one job only: checking that retrieval passes enough relevant products to a reranker. Even then, read it as an overestimate.
  2. Score the first relevant result with reciprocal rank
    • Run reciprocal-rank-by-hand.py
    • Working rule Report MRR as the headline only when shoppers need one right product, as for a model number or a named product such as kohler whitehaven farmhouse kitchen sink. For queries where shoppers browse a page of options, such as wooden chair outdoor, use nDCG@10 as the headline and report MRR beside it.
  3. Build nDCG from discounted gains
    • Run ndcg-from-scratch.py
    • Working rule Use the linear-gain, log2(i + 1) form at a cutoff of 10, with the ideal ranking built from every judged product, and state that variant whenever you quote a number. Compare nDCG only between runs scored on the same judgments: new judgments change the ideal ranking, and with it every score.
  4. Match scikit-learn's ndcg_score
    • Run ndcg-matches-scikit-learn.py
    • Working rule Report your own nDCG, and use ndcg_score as an independent check, with every judged product in each query's row and scores taken from ranks. Before you compare the two, check that every run holds at least k products. If you pass a ranker's raw scores instead, break ties in a fixed order first, such as by product_id, or the two will disagree wherever scores tie.
  5. Choose a metric before you compare
    • Run metric-choice-on-train.py
    • Working rule Choose one headline metric, its gains and its cutoff before you compare rankers, and write them down. We recommend nDCG@10 with linear gains when judgments are graded and shoppers browse a page. Switch to exponential gains when an exact match is worth far more than a near one, as for a spare part that must fit, and to MRR when most queries have one right answer. Report the other metrics beside the headline, and never change the headline after you've seen which ranker it favours.
  6. Read per-query differences, not only the mean
    • Run per-query-hybrid-versus-bm25.py
    • Working rule For every comparison, print the mean and median difference, the counts of wins, losses and ties, and the queries with the largest losses, then read those queries. Check judged@10 on the queries that move most before you credit the ranker. Report absolute differences of means, not percentages.
  7. Test a gap with a permutation test
    • Ask first Before you read the output, predict: will the test find the hybrid's deficit larger than chance?
    • Run permutation-test-by-hand.py
    • Working rule Test every gap you intend to act on with a paired permutation test on the per-query differences. The book decides on this test, as its rule sets out. Read a p-value above the threshold as "no evidence of a difference", never as "no difference": with this few queries, a test can miss a real gap, and McKechnie et al. argue for measuring such misses alongside false positives (McKechnie et al., 2025). Never explain a gap the test can't tell from chance.
  8. Get the p-value from a paired t-test
    • Run paired-t-test-by-hand.py
    • Working rule Report the paired t-test, scipy.stats.ttest_rel, beside the permutation test, as a check that needs no resampling. When the two fall on different sides of the threshold, say so, and let the permutation test decide. Don't use the Wilcoxon or sign test to compare mean scores.
  9. Report how big the gap is
    • Key terms confidence interval
    • Run bootstrap-interval-by-hand.py
    • Run interval-coverage-check.py
    • Working rule Report every comparison as the mean difference, with its bootstrap interval, its effect size and its p-value. If the interval includes 0, the comparison shows no improvement. If it excludes 0 but lies below the smallest gap worth the change's cost, the difference may be real and still not worth shipping. Use the interval to describe a gap's size, not to decide whether it exists.
  10. Decide in advance what counts as established
    • Key terms Bonferroni correction
    • Working rule Write the list of planned comparisons before you score anything on the test queries, and keep it short, since each one raises the bar for the others. When you need to compare many rankers at once, Bonferroni gets very strict. Sakai recommends an analysis of variance, which tests whether any of the rankers differ at all, followed by the randomised Tukey HSD test, which compares every pair while holding the chance of any false positive at the level (Sakai, 2014).
  11. Compare runs in an evaluation harness
    • Run compare-chapter-one-rankers.py
    • Working rule For every change, report the mean nDCG@10 and judged@10 of each run, then a table like this one, with a column for each comparison and the exploratory ones labelled. A change ships only when it is established against the ranker to beat.
  12. What goes wrong in production
    • Run split-changes-the-verdict.py
    • Working rule Log every comparison you run on the test queries, treat a borderline or exploratory result as a question for new queries, break score ties in a fixed order, and report judged@10 next to every nDCG@10.
  13. Decisions this chapter made

4. Judgments from people and LLMs

About 33 minutes to read, 11 examples.

By the end, students can write judging guidelines, measure how well two judges agree with kappa and weighted kappa, read an LLM judge's errors from a confusion matrix, and decide where LLM judgments can stand in for people's and where they can't.

Lecture outline: 13 sections
  1. Write judging guidelines
    • Key terms judgment, grade
    • Run print-judging-guidelines.py
    • Working rule Write guidelines as definitions, a procedure, a worked example with a reason for each grade, and a rule for each case your judges have disagreed on. Log disagreements and revise the guidelines when one kind keeps coming back. Give an LLM judge the same guidelines as people, plus a rule for what it can't see, and then measure what it does, because a rule you write isn't a rule the judge follows.
  2. Measure agreement beyond chance with kappa
    • Key terms observed agreement
    • Run kappa-by-hand.py
    • Working rule Report kappa, with observed agreement beside it, never observed agreement alone. Measure it on pairs people judged for train queries, and label any threshold you use as a rule of thumb. With more than two judges or missing labels, use Krippendorff's alpha, which compares observed with chance disagreement and handles "Any number of observers, not just two" and "Incomplete or missing data" (Krippendorff, 2011).
  3. Give near misses partial credit with weighted kappa
    • Run weighted-kappa-by-hand.py
    • Working rule For ordered grades, report linear weighted kappa next to plain kappa, and count the far misses separately. Weight each disagreement by the gap in gain it costs. With the book's linear gains, that's linear weights. With exponential gains of 0, 1 and 3 (Choose a metric), the gaps are 1, 2 and 3, which neither of scikit-learn's options gives, so build that matrix as the example builds the linear one.
  4. Read kappa against the label mix
    • Run kappa-under-two-label-mixes.py
    • Working rule Report the label mix with every kappa, and never compare kappas from samples with different mixes, your own or published ones. We recommend drawing the agreement sample the way you'll use the judge: if it will grade your rankers' top 10s, sample pairs from top 10s on train queries.
  5. Read the direction of the LLM's errors
    • Run llm-confusion-matrix.py
    • Working rule Read the confusion matrix before you read kappa, on pairs drawn the way you'll use the judge, and name the direction and the largest error for that pool. Don't carry a direction from one pool to another, and don't predict a ranker's nDCG@10 from it. Compare rankers with each other under the same grades, and never compare a score under LLM grades with a score under human grades.
  6. Compare what the LLM saw with what the judges saw
    • Run llm-grades-follow-written-words.py
    • Working rule Give an LLM judge every field your judges relied on that you can turn into text, such as attributes and the full description, and write down what it can't see. Before you count a disagreement against the LLM, check whether the evidence was in what it saw. When your judges' evidence is visual, measure whether the LLM grades exact products lower, as this one does on both pools, or give it the images if your model accepts them.
  7. Measure how much the prompt changes the labels
    • Run prompt-disagreement.py
    • Working rule Treat the prompt as part of the judge. Record every prompt word for word with the model version, as the recording's manifest does, and the settings and answer schema too, which this recording keeps in its script. Treat a change of prompt as a different judge whose agreement you measure again. Run at least two prompts, and trust a conclusion from LLM grades only when both give it. Choose one prompt over another on train pairs, and only when the interval for their difference excludes 0.
  8. Fill unjudged pairs with LLM grades
    • Run fill-unjudged-pairs.py
    • Working rule Fill the unjudged pairs in your rankers' top 10s with LLM grades as a diagnostic, under every prompt you run, and report the filled score beside the human-only one with the number of pairs filled. If filling changes the order of two rankers or a verdict, have people judge those pairs. Never let a filled grade replace a human one.
  9. Check whether LLM grades order the rankers the same way
    • Ask first Before you run it, predict: will LLM grades put the four rankers in the same order as human grades?
    • Run kendall-tau-four-rankers.py
    • Working rule Report tau with the list of swapped pairs, never alone. With a handful of rankers, where one swap moves tau by 0.333, don't read it against 0.9. Then test whether the swapped pairs differ at all, with the planned comparisons under each set of grades.
  10. Rerun the planned comparisons with LLM grades
    • Run planned-comparisons-llm-grades.py
    • Working rule Never ship a change on a verdict from LLM grades alone. Use LLM grades to decide which candidates are worth judging, and decide on human grades. When two prompts give different verdicts, the LLM can't settle the comparison, and people must.
  11. Correct LLM scores with a human sample
    • Key terms Spread
    • Run correct-llm-scores-with-humans.py
    • Working rule Whenever you report a score from LLM grades, have people judge a random sample of queries and report the corrected estimate. On that sample, check how well the LLM's per-query gaps correlate with the human ones. When they correlate weakly, spend the budget on judging more queries.
  12. What goes wrong in production
    • Working rule Pin the judge, keep a human sample in every evaluation, rerun agreement when anything changes, and never score a ranker with the judge you tuned it on.
  13. Decisions this chapter made

5. Lexical retrieval

About 30 minutes to read, 13 examples.

By the end, students can build lexical search from scratch:

  • turn product text into terms
  • index the terms
  • score products with TF-IDF and BM25, matching the bm25s library score for score
Lecture outline: 12 sections
  1. Split product text into terms
    • Run tokenize-product-text.py
    • Working rule Print the terms that your queries and products become before you tune any scoring, and run both through the same tokenizer. Start from bm25s's defaults, as the store does. When shoppers type sizes, quantities or model numbers, use a token pattern that keeps single digits and splits a number from its unit, such as the one above. When they type negations, turn the stop list off. Each is a change to search, so measure it on the train queries and check it on test queries before you keep it.
  2. Build an inverted index
    • Run inverted-index-by-hand.py
    • Working rule When a query's results look wrong, look up its terms in the index before you change the scoring. A term with no postings, such as 7qt, can't help any product whatever the scorer does, so fix the tokenizer or the query instead. For queries that shoppers type, match products that contain any of the terms rather than all of them, because one unmatched word would otherwise empty the results. Serve from a library's index, and keep a small one like this to check what a term matches.
  3. Weight terms with TF-IDF
    • Run tf-idf-by-hand.py
    • Working rule Weight terms by IDF in any lexical scorer, because a match on a rare term is better evidence. Don't rank products by raw TF-IDF: cap what a repeated term can add, and correct for long texts, which is what BM25 does. Keep TF-IDF for explaining a ranking, where you can read off each term's tf and idf.
  4. Rebuild BM25 by hand to match bm25s
    • Run bm25-by-hand.py
    • Working rule Serve BM25 from a library such as bm25s, and keep a small version you wrote yourself to check it, because yours shows where every part of a score comes from. Compare two implementations by their scores first, then by their rankings. Before you compare rankings, break ties the same way in both, by product ID as the book's BM25 does, or tied products alone can make two correct implementations rank differently. When two BM25 systems rank differently, compare their tokenizers before their formulas. The study of variants held tokenization and stop words fixed, and notes that more mundane settings, such as the choice of stop words, often make larger differences than the formula (Kamphuis et al.).
  5. Stem terms so word forms match
    • Run stemmer-merges-word-forms.py
    • Run bm25s-with-a-stemmer.py
    • Working rule Stem English product text and queries with the Snowball English stemmer, at indexing time and at query time alike: in bm25s, pass stemmer= to bm25s.tokenize on both sides. Read the products that a stem adds, as the floats counts do, before you trust a gain. When a stem merges words that your shoppers keep apart, such as molding and mold, keep a list of words you don't stem. For text in another language, use that language's stemmer.
  6. Check stemming on the test queries
    • Run measure-stemming.py
    • Working rule Choose a text-processing change on train queries, and keep it only if its planned test comparison is established. When the permutation test and the t-test disagree, read the largest per-query differences, and report the gain without the largest one beside the mean. Stemming is established, so stem when you build lexical search. The book's shared BM25 stays unstemmed, so that later chapters' results stay comparable (decisions). Treat the size of the gain as fragile.
  7. Tune BM25's parameters on train queries
    • Ask first Before you run it, predict: at the defaults, stemmed BM25 scores 0.774 on the test queries. How much will the best setting on train add?
    • Run sweep-k1-and-b.py
    • Working rule Start from the library's defaults. Tune k1 and b on train queries when you change what you index, such as fields or stemming, and keep a tuned setting only if its planned test comparison is established. The store's tuned setting isn't, so the store keeps k1=1.5 and b=0.75.
  8. Weight the product name above the description
    • Run fields-scored-apart.py
    • Run repeat-the-name-field.py
    • Working rule Don't add up BM25 scores from separate fields. To weight a field, combine term frequencies before saturation, with BM25F or by repeating the field. Tune the weight on train queries, and keep it only if its planned test comparison is established. For the store, one combined field stays.
  9. Index product attributes as a field
    • Run add-attributes-field.py
    • Working rule Keep structured attributes, and read the queries they help, but don't add them to the BM25 text on the strength of a few queries. Index the values without the names, so that a query word such as size doesn't match every product with a size, and keep the field only if its planned test comparison is established. For the store, the attributes stay out of the BM25 text. When a modifier names an attribute value exactly, such as a color, a filter or a ranking feature can use it without adding its words to every product's text.
  10. Match phrases with term positions
    • Run phrase-positions-by-hand.py
    • Run phrase-bonus-train-and-test.py
    • Working rule Don't add a phrase bonus to every query by default, and keep one only if its planned test comparison is established. Build positions when shoppers quote phrases, or when word order changes what they want, as between table lamp and lamp table. Prefer a bonus that you measure on train queries to a filter, because a filter removes products like the hardware kit.
  11. What goes wrong in production
    • Working rule Keep a set of probe queries for negations, sizes and codes, and read their results after every change to text processing. Store the tokenizer's and stemmer's settings with the index they built, log the terms that each query becomes, and watch for queries whose terms have no postings.
  12. Decisions this chapter made

6. Semantic retrieval

About 30 minutes to read, 16 examples.

By the end, students can build semantic retrieval for a catalog from stored embeddings:

  • compare vectors by cosine similarity
  • find a query's nearest products exactly with NumPy
  • switch to an approximate index when a catalog outgrows exact search and measure what the index misses
  • find the queries where embeddings go wrong
Lecture outline: 13 sections
  1. Turn text into embeddings
    • Key terms embedding
    • Run inspect-recorded-embeddings.py
    • Working rule Embed products ahead of time and queries at search time, with the same model at the same revision, and follow the model card on prompts: here, a prefix on queries only. Record the model, revision and prompts next to the vectors, as each manifest.json in examples/data/recorded/ does. Check that your longest product fits the model's input limit before you trust its embedding.
  2. Compare embeddings with cosine similarity
    • Key terms Cosine similarity, normalizing
    • Run cosine-by-hand.py
    • Working rule Normalize every vector once, when you store it, and rank by dot product. Normalize again after anything that changes a vector, such as shortening it or averaging several. If a model's documentation says it was trained for the dot product of unnormalized vectors, follow the documentation instead, and never mix the two in one index.
  3. Find the exact nearest neighbours with NumPy
    • Run exact-search-numpy.py
    • Run exact-search-memory.py
    • Working rule We recommend exact search until it no longer fits in memory or meets your latency budget, measured with your own catalog on your own hardware. It has no settings to tune and never misses a neighbour, so it is also the ground truth for any approximate index. Store the normalized vectors as one 32-bit matrix, take the top k with argpartition, and sort only those.
  4. Trade recall for speed with HNSW
    • Run hnsw-efsearch-sweep.py
    • Working rule Use an ANN index only when exact search is too slow or too large. Tune it on train queries by its recall against exact search, at the k your pipeline uses, and choose by the worst query, not the mean. We recommend the smallest efSearch at which every query keeps at least 9 of its true top 10, with M = 32. This chapter tunes only efSearch; M and efConstruction change the graph itself, and deserve the same sweep on a large catalog. Very different ANN methods "yield comparable quality-performance trade-offs" (Aumüller et al., 2018), so measure on your own data rather than choosing a method by reputation.
  5. Check the index on the test queries
    • Run hnsw-test-recall.py
    • Run hnsw-index-memory.py
    • Working rule Check a tuned index on held-out queries by its recall against exact search, and read the queries that lose their whole top k before the mean score. Treat a fall in nDCG@10 that rests on a handful of queries as unproven until a paired test establishes it. Budget memory for the links as well as the vectors.
  6. Check queries that contain digits
  7. Expect a code to carry little weight
    • Run code-probes.py
    • Working rule List the queries in your log that hold sizes, quantities or codes, and compare both rankers on them before you trust either. When a code matters, match it lexically. When an exact size matters, match it against a product field such as the width in attributes, or lexically once the tokenizer keeps digits and units are written one way (chapter 5, chapter 8). Leave the rest of the query to semantic retrieval. Chapter 7 tests combining the two.
  8. Test embeddings on negation
    • Ask first Before you run it, predict: how many of semantic retrieval's top 10 for rug that is not round are named round?
    • Run negation-probes.py
    • Working rule Don't rely on a first-stage retriever, lexical or semantic, to handle negation. When shoppers exclude an attribute your catalog stores, such as shape, material or arms, detect the exclusion and filter on the attribute. Otherwise, rerank: in van den Elsen et al.'s reproduction, cross-encoders and LLM rerankers handled negation far better. Chapter 9 reranks the store's semantic retrieval with a cross-encoder and LLMs, and none of its rerankers beats semantic retrieval on the store's judged queries, which can't test negation. Keep probes like these as a check to rerun whenever you change models or add a reranker.
  9. Choose an embedding model
  10. Read benchmarks as a shortlist
  11. Truncate dimensions, then test the loss
    • Run truncate-embeddings.py
    • Run truncation-paired-test.py
    • Working rule Shortlist models on licence, language, size and domain, and treat benchmark ranks as a hint. Decide between models, or between vector sizes, on your own train queries with a paired test, and read the interval as well as the p-value. Shorten vectors only for a model trained for it, normalize after shortening, and only when memory or search time requires it.
  12. What goes wrong in production
    • Run similarity-scale.py
    • Run spelling-probes.py
    • Working rule Don't filter on a fixed similarity. Record the model, revision and prompts with every vector, and check them where queries are embedded. Rebuild and re-measure the index as the catalog changes, count tokens before you embed, and correct spelling before you embed a query when your log shows typos, with a corrector that checks each word's neighbours (chapter 8).
  13. Decisions this chapter made

7. Hybrid retrieval

About 32 minutes to read, 12 examples.

By the end, students can combine a lexical and a semantic ranking into one, either by adding normalized scores or by reciprocal rank fusion, tune how much each ranking counts on train queries, and decide with a paired test on test queries whether the combination beats the better of its two inputs.

Lecture outline: 12 sections
  1. Find the queries each retriever wins
    • Key terms fusion
    • Run bm25-semantic-wins-by-query.py
    • Run overlap-and-union-recall.py
    • Run two-queries-two-winners.py
    • Working rule Before you fuse, count per query where each retriever wins on your train queries, and check how many relevant products the union of their lists holds that the better one misses. Consider fusing when each retriever wins a real share of queries and the union holds more relevant products than either list. That is necessary, not sufficient: next, check each retriever alone, and test the tuned blend on held-out queries before you adopt it.
  2. Check each retriever before you fuse
    • Run fusion-gain-by-winning-retriever.py
    • Working rule Check each retriever alone on train queries, with judged@10 and a paired test, before you fuse. When one is weaker and wins fewer queries, don't give it an equal vote: weight it and tune the weight, or leave it out. Equal votes suit two retrievers of about the same quality.
  3. Normalize scores before you add them
    • Run raw-scores-versus-min-max.py
    • Working rule Never add raw BM25 and cosine scores with a guessed weight. We recommend per-query min-max as the default, because it needs no knowledge of either scoring function. Use theoretical min-max when you know each score's lowest possible value and want a floor that doesn't move with the candidates. In either case, tune α on train queries.
  4. Test reciprocal rank fusion's settings on train queries
    • Run rrf-k-on-train.py
    • Working rule Keep k = 60 as the default, and fuse lists at least 100 deep, because a deeper list can only add candidates. Move off k = 60 only when the best k on train beats it by a margin the permutation test can tell from chance, with p below 0.05. When one retriever is the stronger one, as in the store, k isn't the setting to change: RRF's equal vote is. Give the rankings weights and tune them.
  5. Tune the blend's weight on train queries
    • Run weighted-rrf-one-product.py
    • Run tune-blend-weights-on-train.py
    • Working rule Tune one weight on train queries over a coarse grid, and keep the normalization at min-max unless another one beats it by a margin the test on train queries can tell from chance, with p below 0.05. Treat the train gain over the best single retriever as optimistic, and expect it to shrink on test queries, a rule of thumb for any setting picked as the best of several. Before you trust the setting, check that it's stable, then report it on test queries.
  6. Check that the tuned weight is stable
    • Run blend-weight-bootstrap.py
    • Run tuning-with-ties-reversed.py
    • Working rule Bootstrap the train queries and tune on each sample, then tune once more with ties listed in another order that carries no information. If the chosen weight scatters across the grid, use a coarser grid or more train queries, or prefer the simpler setting. If the tie order alone moves it, say so wherever you report it. If it holds steady under both checks, take it to the test queries.
  7. Report the tuned blends on test queries
    • Ask first Before you read the output, predict: on the test queries, will the weighted sum with α = 0.7 and weighted RRF with α = 0.6 score above semantic retrieval, level with it, or below it?
    • Run tuned-blends-on-test.py
    • Working rule Adopt a blend only when it beats the best single retriever on held-out queries, with a permutation p-value below a threshold corrected for the number of planned comparisons. When it doesn't, keep the simpler system. For the store, that leaves semantic retrieval alone as the ranker to beat. Keep the tuned weights and the code, and test the blends again when a retriever or the catalog changes.
  8. Learn sparse term weights with SPLADE
    • Run exact-products-missing-query-words.py
    • Working rule Consider learned sparse retrieval when BM25's misses are mostly vocabulary misses and you want to keep an inverted index. Check the model's licence first, and measure it like any retriever: alone, per query, on train queries, then on test.
  9. Match token by token with ColBERT
    • Working rule Consider late interaction when single-vector retrieval misses details that hang on one word, and you can afford a vector per token. For a catalog the size of the store's, we recommend testing a reranker on a shortlist first (chapter 9): it reads the same details at query time and needs no new index, though none of chapter 9's rerankers shows an established gain over the store's semantic retrieval.
  10. Decide when hybrid retrieval pays
    • Working rule We recommend trying a hybrid when your two retrievers are close in quality, when your embedding model works outside the domain it was trained for, or when many queries depend on exact terms your embeddings miss. Start with weighted RRF: it needs ranks only, and on the store's train queries the permutation test couldn't tell it from the weighted sum. Move to a weighted sum of normalized scores when both engines give you scores for every candidate: two studies found it ahead of RRF on their data (Bruch et al., 2023; Abdallah et al., 2026), and on the store's train queries its weight was the more stable. When one retriever is established as the stronger on held-out queries and no tuned blend beats it there, use it alone, keep the other as a baseline, and test a lexical filter for queries with exact constraints. That is the store's case.
  11. What goes wrong in production
    • Working rule Log both retrievers' ranks next to the fused result, check the list depth in every engine's configuration, and re-tune α whenever a retriever changes.
  12. Decisions this chapter made

8. Understanding queries

About 32 minutes to read, 12 examples.

By the end, students can change a shopper's query before retrieval:

  • correct its spelling
  • write its units one way
  • expand it with synonyms
  • predict its product class
  • rewrite it with a large language model
  • search with a product that model imagines
  • turn a follow-up into a standalone query
Lecture outline: 10 sections
  1. Correct spelling against the catalog's words
    • Run catalog-spelling-corrector.py
    • Run spelling-context-counts.py
    • Run measure-spelling-corrections.py
    • Working rule Build the lexicon from your catalog, and from your query log once you have one, and correct only words it doesn't hold, so that brand names survive. Before you correct a word, check its neighbours: a corrector without context turns variant spellings into common words and misses real-word errors. Score each correction on the typo queries' own judgments, and collect many more typo queries than the store has before you test a corrector.
  2. Normalize units the way the catalog writes them
    • Run normalize-units-both-sides.py
    • Working rule Write normalization rules from the catalog's text, not from the queries, by counting every way it writes each unit. Print the terms the tokenizer makes of the normalized text, because a rule that looks right can produce terms that match the wrong products. Apply the same rules to products and queries in one release, and rebuild the index. When the catalog keeps sizes in fields, such as the width in attributes, match the size against the field instead, as chapter 6 suggests.
  3. Expand queries with a synonym list
    • Key terms query drift
    • Run synonym-expansion-train-and-test.py
    • Working rule Keep a synonym list for vocabulary gaps you have seen, write each entry from a failing query, and read the products each entry adds before you keep it. Never report a list's gain on the queries you wrote it from, and keep an entry only if a test on held-out queries shows a gain. Test any expansion of semantic retrieval first: the LLM's rewrites, which add synonyms and also broaden and drop words, hurt it. When the gaps are too many to list, try stemming or learned sparse retrieval instead.
  4. Train a query classifier on product text
    • Run query-classes-by-hand.py
    • Working rule When you have no labelled queries, train a query classifier on your products' classes and check it against the classes of judged exact products, not against another classifier's guesses. Once your logs give you labelled queries, train on those. Treat the predicted class as a hint for ranking, never as a label.
  5. Boost the predicted class instead of filtering
    • Run class-boost-train-and-test.py
    • Working rule Don't filter on a predicted class. Boost it with a weight tuned on train queries, and keep the boost only if its planned test comparison is established. The store's isn't, so its semantic retrieval stays unboosted. We recommend a class filter only when the shopper chose the class, for example by clicking a category.
  6. Rewrite queries with an LLM
    • Ask first Before you run them, predict: on the test queries, will the rewrite help BM25, the weaker retriever, hurt semantic retrieval, the stronger one, or neither?
    • Run llm-rewrites-on-train.py
    • Run llm-rewrites-on-test.py
    • Run rewrite-largest-changes.py
    • Working rule Don't put an LLM rewrite in front of a strong retriever without a planned test on held-out queries. Read the per-query losses for dropped requirements and broadened products, which the prompt's "keep every requirement" doesn't prevent. For the store, semantic retrieval, the ranker to beat, keeps searching the query as typed.
  7. Search with a hypothetical product (HyDE)
    • Run hyde-train-and-test.py
    • Working rule When you try HyDE, always average the query's own embedding in. Expect the largest gains with an encoder trained without labels; with a fine-tuned encoder, Gao et al. saw gains only from their largest LLM, and the store, with its model, prompt and encoder, saw none. On an encoder like the store's, which already beats BM25 on these queries, test HyDE before you pay for an LLM call per query. The store doesn't use it.
  8. Turn a follow-up into a standalone query
    • Run follow-up-refinements.py
    • Working rule Never search a follow-up alone. Join it to the earlier turns at least, or rewrite it to stand alone. When the refinement names a value the catalog states, extract it and boost or filter on it, and fall back to the unfiltered list when too few products remain. Measure on conversations from your own logs before you trust any of it.
  9. What goes wrong in production
    • Working rule Log the query as typed and as rewritten, cache LLM rewrites, version the model and prompt with the outputs, and rebuild the lexicon and the classifier whenever the catalog changes.
  10. Decisions this chapter made

9. Ranking

About 32 minutes to read, 14 examples.

By the end, students can describe each product that retrieval returns with features, train a learning-to-rank model with LightGBM on your own judgments, rerank the top of the list with a cross-encoder or an LLM, measure an LLM reranker's positional bias, set the rerank depth by counting its costs, and distil a slow reranker into a fast one.

Lecture outline: 12 sections
  1. Describe each candidate with features
    • Run ranking-features-by-grade.py
    • Working rule Start with the features you already compute: each retriever's score, a reranker's score if you have one, matches on fields such as the name, and popularity signals. Compare each feature by grade on train queries before you train, with means for scores and medians for skewed counts. A feature that doesn't move with the grade has little to teach the model. Compute features with the same code when you train and when you serve.
  2. Learn a ranking from pairs of products
    • Run lambdas-for-one-query.py
    • Run lambdas-match-lightgbm.py
    • Working rule Train rankers with a gradient that weighs each pair by the metric you report, as LambdaMART does, rather than regressing on grades. Know which defaults of your library change that gradient, and check on train queries that it beats a pointwise model on your data, because the families' order in a benchmark doesn't transfer by itself.
  3. Train LambdaMART with LightGBM
    • Run lambdamart-tree-size-on-train.py
    • Run lambdamart-choices-on-train.py
    • Working rule Sort rows by query and pass group, set label_gain to the gains your metric uses, stop early on held-out train queries through eval_X and eval_y, and fix the seed and thread count. Put each query's candidates in an order that carries no information, such as by product ID, and break score ties the same way. Count unjudged candidates as grade 0 unless cross-validation on train queries says otherwise. With few judged queries, try small trees.
  4. Report LambdaMART on the test queries
    • Run lambdamart-on-test.py
    • Working rule Adopt a learned ranker only when its planned test comparison is established. Read its gain importance before you trust it, because the features it leans on are the ones you'll pay for at query time. Early stopping on a few held-out queries keeps very different numbers of trees from one split to the next, from 99, 20, 8, 43, 38 across the cross-validation folds to 213 here. A rule of thumb, not a measurement: with this few queries, judge more queries before you tune further, and treat any one stopping point as noise.
  5. Rerank semantic retrieval with a cross-encoder
    • Run cross-encoder-on-semantic.py
    • Working rule Rerank your strongest retrieval, at a depth fixed before you look at test results, and adopt the reranker only on an established gain on held-out queries. For the store, the general-purpose cross-encoder stays out of the pipeline.
  6. Score each product with an LLM
    • Run llm-pointwise-rerank.py
    • Working rule If you rerank pointwise with an LLM, ask for a graded label or a score, count the distinct scores it gives per query, and decide the tie-break in advance: retrieval's order is a safe default. Score the result with human judgments, never with the same LLM's grades.
  7. Order the 20 products with one LLM prompt
    • Run llm-listwise-rerank.py
    • Working rule Check every listwise answer before you use it: count missing, repeated and unknown numbers, and log how many answers needed repair. Constrain the answer's shape with a schema when your runtime supports one. Fix the window, the step and the order you show the products in before you measure, because the order changes the answer, as the next check shows.
  8. Reverse the input to measure positional bias
    • Ask first Before you run the comparison, predict: for how many of the 30 test queries will the LLM put the same product first both times?
    • Run listwise-position-bias.py
    • Working rule Before you trust a listwise LLM reranker, run each query in at least two orders and report how much the answers agree. When agreement is low, aggregate several orders or ask each pair in both orders, and count those calls against your budget. In production, show the products in retrieval's order every time, so that the same list at least gets the same answer.
  9. Set the rerank depth from a budget
    • Run reranker-prompt-sizes.py
    • Run rerank-depth-on-train.py
    • Working rule For each reranker you consider, count calls and characters per query at each depth, and set the depth on train queries. A rule of thumb, not a measurement: rerank only as deep as the depth where train nDCG@10 stops improving, because every product past it costs the same and adds nothing. Then time that configuration on your own hardware and traffic, and keep the stage only if it fits the latency budget.
  10. Distill the cross-encoder into a small ranker
    • Run distillation-on-train.py
    • Run distilled-ranker-on-test.py
    • Working rule Distil when a reranker earns an established gain but costs too much per query: label every candidate of your train queries with the teacher, train the student on features you can compute without the teacher, and test the student against your retrieval on held-out queries. Read the student's gain importance: when retrieval's own score dominates it, the student is mostly your retrieval. Use margins between pairs when the student can't match the teacher's scale. The store has no teacher that beats semantic retrieval yet, so it keeps the code and doesn't ship the student.
  11. What goes wrong in production
    • Working rule Log features and model versions with every ranking, put candidates in an order that carries no information, score on human judgments, check judged@10, and give each reranking stage a time limit and a fallback.
  12. Decisions this chapter made

10. Retrieval for LLM applications

About 32 minutes to read, 10 examples.

By the end, students can measure the retrieval behind an LLM that answers from it:

  • score an answer by the products it cites
  • check each citation against what retrieval returned
  • measure the products the LLM reads
  • choose what to index as one retrievable unit
  • read an agent's searches from the record of its tool calls
Lecture outline: 10 sections
  1. Put retrieved products in the prompt
    • Run catalog-against-the-context.py
    • Working rule Put a few retrieved items in the prompt, not the collection, and choose how many by measuring answers on train queries at several sizes, because more context can help or hurt. The store's recording answered only the test queries, five products each, so five is a fixed setting here, not a measured one. Consider sending the whole collection only when it fits well inside the window, and then test it against retrieval on the same queries.
  2. Score each answer by the products it cites
    • Ask first Before you run it, predict what the LLM does with the random context, where the prompt tells it to "say plainly if none of them fit". Of the 30 answers, how many will recommend nothing?
    • Run rag-answers-by-context.py
    • Working rule Score every answer by its cited products' human grades, and report beside it how many answers recommend nothing, split by whether the context held an exact product. Keep a random context as a floor whenever you change the model or the prompt: an answer that recommends something from random products has invented a fit. Test a retriever's gain on the answers too, because a gain in ranking needn't carry through.
  3. Check every citation against the context
    • Key terms Grounding
    • Run citations-inside-the-context.py
    • Working rule Run the ID check on every answer, offline and before a shopper sees it, and treat an ID outside the context as a failed answer: drop it or retry, and never show it. Ask for citations in a field of their own so that the check doesn't parse prose, and check that the text and the field agree. Count the instructions the answers break, such as the number of recommendations, and enforce in code the ones that matter.
  4. Check what the cited product supports
    • Key terms Faithfulness
    • Run faithful-but-partial.py
    • Working rule Score answers on your judgments first, because they say whether a recommendation fits. Add a support check for the claims an answer makes about each product, by an NLI model or an LLM judge. Calibrate that judge on a human sample, as chapter 4 does and as ARES does: "To mitigate potential prediction errors," it uses "a small set of human-annotated datapoints for prediction-powered inference (PPI)", a method that corrects the judge's estimate with the human labels (Saad-Falcon et al., 2024). For queries you have no judgments for, a support check is all you have, so say so when you report it.
  5. Measure the context, then the answer
    • Key terms Context precision, Context recall
    • Run context-measures-by-hand.py
    • Run context-measures-against-answers.py
    • Working rule For a context of fixed size, report the context's precision@k or context precision from your judgments, with recall@k beside it. Use recall when an answer must cover every relevant item, such as every part that fits one appliance. Without judgments, Ragas's LLM-judged versions estimate similar quantities; calibrate them on a human sample before you trust them. Choose k by measuring answers on train queries at several values of k, which takes answers recorded for those queries.
  6. Choose the retrievable unit
    • Key terms retrievable unit, chunking
    • Run retrievable-unit-on-train.py
    • Working rule When your items are short, index each whole item as one unit, as the store does. When they are long, chunk them, put the item's title in front of every chunk, and choose the chunk size on train queries by the context measures of the previous section. Pass the LLM the whole item, or the chunk with its title, so it knows what it is reading. Before you trust a small gap, count the queries whose top k changes when ties break the other way.
  7. Read an agent's searches from its trace
    • Key terms tool, tool call, trace
    • Run agent-searches-in-traces.py
    • Run agent-later-searches.py
    • Working rule Read the traces before you score an agent. Count its searches per task, compare each search with the shopper's words, and find where added words come from. Add the agent's searches to the queries you test your retriever on, scored with the judgments for the shopper's request or judged afresh, because a retriever tuned on people's queries hasn't been tested on an agent's. Expect an agent that mostly passes the shopper's words through to be bounded by the same retrieval as a RAG answer.
  8. Compare the retrievers behind the agent's tool
    • Run agent-picks-by-tool.py
    • Working rule Test the retriever behind an agent's tool as you test it for people, on the agent's own searches and on its final picks, scored with human judgments. Check that every pick came from a tool result. Count the tasks where the tool never returned a relevant product, because no agent can fix those and only retrieval can.
  9. What goes wrong in production
    • Working rule Version the model, prompt and context settings with the answers, check citations on every answer, track abstentions, and keep agents' searches in a test set of their own.
  10. Decisions this chapter made

11. Testing changes

About 30 minutes to read, 11 examples.

By the end, students can decide whether a change to search should ship:

  • diff two rankers' results query by query
  • keep your test queries from flattering a change
  • size an A/B test
  • compare two rankers by interleaving their results
Lecture outline: 12 sections
  1. Diff two runs query by query
    • Run diff-semantic-against-bm25.py
    • Working rule Keep a regression set of judged queries that nothing is ever tuned on, and run every candidate against the baseline on it. List your planned comparisons first, and report each with its permutation p-value, its t-test and its interval. Count wins, losses and ties, and read every loss before you ship. When the losses seem to share a pattern, check it on train queries before you act on it, then fix that kind of query or send it to the baseline.
  2. Weigh losses more than wins with URisk
    • Run urisk-of-two-candidates.py
    • Working rule Report URisk with α = 1 next to the mean change for every candidate. Don't ship a candidate whose URisk is negative, whatever its mean, and read its losses to find out why. Raise α when failures are expensive, and report URisk separately on the store's most frequent queries. Use α = 0 only when a loss on one query is no worse than a gain on another.
  3. Judge the products a new ranker surfaces
    • Run unjudged-score-range.py
    • Working rule For every candidate, compute judged@10 for both runs and rescore them under the two scenarios. When the leader changes, judge the unjudged pairs before you compare, and add the new judgments to the regression set for good, so that it covers every ranker you have tested. When the leader holds in both scenarios, the unjudged products are unlikely to reverse the comparison.
  4. Keep the test queries out of every choice
    • Run tuning-on-test-inflates.py
    • Working rule Choose every setting on train queries, including which candidate to take forward. Look at the test queries once per decision, and log each look: what you compared and what you chose. When a test set has guided many decisions, move it to the train side and judge a fresh set of test queries.
  5. Run an A/B test
    • Key terms sample ratio mismatch
    • Run aa-tests-with-peeking.py
    • Working rule Before the test, choose the primary metric and the guardrails that must not get worse, and size the test (with power). Run an A/A test on the same setup first. Check the sample ratio before you read any result, and read the result once, at the planned size. If you must watch a test as it runs, use a sequential method built for continuous monitoring, such as always-valid p-values (Johari et al., 2015).
  6. Size the test with power
    • Run sample-size-from-power.py
    • Run queries-needed-for-power.py
    • Working rule Size every A/B test before it starts. If it needs more users than the store sees in a few weeks, test a bigger change, use a metric that varies less, or adjust the metric with each user's data from before the test. On Bing's experiments, Deng et al. report that this "can reduce variance by about 50%" (Deng et al., 2013); how much it saves depends on how well a user's past behaviour predicts the metric. Offline, compute the smallest change your regression set can detect, and add queries when your changes are smaller. Fix the number in advance: adding queries until a power estimate says you have enough biases you toward finding significance (Webber et al.).
  7. Interleave two rankings with team draft
    • Run team-draft-one-query.py
    • Working rule Use team-draft interleaving to compare two rankers online when you need to know which one users prefer, with fewer searches than an A/B test. Use an A/B test instead when you need an absolute number, such as revenue, or when the change isn't in the ranking. By our reasoning, an interleaved list can't show the effect of latency, page layout or a different number of results.
  8. Rehearse interleaving with simulated shoppers
    • Key terms click model
    • Run one-simulated-shopper.py
    • Run simulated-interleaving-verdict.py
    • Working rule Rehearse an interleaving test in simulation to check your code and your sign test. Treat any count of searches it suggests as only as good as the assumed shopper, and rerun a simulated verdict with ties broken the other way before you read anything into it. Never report a simulated verdict as evidence about a ranker. On real traffic, fix the number of searches in advance and decide with a sign test.
  9. Find the cause when offline and online disagree
    • Ask first Before you run it, predict: which ranker gets more clicks per search?
    • Run clicks-per-search-misleads.py
    • Working rule When offline and online results disagree, find the cause before you trust either. Check that the online metric measures success, such as a purchase, and not activity. Check judged@10 on the online winner's results, and judge what is missing. Then decide on the online result of the metric you chose in advance, and add queries or judgments to the regression set so that it predicts better next time.
  10. Decide whether to ship
    • Working rule Ship when every offline item passes and the online test, sized in advance, shows a significant gain on the primary metric with no guardrail worse. When an offline item fails, fix it before going online, because online tests spend shoppers' time. When the online test is inconclusive at its planned size, keep the baseline.
  11. What goes wrong in production
    • Working rule Automate the A/A test, the sample ratio check and the fixed test size in the experiment platform, keep a log of every look at the test queries, break ties by an ID, and confirm LLM-judged gains on human judgments.
  12. Decisions this chapter made

Quiz bank

Questions on every chapter, with answer keys, for verified teachers with the Educator plan. Each answer is tied to its section. Ask for the answer keys