Glossary

The 184 terms A Primer on Search Relevance defines, each in the book's own words, with the section that defines it.

From Edition 2026.11.10, the book's current edition. Where a definition depends on a package, it says the version this edition pins and runs its examples on.

A

A/A test also A/A tests

an A/A test, in which both groups get the same system

Defined in Run an A/B test, chapter 11. In the book since Edition 2026.11.10. Permanent link to A/A test

A/B test also A/B tests, controlled experiment

An A/B test, or controlled experiment, assigns users at random to the current system or to the change, and compares a metric chosen before the test starts

Defined in Run an A/B test, chapter 11. In the book since Edition 2026.11.10. Permanent link to A/B test

agent also agents

an agent, an LLM that runs the searches itself and decides how many to run

Defined in Retrieval for LLM applications, chapter 10. In the book since Edition 2026.11.10. Permanent link to agent

agent loop

In the agent loop, the program runs each tool call, sends back the result, and the model decides what to do next.

Defined in Read an agent's searches from its trace, chapter 10. In the book since Edition 2026.11.10. Permanent link to agent loop

approximate nearest-neighbour index also ANN, ANN index

When exact search costs too much, an approximate nearest-neighbour (ANN) index skips most of its comparisons, and in return may miss some of the true nearest neighbours.

Defined in Trade recall for speed with HNSW, chapter 6. In the book since Edition 2026.11.10. Permanent link to approximate nearest-neighbour index

attribute

a catalog often keeps facts about a product as structured attributes, such as color: clear or material: glass

Defined in Index product attributes as a field, chapter 5. In the book since Edition 2026.11.10. Permanent link to attribute

attribution

attribution asks whether a cited source supports the statement it is attached to

Defined in Check every citation against the context, chapter 10. In the book since Edition 2026.11.10. Permanent link to attribution

B

baseline

The ranker the store runs today is the baseline

Defined in Diff two runs query by query, chapter 11. In the book since Edition 2026.11.10. Permanent link to baseline

bi-encoder

Search uses the encoder as a bi-encoder: it encodes each product by itself, once, ahead of time, and each query by itself when the shopper types it.

Defined in Turn text into embeddings, chapter 6. In the book since Edition 2026.11.10. Permanent link to bi-encoder

binary label

A binary label says relevant or not.

Defined in Grade relevance instead of yes or no, chapter 2. In the book since Edition 2026.11.10. Permanent link to binary label

blend

blends, ways to fuse a lexical and a semantic ranking with a weight that sets how much each counts

Defined in Hybrid retrieval, chapter 7. In the book since Edition 2026.11.10. Permanent link to blend

BM25

BM25 is a lexical scoring function with three parts.

Defined in Rank products with BM25, chapter 1. In the book since Edition 2026.11.10. Permanent link to BM25

BM25F

Combining a term's weighted frequencies across fields before saturating the total is BM25F.

Defined in Weight the product name above the description, chapter 5. In the book since Edition 2026.11.10. Permanent link to BM25F

Bonferroni correction also Bonferroni

The Bonferroni correction is a simple, conservative fix: when you test m hypotheses, test each at 0.05 divided by m

Defined in Decide in advance what counts as established, chapter 3. In the book since Edition 2026.11.10. Permanent link to Bonferroni correction

boost

A boost adds a fixed amount to the scores of products in the predicted class, so that they move up without anything else being removed.

Defined in Boost the predicted class instead of filtering, chapter 8. In the book since Edition 2026.11.10. Permanent link to boost

bootstrap

The bootstrap builds a confidence interval by resampling.

Defined in Report how big the gap is, chapter 3. In the book since Edition 2026.11.10. Permanent link to bootstrap

C

candidate also candidates

A ranker scores only candidates, the products a cheaper retrieval stage returns

Defined in Describe each candidate with features, chapter 9. In the book since Edition 2026.11.10. Permanent link to candidate

candidate ranker

the ranker with the change is the candidate ranker

Defined in Diff two runs query by query, chapter 11. In the book since Edition 2026.11.10. Permanent link to candidate ranker

candidate set

The candidate set for a query is the union of BM25's top 100 and semantic retrieval's top 100

Defined in Normalize scores before you add them, chapter 7. In the book since Edition 2026.11.10. Permanent link to candidate set

cascade model

In the cascade model, users read results from top to bottom and leave at the first worthwhile document

Defined in Treat clicks as biased evidence, chapter 2. In the book since Edition 2026.11.10. Permanent link to cascade model

chunking

chunking is the rule that cuts a document, such as a fixed number of tokens, sentences, paragraphs, or breaks chosen by meaning

Defined in Choose the retrievable unit, chapter 10. In the book since Edition 2026.11.10. Permanent link to chunking

citation also citations

A citation here is the ID an answer gives for a product it recommends.

Defined in Score each answer by the products it cites, chapter 10. In the book since Edition 2026.11.10. Permanent link to citation

click model also click models

A click model is a probabilistic model of how users scan and click a result list, used both to build metrics and to simulate users

Defined in Rehearse interleaving with simulated shoppers, chapter 11. In the book since Edition 2026.11.10. Permanent link to click model

Cohen's kappa also kappa

Cohen's kappa corrects the observed agreement for chance by comparing it with the agreement expected if each judge kept its own label shares but chose independently of the other.

kappa = (observed - expected) / (1 - expected)

Defined in Measure agreement beyond chance with kappa, chapter 4. In the book since Edition 2026.11.10. Permanent link to Cohen's kappa

ColBERT

ColBERT encodes the query and the product separately, as an embedding model does, and compares them only at the end.

Defined in Match token by token with ColBERT, chapter 7. In the book since Edition 2026.11.10. Permanent link to ColBERT

concordant pair also concordant

A pair of systems is concordant when two orderings put it the same way round

Defined in Check whether LLM grades order the rankers the same way, chapter 4. In the book since Edition 2026.11.10. Permanent link to concordant pair

condensed list also condensed

the condensed list, a ranking with its unjudged products dropped before scoring, so the judged ones move up

Defined in Count unjudged products before you compare scores, chapter 2. In the book since Edition 2026.11.10. Permanent link to condensed list

confidence interval

A confidence interval is a range of plausible values for the true mean difference

Defined in Report how big the gap is, chapter 3. In the book since Edition 2026.11.10. Permanent link to confidence interval

confusion matrix

A confusion matrix shows how two judges differ: a table of counts with one row for each of the store's grades and one column for each of the LLM's, both in the order exact, partial, irrelevant.

Defined in Read the direction of the LLM's errors, chapter 4. In the book since Edition 2026.11.10. Permanent link to confusion matrix

context

the retrieved text in a prompt is the context

Defined in Put retrieved products in the prompt, chapter 10. In the book since Edition 2026.11.10. Permanent link to context

context precision also context precision@k

Ragas's context precision is the rank-aware one of its context measures, the mean of precision@k over the positions that hold a relevant chunk

context precision@k = mean of precision@i over the positions i ≤ k that hold an exact product

Defined in Measure the context, then the answer, chapter 10. In the book since Edition 2026.11.10. Permanent link to context precision

context recall also context recall@k

context recall is the share of a reference answer's claims that the context supports

context recall@k = (exact products in the top k) / (exact products judged for the query)

Defined in Measure the context, then the answer, chapter 10. In the book since Edition 2026.11.10. Permanent link to context recall

context window

A model's context window is the most tokens it can read, a fixed number of the word pieces in its own vocabulary.

Defined in Put retrieved products in the prompt, chapter 10. In the book since Edition 2026.11.10. Permanent link to context window

control

The current system is the control

Defined in Run an A/B test, chapter 11. In the book since Edition 2026.11.10. Permanent link to control

conversational query rewriting

Conversational query rewriting turns a context-dependent question into a self-contained one with the same answer

Defined in Turn a follow-up into a standalone query, chapter 8. In the book since Edition 2026.11.10. Permanent link to conversational query rewriting

convex combination also weighted sum, CC

A sum whose weights add up to 1 is a convex combination, or weighted sum.

Defined in Normalize scores before you add them, chapter 7. In the book since Edition 2026.11.10. Permanent link to convex combination

Cosine similarity also cosine

Cosine similarity is the cosine of the angle between two vectors: 1 when they point the same way, 0 at right angles and −1 when they point in opposite directions.

cosine(q, d) = dot(q, d) / (length(q) × length(d))

  • Computed from: dot product
  • Implemented by: scikit-learn 1.9.1

Defined in Compare embeddings with cosine similarity, chapter 6. In the book since Edition 2026.11.10. Permanent link to Cosine similarity

cross-encoder

One kind of reranking model is a cross-encoder, which reads the query and the product text together, as one input, and outputs one relevance score for the pair, scoring each pair independently

Defined in Rerank the top 50 with a cross-encoder, chapter 1. In the book since Edition 2026.11.10. Permanent link to cross-encoder

cross-validation

Choose the tree size by cross-validation on the train queries: split them into five folds, rank each fold with a model fitted on three others and stopped early on the fifth, and score every query once.

Defined in Train LambdaMART with LightGBM, chapter 9. In the book since Edition 2026.11.10. Permanent link to cross-validation

D

dimension

Every text gets the same count of numbers, the embedding's dimension, whether it's a three-word query or a product's whole description.

Defined in Turn text into embeddings, chapter 6. In the book since Edition 2026.11.10. Permanent link to dimension

discordant pair also discordant

A pair of systems is concordant when two orderings put it the same way round, and discordant when two orderings put it opposite ways round.

Defined in Check whether LLM grades order the rankers the same way, chapter 4. In the book since Edition 2026.11.10. Permanent link to discordant pair

Discounted cumulative gain also DCG, DCG@k

Each result contributes its grade as a gain, shrunk by a discount that grows with its position, and the sum is the discounted cumulative gain (DCG).

DCG@k = sum over positions i from 1 to k of grade(i) / log2(i + 1)

Defined in Build nDCG from discounted gains, chapter 3. In the book since Edition 2026.11.10. Permanent link to Discounted cumulative gain

distillation

Distillation trains a small model, the student, to imitate a large model, the teacher.

Defined in Distill the cross-encoder into a small ranker, chapter 9. In the book since Edition 2026.11.10. Permanent link to distillation

distractor also distractors

Products that look right but aren't are distractors

Defined in Score each answer by the products it cites, chapter 10. In the book since Edition 2026.11.10. Permanent link to distractor

document frequency

The number of products in a term's postings is its document frequency

Defined in Build an inverted index, chapter 5. In the book since Edition 2026.11.10. Permanent link to document frequency

dot product

The dot product multiplies two vectors number by number and adds up the results, so it equals the cosine times both lengths.

dot(q, d) = q1 × d1 + q2 × d2 + ... + qn × dn

Defined in Compare embeddings with cosine similarity, chapter 6. In the book since Edition 2026.11.10. Permanent link to dot product

E

early stopping

Early stopping scores held-out train queries after every tree, and keeps the trees up to the best score once 30 more bring no gain.

Defined in Train LambdaMART with LightGBM, chapter 9. In the book since Edition 2026.11.10. Permanent link to early stopping

edit distance

The number of insertions, deletions and replacements that turn one string into another is their edit distance

Defined in Correct spelling against the catalog's words, chapter 8. In the book since Edition 2026.11.10. Permanent link to edit distance

effect size

The effect size of a paired comparison is the mean difference divided by the standard deviation of the differences

Defined in Report how big the gap is, chapter 3. In the book since Edition 2026.11.10. Permanent link to effect size

embedding

An embedding is a list of numbers, a vector, that a neural network computes for a piece of text.

Defined in Search by meaning with embeddings, chapter 1. In the book since Edition 2026.11.10. Permanent link to embedding

encoder

The model that computes an embedding is an encoder, trained so that texts with similar meanings get vectors that point in similar directions, whether or not they share words.

Defined in Turn text into embeddings, chapter 6. In the book since Edition 2026.11.10. Permanent link to encoder

evaluation harness

an evaluation harness, which holds everything fixed except the ranker: the same queries, judgments, metrics and tests, so that two runs differ only in their rankings

Defined in Compare runs in an evaluation harness, chapter 3. In the book since Edition 2026.11.10. Permanent link to evaluation harness

exact search

Exact search compares the query with every product and keeps the best k, so it never misses one.

Defined in Find the exact nearest neighbours with NumPy, chapter 6. In the book since Edition 2026.11.10. Permanent link to exact search

expansion terms

The nonzero terms of a SPLADE vector include expansion terms: words the text doesn't contain but the model weights anyway, so a product can match a query word it never uses.

Defined in Learn sparse term weights with SPLADE, chapter 7. In the book since Edition 2026.11.10. Permanent link to expansion terms

F

Faithfulness also faithful

Faithfulness is the share of an answer's claims that the context supports.

Defined in Check what the cited product supports, chapter 10. In the book since Edition 2026.11.10. Permanent link to Faithfulness

feature also features, ranking feature

A learned ranker scores each pair of a query and a product from a list of numbers, its features, and learns from judgments how to weigh them.

Defined in Describe each candidate with features, chapter 9. In the book since Edition 2026.11.10. Permanent link to feature

feature importance also gain importance

Feature importance shows what a model learned: LightGBM's "gain" importance is the "total gains of splits which use the feature"

True for lightgbm 4.7.0, as this edition pins it.

  • Implemented by: lightgbm 4.7.0

Defined in Report LambdaMART on the test queries, chapter 9. In the book since Edition 2026.11.10. Permanent link to feature importance

field

A product's text has parts, called fields

Defined in Weight the product name above the description, chapter 5. In the book since Edition 2026.11.10. Permanent link to field

filter

A filter keeps only products in the predicted class.

Defined in Boost the predicted class instead of filtering, chapter 8. In the book since Edition 2026.11.10. Permanent link to filter

fusion

Combining rankings is called fusion.

Defined in Find the queries each retriever wins, chapter 7. In the book since Edition 2026.11.10. Permanent link to fusion

G

gain also gains

A gain is what a result at a given position adds to the score before the position discount.

Defined in Grade relevance instead of yes or no, chapter 2. In the book since Edition 2026.11.10. Permanent link to gain

grade

a grade is the number that stands for a judgment's label.

Defined in Load the store's data, chapter 1. In the book since Edition 2026.11.10. Permanent link to grade

graded label also graded judgments

A graded label, such as WANDS's exact, partial or irrelevant, says how relevant.

Defined in Grade relevance instead of yes or no, chapter 2. In the book since Edition 2026.11.10. Permanent link to graded label

gradient-boosted trees

gradient-boosted trees, which add small decision trees one at a time, each fitted to the gradients of the trees before it

Defined in Learn a ranking from pairs of products, chapter 9. In the book since Edition 2026.11.10. Permanent link to gradient-boosted trees

Grounding also grounded

Grounding means that an answer rests on the context it was given, as the exact share assumes, not on what the model remembers or invents.

Defined in Check every citation against the context, chapter 10. In the book since Edition 2026.11.10. Permanent link to Grounding

H

HNSW also hierarchical navigable small world

HNSW, short for hierarchical navigable small world, is a graph index.

Defined in Trade recall for speed with HNSW, chapter 6. In the book since Edition 2026.11.10. Permanent link to HNSW

hybrid retrieval

Hybrid retrieval combines the rankings of a lexical and a semantic retriever into one ranking.

Defined in Find the queries each retriever wins, chapter 7. In the book since Edition 2026.11.10. Permanent link to hybrid retrieval

HyDE also hypothetical document embeddings

Where a rewrite asks the LLM for a better query, HyDE, short for hypothetical document embeddings, asks it for the document it imagines would answer the query.

hyde(query) = (embedding(hypothetical product) + embedding(query)) / 2, then scaled to length 1

Defined in Search with a hypothetical product (HyDE), chapter 8. In the book since Edition 2026.11.10. Permanent link to HyDE

hypothetical document

hypothetical document, written by an LLM to answer the query

Defined in Search with a hypothetical product (HyDE), chapter 8. In the book since Edition 2026.11.10. Permanent link to hypothetical document

I

information need

The judge grades the product against the information need behind the query's words, the thing the shopper wanted, not against the words.

Defined in Judge relevance against the shopper's need, chapter 2. In the book since Edition 2026.11.10. Permanent link to information need

intent

A shopper's intent is what they want to do.

Defined in Read intent as a product and its modifiers, chapter 2. In the book since Edition 2026.11.10. Permanent link to intent

Interleaving

Interleaving shows each shopper one list that mixes two rankers' results.

Defined in Interleave two rankings with team draft, chapter 11. In the book since Edition 2026.11.10. Permanent link to Interleaving

inverse document frequency also IDF

the inverse document frequency, IDF: the logarithm of the number of products divided by the term's document frequency, high for a rare term and low for a common one

idf(t) = ln(N / df(t))

Defined in Weight terms with TF-IDF, chapter 5. In the book since Edition 2026.11.10. Permanent link to inverse document frequency

inverted index

An inverted index turns the texts around: for each term, it keeps the list of products that contain the term

Defined in Build an inverted index, chapter 5. In the book since Edition 2026.11.10. Permanent link to inverted index

J

judge

This chapter calls whoever gives a judgment, person or LLM, a judge

Defined in Write judging guidelines, chapter 4. In the book since Edition 2026.11.10. Permanent link to judge

Judged@k also judged@10

Judged@k is the share of the top k that has a judgment of any grade

judged@k = (number of the top k products that have a judgment for the query) / k

Defined in Count unjudged products before you compare scores, chapter 2. In the book since Edition 2026.11.10. Permanent link to Judged@k

Judging guidelines

Judging guidelines say what each grade means, how to choose between grades, and what to do with the cases that fall between them.

Defined in Write judging guidelines, chapter 4. In the book since Edition 2026.11.10. Permanent link to Judging guidelines

judgment

A judgment is a person's label for one pair of a query and a product

Defined in Load the store's data, chapter 1. In the book since Edition 2026.11.10. Permanent link to judgment

K

Kendall's tau also tau

Tau is concordant pairs minus discordant pairs, divided by all pairs, from 1 for the same order to −1 for the reverse.

Defined in Check whether LLM grades order the rankers the same way, chapter 4. In the book since Edition 2026.11.10. Permanent link to Kendall's tau

L

LambdaMART

LambdaMART uses LambdaRank's λ as the gradients for gradient-boosted trees

Defined in Learn a ranking from pairs of products, chapter 9. In the book since Edition 2026.11.10. Permanent link to LambdaMART

large language model also LLM

An LLM is a model trained on large amounts of text to generate text in response to a prompt.

Defined in Rerank the top 50 with a cross-encoder, chapter 1. In the book since Edition 2026.11.10. Permanent link to large language model

late interaction also multi-vector retrieval

Late interaction keeps one vector per token instead of one per text, so it is also called multi-vector retrieval.

Defined in Match token by token with ColBERT, chapter 7. In the book since Edition 2026.11.10. Permanent link to late interaction

learned sparse retrieval

Learned sparse retrieval keeps BM25's machinery, an inverted index of terms, but has a neural model choose each product's terms and their weights.

Defined in Learn sparse term weights with SPLADE, chapter 7. In the book since Edition 2026.11.10. Permanent link to learned sparse retrieval

learning to rank

Learning to rank trains a model on judged queries to order candidates by their features.

Defined in Learn a ranking from pairs of products, chapter 9. In the book since Edition 2026.11.10. Permanent link to learning to rank

length normalization

With length normalization, a long text, which has more chances to contain any term, gets its term frequencies discounted.

Defined in Rebuild BM25 by hand to match bm25s, chapter 5. In the book since Edition 2026.11.10. Permanent link to length normalization

Lexical retrieval

Lexical retrieval finds and ranks products by the words they share with the query.

Defined in Lexical retrieval, chapter 5. In the book since Edition 2026.11.10. Permanent link to Lexical retrieval

lexicon

The lexicon, the list of words that count as correct, decides what gets corrected.

Defined in Correct spelling against the catalog's words, chapter 8. In the book since Edition 2026.11.10. Permanent link to lexicon

linear weights

With linear weights, a disagreement one grade apart weighs 0.5 and one two grades apart weighs 1

Defined in Give near misses partial credit with weighted kappa, chapter 4. In the book since Edition 2026.11.10. Permanent link to linear weights

listwise LLM reranking

Listwise, an LLM orders a whole list in one answer

Defined in Score each product with an LLM, chapter 9. In the book since Edition 2026.11.10. Permanent link to listwise LLM reranking

listwise model

A listwise model scores a whole ranking.

Defined in Learn a ranking from pairs of products, chapter 9. In the book since Edition 2026.11.10. Permanent link to listwise model

M

Matryoshka training also Matryoshka

Matryoshka training teaches a model to put the most useful information in its first dimensions, so that the start of a vector works as a smaller embedding on its own.

Defined in Truncate dimensions, then test the loss, chapter 6. In the book since Edition 2026.11.10. Permanent link to Matryoshka training

MaxSim

For each query token ColBERT finds the most similar product token, and it sums those maxima, an operation called MaxSim

score(query, product) = sum over query tokens q of max over product tokens d of cosine(q, d)

Defined in Match token by token with ColBERT, chapter 7. In the book since Edition 2026.11.10. Permanent link to MaxSim

Mean reciprocal rank also MRR

the mean of RR over queries is the mean reciprocal rank (MRR)

Defined in Score the first relevant result with reciprocal rank, chapter 3. In the book since Edition 2026.11.10. Permanent link to Mean reciprocal rank

modifier also modifiers

the query's modifiers are the attributes the shopper asked for.

Defined in Read intent as a product and its modifiers, chapter 2. In the book since Edition 2026.11.10. Permanent link to modifier

N

naive Bayes

In words, naive Bayes finds a class likely for a query when the class is common among products, and when the query's words are common in that class's product text.

predicted class = argmax over classes c of log P(c) + sum over the query's words w of log P(w | c)

  • Implemented by: scikit-learn 1.9.1

Defined in Train a query classifier on product text, chapter 8. In the book since Edition 2026.11.10. Permanent link to naive Bayes

nearest-neighbour search

Retrieval by embeddings is a nearest-neighbour search: find the k products whose vectors are most similar to the query's.

Defined in Find the exact nearest neighbours with NumPy, chapter 6. In the book since Edition 2026.11.10. Permanent link to nearest-neighbour search

noisy channel model also noisy channel

The classic corrector is the noisy channel model: for a typed word t, choose the correction c that maximizes P(c) × P(t | c)

correction(t) = argmax over candidates c of P(c) × P(t | c)

Defined in Correct spelling against the catalog's words, chapter 8. In the book since Edition 2026.11.10. Permanent link to noisy channel model

normalization also unit normalization

Normalization maps the different ways of writing one thing to a single form, on the products and on the query alike.

Defined in Normalize units the way the catalog writes them, chapter 8. In the book since Edition 2026.11.10. Permanent link to normalization

Normalized DCG also nDCG, nDCG@k, nDCG@10

Normalized DCG (nDCG) divides a ranking's DCG by the DCG of the ideal ranking.

nDCG@k = DCG@k / DCG@k of the ideal ranking

Defined in Build nDCG from discounted gains, chapter 3. In the book since Edition 2026.11.10. Permanent link to Normalized DCG

normalizing

Scaling every vector to length 1, called normalizing

Defined in Compare embeddings with cosine similarity, chapter 6. In the book since Edition 2026.11.10. Permanent link to normalizing

null hypothesis

the null hypothesis, the assumption that the rankers are equally good

Defined in Test a gap with a permutation test, chapter 3. In the book since Edition 2026.11.10. Permanent link to null hypothesis

O

observed agreement

The share of pairs on which two judges give the same grade, the observed agreement

Defined in Measure agreement beyond chance with kappa, chapter 4. In the book since Edition 2026.11.10. Permanent link to observed agreement

P

p-value

p-value: the probability, under the null hypothesis, of a gap at least as large as the one you measured

Defined in Test a gap with a permutation test, chapter 3. In the book since Edition 2026.11.10. Permanent link to p-value

paired test also paired

The tests in this book are paired: both rankers are scored on the same queries, so each test works on the per-query differences, and a hard query's low scores cancel out.

Defined in Test a gap with a permutation test, chapter 3. In the book since Edition 2026.11.10. Permanent link to paired test

pairwise LLM reranking

Pairwise, an LLM says which of two products fits the query better.

Defined in Score each product with an LLM, chapter 9. In the book since Edition 2026.11.10. Permanent link to pairwise LLM reranking

pairwise model

A pairwise model learns which of two products should rank higher.

Defined in Learn a ranking from pairs of products, chapter 9. In the book since Edition 2026.11.10. Permanent link to pairwise model

peeking

Stopping a test the moment it looks significant, called peeking

Defined in Run an A/B test, chapter 11. In the book since Edition 2026.11.10. Permanent link to peeking

per-query differences

per-query differences: one number per query, the new ranker's score minus the old one's

Defined in Read per-query differences, not only the mean, chapter 3. In the book since Edition 2026.11.10. Permanent link to per-query differences

per-query min-max scaling also min-max, per-query min-max

Per-query min-max scaling maps the lowest score among the query's ranked products to 0 and the highest to 1.

min-max(s) = (s − lowest score among the query's candidates) / (highest − lowest)

Defined in Normalize scores before you add them, chapter 7. In the book since Edition 2026.11.10. Permanent link to per-query min-max scaling

percentile interval

The percentile interval is the middle 95% of the resampled means.

Defined in Report how big the gap is, chapter 3. In the book since Edition 2026.11.10. Permanent link to percentile interval

permutation test

A permutation test builds the gaps you'd see under the null hypothesis directly.

Defined in Test a gap with a permutation test, chapter 3. In the book since Edition 2026.11.10. Permanent link to permutation test

phrase match

A phrase match needs the terms next to each other and in order

Defined in Match phrases with term positions, chapter 5. In the book since Edition 2026.11.10. Permanent link to phrase match

pointwise LLM reranking

Pointwise, an LLM judges each product alone and returns a label or a score.

Defined in Score each product with an LLM, chapter 9. In the book since Edition 2026.11.10. Permanent link to pointwise LLM reranking

pointwise model

A pointwise model predicts each product's grade on its own, as a regression.

Defined in Learn a ranking from pairs of products, chapter 9. In the book since Edition 2026.11.10. Permanent link to pointwise model

pool

Nobody judges every product for every query, so test collections judge a pool instead, a set of candidates gathered for each query.

Defined in Know where the judgments came from, chapter 2. In the book since Edition 2026.11.10. Permanent link to pool

position bias

position bias: people click higher-ranked results more, partly because they trust the ranking.

Defined in Treat clicks as biased evidence, chapter 2. In the book since Edition 2026.11.10. Permanent link to position bias

positional bias

A listwise answer can suffer from positional bias: an LLM's tendency to let a product's place in the prompt affect its rank, not the click position bias of shoppers.

Defined in Reverse the input to measure positional bias, chapter 9. In the book since Edition 2026.11.10. Permanent link to positional bias

positional index

A positional index stores, for each term and product, the positions where the term occurs, so that a phrase becomes a check on positions

Defined in Match phrases with term positions, chapter 5. In the book since Edition 2026.11.10. Permanent link to positional index

postings

the list of products that contain the term, called the term's postings

Defined in Build an inverted index, chapter 5. In the book since Edition 2026.11.10. Permanent link to postings

power also statistical power

the test's power: the chance that it detects a real change of a given size

Defined in Size the test with power, chapter 11. In the book since Edition 2026.11.10. Permanent link to power

Precision at k also precision@k, precision@10

Precision at k, written precision@k, is the share of the top k results that are relevant.

Defined in Count the relevant products in the top k, chapter 3. In the book since Edition 2026.11.10. Permanent link to Precision at k

Prediction-powered inference

Prediction-powered inference rigorously corrects a model's predictions with a human sample, with confidence intervals that hold "without making any assumptions on the machine-learning algorithm that supplies the predictions"

corrected = mean over all queries of the LLM score + mean over the human sample of (human score − LLM score)

Defined in Correct LLM scores with a human sample, chapter 4. In the book since Edition 2026.11.10. Permanent link to Prediction-powered inference

presentation bias

Presentation bias goes beyond position: clicks depend on how a result is shown, not only on where.

Defined in Treat clicks as biased evidence, chapter 2. In the book since Edition 2026.11.10. Permanent link to presentation bias

probe query

a probe query: one written for this chapter to test a behaviour, not one of the store's queries

Defined in Split product text into terms, chapter 5. In the book since Edition 2026.11.10. Permanent link to probe query

prompt

The retrieved text goes into the prompt, the input the model reads

Defined in Put retrieved products in the prompt, chapter 10. In the book since Edition 2026.11.10. Permanent link to prompt

pseudo-relevance feedback

Pseudo-relevance feedback assumes the top results are relevant and adds their terms to the query

Defined in Expand queries with a synonym list, chapter 8. In the book since Edition 2026.11.10. Permanent link to pseudo-relevance feedback

Q

query classification

Query classification assigns a query to a class in the store's taxonomy

Defined in Train a query classifier on product text, chapter 8. In the book since Edition 2026.11.10. Permanent link to query classification

query drift

A word that expansion adds can match products the shopper didn't ask for, and move the query away from its meaning, which is called query drift.

Defined in Expand queries with a synonym list, chapter 8. In the book since Edition 2026.11.10. Permanent link to query drift

query expansion

Query expansion adds terms to a query so that it can match products that use other words for the same thing.

Defined in Expand queries with a synonym list, chapter 8. In the book since Edition 2026.11.10. Permanent link to query expansion

query rewriting

Query rewriting replaces the query with a text written to retrieve better.

Defined in Rewrite queries with an LLM, chapter 8. In the book since Edition 2026.11.10. Permanent link to query rewriting

query understanding

Query understanding is everything a search system does to a query before retrieval

Defined in Understanding queries, chapter 8. In the book since Edition 2026.11.10. Permanent link to query understanding

query-dependent feature

A query-dependent feature describes the query and the product together, such as a BM25 score.

Defined in Describe each candidate with features, chapter 9. In the book since Edition 2026.11.10. Permanent link to query-dependent feature

query-independent feature

A query-independent feature describes the product alone, such as how many shoppers rated it.

Defined in Describe each candidate with features, chapter 9. In the book since Edition 2026.11.10. Permanent link to query-independent feature

R

ranking

Ranking puts the retrieved products in order, best first.

Defined in Rank products with BM25, chapter 1. In the book since Edition 2026.11.10. Permanent link to ranking

real-word error

real-word error, a typo in which every word is spelled right, as in "flew form Heathrow"

Defined in Correct spelling against the catalog's words, chapter 8. In the book since Edition 2026.11.10. Permanent link to real-word error

Recall at k also recall@k, recall@10

Recall at k, recall@k, is the share of all relevant products that appear in the top k.

Defined in Count the relevant products in the top k, chapter 3. In the book since Edition 2026.11.10. Permanent link to Recall at k

recall@k against exact search also recall against exact search

An ANN index's quality is its recall@k against exact search: the share of exact search's top k that it also returns.

Defined in Trade recall for speed with HNSW, chapter 6. In the book since Edition 2026.11.10. Permanent link to recall@k against exact search

Reciprocal rank also RR

Reciprocal rank (RR) scores a query as 1 divided by the position of the first relevant result, or 0 if there is none

Defined in Score the first relevant result with reciprocal rank, chapter 3. In the book since Edition 2026.11.10. Permanent link to Reciprocal rank

reciprocal rank fusion also RRF

Reciprocal rank fusion (RRF) scores each product by adding, over the rankings that contain it, 1 divided by the sum of a constant k and its rank

Defined in Test reciprocal rank fusion's settings on train queries, chapter 7. In the book since Edition 2026.11.10. Permanent link to reciprocal rank fusion

regression set

regression set: fixed queries, with judgments, that every change runs against before it ships

Defined in Diff two runs query by query, chapter 11. In the book since Edition 2026.11.10. Permanent link to regression set

relevance feedback

Relevance feedback improves a query from judgments of its first results

Defined in Search with a hypothetical product (HyDE), chapter 8. In the book since Edition 2026.11.10. Permanent link to relevance feedback

relevant

In this book, a relevant product is one judged exact for the query, unless a sentence says otherwise.

Defined in Load the store's data, chapter 1. In the book since Edition 2026.11.10. Permanent link to relevant

rerank depth

the rerank depth, how many products the reranker reads per query, is a budget decision

Defined in Set the rerank depth from a budget, chapter 9. In the book since Edition 2026.11.10. Permanent link to rerank depth

reranking

Reranking reorders a short list from retrieval with a slower model, one that can afford to look at the query and each product together because it only sees a few dozen products.

Defined in Rerank the top 50 with a cross-encoder, chapter 1. In the book since Edition 2026.11.10. Permanent link to reranking

retrievable unit also retrieval unit

A retrievable unit is the piece of text that retrieval scores and returns, and that a context holds: a whole document, a passage or a sentence.

Defined in Choose the retrievable unit, chapter 10. In the book since Edition 2026.11.10. Permanent link to retrievable unit

retrieval

Retrieval finds the products that might match a query, out of the whole catalog.

Defined in Rank products with BM25, chapter 1. In the book since Edition 2026.11.10. Permanent link to retrieval

Retrieval-augmented generation also RAG

Retrieval-augmented generation (RAG), the design behind a shopping assistant, pairs an LLM with a retriever.

Defined in Put retrieved products in the prompt, chapter 10. In the book since Edition 2026.11.10. Permanent link to Retrieval-augmented generation

risk-sensitive evaluation

Risk-sensitive evaluation counts losses more heavily than wins.

Defined in Weigh losses more than wins with URisk, chapter 11. In the book since Edition 2026.11.10. Permanent link to risk-sensitive evaluation

run

The rankings for every query in a set form a run, written as {query_id: [product_id, ...]}, best first.

Defined in Score every test query, chapter 1. In the book since Edition 2026.11.10. Permanent link to run

S

sample ratio mismatch also SRM

A sample ratio mismatch (SRM) means users went missing from one group or were added to it unevenly

Defined in Run an A/B test, chapter 11. In the book since Edition 2026.11.10. Permanent link to sample ratio mismatch

saturation

With saturation, each repeat of a term adds less than the one before, and no number of repeats can push the term's contribution past a ceiling

Defined in Rebuild BM25 by hand to match bm25s, chapter 5. In the book since Edition 2026.11.10. Permanent link to saturation

score normalization

Score normalization rescales each retriever's scores for each query before adding them.

Defined in Normalize scores before you add them, chapter 7. In the book since Edition 2026.11.10. Permanent link to score normalization

semantic retrieval

Semantic retrieval compares the meanings of the query and the product instead of their words.

Defined in Search by meaning with embeddings, chapter 1. In the book since Edition 2026.11.10. Permanent link to semantic retrieval

sensitivity check

a sensitivity check, which rescores both runs under two scenarios, one favouring each side

Defined in Judge the products a new ranker surfaces, chapter 11. In the book since Edition 2026.11.10. Permanent link to sensitivity check

significance test

A significance test asks how surprising a gap between two rankers would be under the null hypothesis

Defined in Test a gap with a permutation test, chapter 3. In the book since Edition 2026.11.10. Permanent link to significance test

sliding window

A prompt can hold only so many products, so a long list is reranked in a sliding window: RankGPT reranks BM25's top 100 with "a window size of 20 and step size of 10", from the bottom of the list up

Defined in Order the 20 products with one LLM prompt, chapter 9. In the book since Edition 2026.11.10. Permanent link to sliding window

Spearman's rank correlation also Spearman

Spearman's rank correlation is the correlation between two measures' ranks rather than their values, 1 when they order the cases the same way and near 0 when the orders are unrelated

Defined in Measure the context, then the answer, chapter 10. In the book since Edition 2026.11.10. Permanent link to Spearman's rank correlation

SPLADE

SPLADE is one learned sparse retrieval model

weight of term j = sum over the text's tokens i of log(1 + ReLU(w_ij))

Defined in Learn sparse term weights with SPLADE, chapter 7. In the book since Edition 2026.11.10. Permanent link to SPLADE

standalone query

standalone query, one that reads without the earlier turns and keeps what the shopper still wants

Defined in Turn a follow-up into a standalone query, chapter 8. In the book since Edition 2026.11.10. Permanent link to standalone query

standard error

standard error: the standard deviation of the differences divided by the square root of the number of queries, which estimates how far the mean would move from one sample of queries to another

Defined in Get the p-value from a paired t-test, chapter 3. In the book since Edition 2026.11.10. Permanent link to standard error

stem

the forms of a word share one term, its stem

Defined in Stem terms so word forms match, chapter 5. In the book since Edition 2026.11.10. Permanent link to stem

Stemming

Stemming cuts common endings off words, so that the forms of a word share one term

  • Comes before: stem
  • Implemented by: pystemmer 3.1.0

Defined in Stem terms so word forms match, chapter 5. In the book since Edition 2026.11.10. Permanent link to Stemming

stop word

stop words, very common words such as the and with

Defined in Split product text into terms, chapter 5. In the book since Edition 2026.11.10. Permanent link to stop word

student

a small model, the student, to imitate a large model

Defined in Distill the cross-encoder into a small ranker, chapter 9. In the book since Edition 2026.11.10. Permanent link to student

T

t distribution

t distribution: bell-shaped, with heavier tails than the normal, and a shape set by its degrees of freedom, here one fewer than the number of queries

Defined in Get the p-value from a paired t-test, chapter 3. In the book since Edition 2026.11.10. Permanent link to t distribution

target entity

A query names a target entity, the kind of product

Defined in Read intent as a product and its modifiers, chapter 2. In the book since Edition 2026.11.10. Permanent link to target entity

teacher

to imitate a large model, the teacher

Defined in Distill the cross-encoder into a small ranker, chapter 9. In the book since Edition 2026.11.10. Permanent link to teacher

Team draft also team-draft interleaving

Team draft interleaves two rankers' lists like picking teams for a game

Defined in Interleave two rankings with team draft, chapter 11. In the book since Edition 2026.11.10. Permanent link to Team draft

term

Removing stop words, very common words such as the and with, leaves the terms that search compares

Defined in Split product text into terms, chapter 5. In the book since Edition 2026.11.10. Permanent link to term

term frequency also tf

With each product, the postings store the term frequency, the number of times the term occurs in that product's text.

Defined in Build an inverted index, chapter 5. In the book since Edition 2026.11.10. Permanent link to term frequency

TF-IDF

TF-IDF multiplies tf and IDF for each query term and adds them up

tf-idf(q, d) = sum over query terms t of tf(t, d) × idf(t)

Defined in Weight terms with TF-IDF, chapter 5. In the book since Edition 2026.11.10. Permanent link to TF-IDF

theoretical min-max

Theoretical min-max uses each score's lowest possible value instead of the lowest one seen: 0 for BM25 and −1 for cosine.

Defined in Normalize scores before you add them, chapter 7. In the book since Edition 2026.11.10. Permanent link to theoretical min-max

token

Splitting the text into pieces gives tokens.

Defined in Split product text into terms, chapter 5. In the book since Edition 2026.11.10. Permanent link to token

token

The store's encoder reads text as tokens, word pieces from its own vocabulary, not the terms of chapter 5's tokenizer.

  • Contrasts with: term

Defined in Turn text into embeddings, chapter 6. In the book since Edition 2026.11.10. Permanent link to token

tool also tools

A tool is a function the program offers the model, described by a name, a description and its parameters.

Defined in Read an agent's searches from its trace, chapter 10. In the book since Edition 2026.11.10. Permanent link to tool

tool call also tool calls

A tool call is the model's request to run a tool with arguments.

Defined in Read an agent's searches from its trace, chapter 10. In the book since Edition 2026.11.10. Permanent link to tool call

trace also traces

The trace is the record of every tool call and its result.

Defined in Read an agent's searches from its trace, chapter 10. In the book since Edition 2026.11.10. Permanent link to trace

treatment

the changed system is the treatment

Defined in Run an A/B test, chapter 11. In the book since Edition 2026.11.10. Permanent link to treatment

U

unjudged

A product without a judgment for a query is unjudged for it

Defined in Know where the judgments came from, chapter 2. In the book since Edition 2026.11.10. Permanent link to unjudged

URisk

The risk-sensitive measure of the TREC 2013 Web track is URisk.

URisk = (1 / N) × [ sum of Δ(q) over improved queries − (α + 1) × sum of |Δ(q)| over hurt queries ]

Defined in Weigh losses more than wins with URisk, chapter 11. In the book since Edition 2026.11.10. Permanent link to URisk

V

vocabulary

all the terms in the catalog form its vocabulary

Defined in Split product text into terms, chapter 5. In the book since Edition 2026.11.10. Permanent link to vocabulary

W

Weighted kappa

Weighted kappa gives each disagreement a weight by its distance and compares the weighted disagreement with the weighted disagreement expected by chance

weighted kappa = 1 - (sum of weight × observed count) / (sum of weight × count expected by chance)

Defined in Give near misses partial credit with weighted kappa, chapter 4. In the book since Edition 2026.11.10. Permanent link to Weighted kappa

weighted RRF also weighted reciprocal rank fusion

In weighted RRF, a product scores (1 − α) / (k + its BM25 rank) plus α / (k + its semantic rank).

Defined in Tune the blend's weight on train queries, chapter 7. In the book since Edition 2026.11.10. Permanent link to weighted RRF

Z

z-score

The z-score subtracts the mean of a query's scores and divides by their standard deviation.

Defined in Normalize scores before you add them, chapter 7. In the book since Edition 2026.11.10. Permanent link to z-score