A Primer on Search Relevance
Written with AI. Verified by people. How we make books
Anyone can check this book or correct it. Report an error · How to fix one
How search engines find and rank results, and how to measure whether a change helps: lexical, semantic and hybrid retrieval, reranking with learned models and LLMs, and judgments from people and LLMs, on a real product catalog.
Who it's for For readers comfortable with Python, pandas and NumPy who are new to search: a data scientist or engineer asked to improve a search box, or to build retrieval for an LLM application.
Edition history1 edition since October 2026Full changelog
2026.10.1
2026.10.1 · Oct 18, 2026First edition132 examples
Contents
-
1Search, end to end
- Load the store's data
- Rank products with BM25
- Check the results against the judgments
- Score one ranking with nDCG@10
- Score every test query
- Search by meaning with embeddings
- Fuse two rankings with reciprocal rank fusion
- Rerank the top 50 with a cross-encoder
- What goes wrong in production
- Decisions this chapter made
-
2What relevance means
- Judge relevance against the shopper's need
- Expect judges to disagree on some pairs
- Read intent as a product and its modifiers
- Grade relevance instead of yes or no
- Choose the gains on purpose
- Read the label mix of each query
- Treat clicks as biased evidence
- Know where the judgments came from
- Count unjudged products before you compare scores
- What goes wrong in production
- Decisions this chapter made
-
3Measuring search
- Count the relevant products in the top k
- Score the first relevant result with reciprocal rank
- Build nDCG from discounted gains
- Match scikit-learn's
ndcg_score - Choose a metric before you compare
- Read per-query differences, not only the mean
- Test a gap with a permutation test
- Get the p-value from a paired t-test
- Report how big the gap is
- Decide in advance what counts as established
- Compare runs in an evaluation harness
- What goes wrong in production
- Decisions this chapter made
-
4Judgments from people and LLMs
- Write judging guidelines
- Measure agreement beyond chance with kappa
- Give near misses partial credit with weighted kappa
- Read kappa against the label mix
- Read the direction of the LLM's errors
- Compare what the LLM saw with what the judges saw
- Measure how much the prompt changes the labels
- Fill unjudged pairs with LLM grades
- Check whether LLM grades order the rankers the same way
- Rerun the planned comparisons with LLM grades
- Correct LLM scores with a human sample
- What goes wrong in production
- Decisions this chapter made
-
5Lexical retrieval
- Split product text into terms
- Build an inverted index
- Weight terms with TF-IDF
- Rebuild BM25 by hand to match bm25s
- Stem terms so word forms match
- Check stemming on the test queries
- Tune BM25's parameters on train queries
- Weight the product name above the description
- Index product attributes as a field
- Match phrases with term positions
- What goes wrong in production
- Decisions this chapter made
-
6Semantic retrieval
- Turn text into embeddings
- Compare embeddings with cosine similarity
- Find the exact nearest neighbours with NumPy
- Trade recall for speed with HNSW
- Check the index on the test queries
- Check queries that contain digits
- Expect a code to carry little weight
- Test embeddings on negation
- Choose an embedding model
- Read benchmarks as a shortlist
- Truncate dimensions, then test the loss
- What goes wrong in production
- Decisions this chapter made
-
7Hybrid retrieval
- Find the queries each retriever wins
- Check each retriever before you fuse
- Normalize scores before you add them
- Test reciprocal rank fusion's settings on train queries
- Tune the blend's weight on train queries
- Check that the tuned weight is stable
- Report the tuned blends on test queries
- Learn sparse term weights with SPLADE
- Match token by token with ColBERT
- Decide when hybrid retrieval pays
- What goes wrong in production
- Decisions this chapter made
-
8Understanding queries
- Correct spelling against the catalog's words
- Normalize units the way the catalog writes them
- Expand queries with a synonym list
- Train a query classifier on product text
- Boost the predicted class instead of filtering
- Rewrite queries with an LLM
- Search with a hypothetical product (HyDE)
- Turn a follow-up into a standalone query
- What goes wrong in production
- Decisions this chapter made
-
9Ranking
- Describe each candidate with features
- Learn a ranking from pairs of products
- Train LambdaMART with LightGBM
- Report LambdaMART on the test queries
- Rerank semantic retrieval with a cross-encoder
- Score each product with an LLM
- Order the 20 products with one LLM prompt
- Reverse the input to measure positional bias
- Set the rerank depth from a budget
- Distill the cross-encoder into a small ranker
- What goes wrong in production
- Decisions this chapter made
-
10Retrieval for LLM applications
- Put retrieved products in the prompt
- Score each answer by the products it cites
- Check every citation against the context
- Check what the cited product supports
- Measure the context, then the answer
- Choose the retrievable unit
- Read an agent's searches from its trace
- Compare the retrievers behind the agent's tool
- What goes wrong in production
- Decisions this chapter made
-
11Testing changes
- Diff two runs query by query
- Weigh losses more than wins with URisk
- Judge the products a new ranker surfaces
- Keep the test queries out of every choice
- Run an A/B test
- Size the test with power
- Interleave two rankings with team draft
- Rehearse interleaving with simulated shoppers
- Find the cause when offline and online disagree
- Decide whether to ship
- What goes wrong in production
- Decisions this chapter made