검색, 처음부터 끝까지
읽는 데 약 57분분 걸린다, 10절에 11개 예제이(가) 있음.
AI로 번역함. 영어 원문을 사람이 검증함. 우리가 책을 만드는 방식
영어판 2026.11.10판을 번역한 것이다. 원문 읽기
이 장을 마치면 상품 카탈로그를 대상으로, 쇼핑객의 질의에서 정렬된 상품 목록에 이르는 거칠지만 완결된 검색 파이프라인을 만들 수 있다. 또한 파이프라인에 가한 변경이 도움이 되는지를 숫자 하나로 측정할 수 있다. 이후의 각 장은 이 파이프라인의 한 부분을 맡아 그것을 견고하게 만든다.
온라인 가구·생활용품 매장의 쇼핑객은 검색창에 clear shower curtain나 48 inch kitchen hood 같은 질의를 입력한다. 이 책은 처음부터 끝까지 이 매장을 따라가며, 이를 '매장'이라고 부른다. 매장에는 두 가지 답이 필요하다. 검색이 얼마나 좋은지, 그리고 변경이 검색을 더 낫게 만드는지다. 어떤 검색 시스템을 다루더라도 같은 두 질문을 마주하게 된다. 그래서 이 책은 모든 방법을 매장에 적용해 보며, 각 방법이 실제 질의에 어떤 영향을 주는지 직접 확인할 수 있게 한다.
매장에는 이 답을 구하는 데 필요한 것이 있다. 카탈로그, 실제 질의의 모음, 그리고 어떤 상품이 어떤 질의에 부합하는지에 대한 사람들의 판단이다. 따라서 이 장에서는 카탈로그를 세 가지 방식으로 검색하고(단어를 맞추는 방식, 의미를 맞추는 방식, 둘을 결합하는 방식), 더 느린 모델로 상위 결과를 다시 정렬하며, 각 버전의 점수를 그 판단에 비추어 매긴다. 모든 단계가 점수를 높이지는 않는다. 이것이 이 장의 핵심 교훈이다. 모든 변경을 측정하라.
이 부분은 아직 번역되지 않았으므로 영어로 표시된다.
Load the store's data
Before you can improve a search system, you need three things: the items it searches, the queries people type, and judgments that say which items match which query. Search engines call the items documents. In this book they are products. A judgment is a person's label for one pair of a query and a product, and a grade is the number that stands for a judgment's label. Without judgments, you can look at results and form opinions, but you can't tell a fix from a regression.
The store's data, which the book's examples search and score, is real. It is a sample of WANDS, a public product-search dataset that Wayfair published with a paper at ECIR 2022 (Chen et al., 2022), under the MIT licence (licence). The full dataset has 42,994 products, 480 queries and 233,448 judgments (repository). The queries are a stratified sample of real shopper queries from a quarter's search log in 2021, and each label is the majority vote of up to three trained annotators (Chen et al., sections 4.1 and 5). The script prep/wands_sample.py draws the book's sample, and the sample's data card, examples/data/wands/README.md, says how. Setup code that many examples need lives in the book's shared modules, short files in examples/lib/ that you can open and read. lib.store, one of those modules, loads the sample, and the next example prints what comes back:
1"""Loads the store's sample and prints its size, how its queries split, and how its judgments are labelled. The prose quotes the split from the `train queries` and `test queries` lines."""2import pandas as pd34from lib.store import DATA, load_labels, load_products, load_queries56products = load_products()7queries = load_queries()8labels = load_labels() # one row per judged (query, product) pair9conflicts = pd.read_csv(DATA / "label_conflicts.csv") # pairs WANDS labelled two ways1011print(f"products: {len(products):,}")12print("queries:", len(queries))13print("train queries:", (queries["split"] == "train").sum())14print("test queries:", (queries["split"] == "test").sum())15print(f"judged pairs: {len(labels):,}")16print("pairs with conflicting labels:", len(conflicts))17for label in ["exact", "partial", "irrelevant"]:18 print(f"{label} pairs: {(labels['label'] == label).sum():,}")19print(f"products judged for at least one query: {labels['product_id'].nunique():,}")2021print()22print(queries.head(4).to_string())
python load-the-store.pyproducts: 12,000
queries: 80
train queries: 50
test queries: 30
judged pairs: 13,077
pairs with conflicting labels: 7
exact pairs: 3,252
partial pairs: 7,972
irrelevant pairs: 1,853
products judged for at least one query: 11,917
query query_class split
query_id
6 acrylic clear chair Dining Chairs train
10 king poster bed Beds train
20 foutains with brick look Wallpaper train
36 laundry basket with wheels Hampers & Baskets train
The sample has 12,000 products and 80 queries. It splits the queries once, before any ranker runs: 50 train queries for tuning and 30 test queries for reporting. Every score in this book is reported on the test queries, and nothing is tuned on them. A score on queries you tuned on measures how well you fit those queries, not how well search works.
WANDS judged some pairs more than once. The sample keeps one row per pair, 13,077 in all. When a pair's labels differ, it keeps the lower grade and lists the pair in label_conflicts.csv, which holds 7 pairs. Code that scores a ranking reads the judgments through lib.store.qrels, which returns them as {query_id: {product_id: grade}}. Chapter 2 reads the judgments closely.
Each judgment is one of three labels, defined by the dataset's authors (Chen et al., section 3.2). Exact means the product fully matches the search query. Partial means it "only matches the target entity of the query, but does not satisfy the modifiers for the query." Irrelevant means the product is not relevant to the query. The book's grades are 2, 1 and 0. Of the judged pairs, 3,252 are exact, 7,972 partial and 1,853 irrelevant. In this book, a relevant product is one judged exact for the query, unless a sentence says otherwise.
Three warnings come with the store's data. First, each query's query_class is WANDS's predicted class for it, not a judgment (Chen et al., Table 1): foutains with brick look is classed as Wallpaper. Treat it as a hint. Second, nobody judged every product for every query. That would have taken about 60 million judgments, so WANDS judged candidates gathered from products shoppers clicked or added to their carts and from several search systems, and stopped when few new exact matches turned up (Chen et al., sections 4.2 and 4.3). A product without a judgment for a query is unjudged: unknown, not irrelevant. Third, nearly every product in the sample, 11,917 of 12,000, is judged for at least one of its queries, so the sample holds fewer distractors than a full catalog would: products that compete for a query's top places without matching it.
Working rule: Split your queries into train and test before you change anything. Read test results to understand a ranker, but choose and tune every change on train queries, and report on test. Treat unjudged products as unknown. If you have no judgments yet, collect some before you change the ranking.
BM25로 상품 순위 매기기
상점의 카탈로그를 검색하려면 두 가지 일이 필요하다. **검색(retrieval)**은 전체 카탈로그에서 질의와 맞을 가능성이 있는 상품을 찾아낸다. **순위 매기기(ranking)**는 검색된 상품을 가장 좋은 것부터 차례로 늘어놓는다. 어휘 기반 접근법은 질의의 단어와 각 상품 텍스트의 단어를 비교하고, 겹치는 정도에 점수를 매겨 두 일을 모두 수행한다.
BM25는 세 부분으로 이루어진 어휘 기반 점수 함수다. 첫째, 상품은 질의의 단어를 포함할수록 높은 점수를 받으며, 드문 단어일수록 더 높다. 적은 수의 상품에만 나타나는 단어가 그 상품들에 대해 더 많은 것을 말해 주기 때문이다. 둘째, 같은 단어가 반복될 때 더해지는 점수는 점점 줄어들어 상한에 이르는데, BM25 문헌에서는 이를 포화(saturation)라고 부른다. 셋째, 긴 텍스트는 짧은 텍스트에 비해 감점되므로 단어를 더 많이 담고 있다는 이유만으로 이길 수 없다. 감점의 크기는 매개변수 b가 정하며, 0(감점 없음)에서 1(완전 감점) 사이의 값을 가진다(Robertson and Zaragoza, 2009). 5장에서 이 공식을 직접 구성해 본다.
이 책의 BM25는 Python용 BM25 라이브러리인 bm25s 0.3.16에서 실행된다. 기본 설정에서는 Lucene 검색 라이브러리가 사용하는 BM25 변형으로 점수를 계산하며, k1=1.5(포화 설정)와 b=0.75을 쓴다(bm25s source). 이 점수는 교과서의 공식과 조금 다르지만, 세 개의 뉴스와이어 컬렉션에서 수행한 재현성 연구는 BM25 변형들 사이에 효과성의 유의미한 차이가 없다고 보고했다(Kamphuis et al., 2020). 점수를 계산하기 전에 bm25s는 텍스트를 토큰으로 나눈다. 토큰을 소문자로 바꾸고, 두 글자 이상인 단어만 남기며, and 같은 영어 불용어를 제거하고, 단어를 공통 어간으로 줄이지는 않는다(bm25s tokenizer). 이 책에서 사용하는 순위 모델인 lib.lexical.LexicalSearch는 각 상품의 이름, 클래스, 카테고리, 설명을 하나의 텍스트로 이어 붙여 색인한다.
상점의 테스트 질의 clear shower curtain는 BM25가 작동하는 모습을 보여 준다. 이 질의를 입력하는 쇼핑객은 투명한 커튼을 원하지만, 카탈로그에는 투명하지 않은 커튼이 많고 같은 단어를 공유하는 액세서리도 있다. 다음은 BM25의 상위 10개 결과다.
1"""Searches the store's catalog with BM25 for one test query and prints the top 10 products. The prose's point that BM25 matches words rather than things rests on the names at the bottom of the list."""2from lib.lexical import LexicalSearch3from lib.store import load_products, load_queries45products = load_products()6query = load_queries().at[246, "query"] # a test query7lexical = LexicalSearch(products) # BM25 over name, class, category and description89scores = lexical.search(query, k=10) # the 10 highest scores, best first10print("query:", query)11print()12# names cut to 55 characters so that each row fits on one line13for rank, (product_id, score) in enumerate(scores.items(), start=1):14 print(f"{rank:>2} {product_id:>5} {score:5.2f} {products.at[product_id, 'name'][:55]}")
python first-bm25-search.pyquery: clear shower curtain
1 9752 7.11 rodas vinyl shower curtain
2 20428 6.97 woolley vinyl single shower curtain
3 20429 6.75 woolley vinyl solid color single shower curtain
4 20425 6.65 siniard single shower curtain
5 20430 6.48 simmon vinyl single shower curtain liner
6 3564 6.10 71 '' x71 '' 4pcs shower curtain sets beach theme showe
7 20132 5.95 newburg check 3d vinyl single shower curtain + hooks
8 33346 5.76 gilded pineapple shower curtain hooks
9 3197 5.72 double shower curtain hooks
10 1050 5.68 stainless steel shower curtain hooks
열 개 중 대부분은 샤워 커튼이나 라이너이지만, 마지막 몇 개는 샤워 커튼 고리다. BM25는 고리의 이름에서 "shower curtain"이라는 단어를 찾아 커튼과 거의 비슷한 높은 점수를 주었다. BM25는 사물이 아니라 단어를 맞춰 보기 때문에 고리가 커튼이 아니라는 사실을 알지 못한다. 여기에 출력된 어떤 것도 커튼이 투명한지 알려 주지 않는다. 그것을 알려면 상점의 판정이 필요하다.
작업 규칙: 라이브러리의 기본값으로 실행한 BM25를 기준선으로 삼아 시작한다. BM25는 학습 데이터도 모델도 필요 없으며, 잘못되었을 때 어떤 단어가 원인인지 볼 수 있다. train queries에서 변화를 측정할 수 있을 때까지는 매개변수를 건드리지 않는다. 쇼핑객과 상품이 같은 것을 두고 서로 다른 단어를 쓴다면 의미 기반 검색을 추가한다.
판정과 결과 대조하기
순위는 쇼핑객이 무엇을 원했는지 알기 전에는 큰 의미가 없다. 판정은 그 정보를 질의와 상품의 쌍마다 하나씩 제공한다. 이 목적에는 판정이 클릭보다 나은 근거다. WANDS의 저자들은 "사용자 행동 로그 데이터는 위치 편향을 겪으며 운영 중인 시스템과 비슷한 순위를 선호하게 된다"(Chen et al., section 3.1)라고 지적한다. 쇼핑객은 가장 먼저 보이는 것을 클릭하므로, 클릭은 대체로 이미 가지고 있는 순위를 확인해 줄 뿐이다.
이제 clear shower curtain에 대한 BM25의 상위 10개 상품을 각각의 판정과 나란히 놓아 보자. 이 질의에 대해 아무도 판정하지 않은 상품은 unjudged로 표시된다.
1"""Puts BM25's top 10 for `clear shower curtain` beside each product's judgment, and counts the labels. The prose's reading of the top 10 rests on the labelled rows and the `in the top 10` lines after them."""2from lib.lexical import LexicalSearch3from lib.store import load_products, load_queries, qrels45LABELS = {2: "exact", 1: "partial", 0: "irrelevant"}67products = load_products()8query_id = 2469query = load_queries().at[query_id, "query"]10judgments = qrels()[query_id] # {product_id: grade} for this query1112top = LexicalSearch(products).search(query, k=10).index13# a product with no judgment for this query is unjudged, not irrelevant14results = [LABELS.get(judgments.get(product_id), "unjudged") for product_id in top]15print("query:", query)16print()17for rank, (product_id, label) in enumerate(zip(top, results), start=1):18 print(f"{rank:>2} {label:<10} {products.at[product_id, 'name'][:55]}")1920print()21for label in ["exact", "partial", "irrelevant", "unjudged"]:22 print(f"{label} in the top 10: {results.count(label)}")23print("products judged for this query:", len(judgments))24print("exact products judged for this query:", sum(grade == 2 for grade in judgments.values()))
python judge-first-results.pyquery: clear shower curtain
1 partial rodas vinyl shower curtain
2 exact woolley vinyl single shower curtain
3 exact woolley vinyl solid color single shower curtain
4 exact siniard single shower curtain
5 exact simmon vinyl single shower curtain liner
6 partial 71 '' x71 '' 4pcs shower curtain sets beach theme showe
7 partial newburg check 3d vinyl single shower curtain + hooks
8 irrelevant gilded pineapple shower curtain hooks
9 irrelevant double shower curtain hooks
10 irrelevant stainless steel shower curtain hooks
exact in the top 10: 4
partial in the top 10: 3
irrelevant in the top 10: 3
unjudged in the top 10: 0
products judged for this query: 253
exact products judged for this query: 12
BM25의 상위 10개에는 exact 상품이 4개, partial 상품이 3개, irrelevant 상품(훅)이 3개, 판정되지 않은 상품이 0개 들어 있다. 첫 번째 결과는 partial이다. 데이터셋의 정의에 따르면 이 상품은 쇼핑객이 찾은 대상인 샤워 커튼에는 부합하지만, 수식어인 clear에는 부합하지 않는다.
판정은 가능한 최선의 상위 10개가 무엇으로 이루어지는지도 알려 준다. 판정자들은 이 질의에 대해 상품 253개에 레이블을 붙였고, 그중 12개가 exact였다. 따라서 이상적인 순위라면 열 자리를 모두 exact 상품으로 채울 수 있으며, BM25가 채운 exact 상품은 4개였다. 지표가 측정해야 하는 것이 바로 이 차이다.
이렇게 결과를 읽으면 순위 모델이 어떤 종류의 실수를 하는지 보인다. 여기서 BM25는 액세서리의 단어에 맞추었고 수식어를 놓쳤다. 많은 질의에 대한 평균은 이런 실수를 가리며, 눈에 보이는 실수라야 고칠 수 있다.
작업 규칙: 지표를 계산하기 전에 질의 다섯 개 또는 열 개의 상위 결과를 판정과 나란히 놓고 읽어라. 액세서리, 놓친 수식어, 잘못된 크기 같은 패턴을 찾아라. 상위 결과 가운데 판정되지 않은 것이 여럿이면, 그 질의의 점수를 신뢰하기 전에 판정을 받아라.
이 부분은 아직 번역되지 않았으므로 영어로 표시된다.
Score one ranking with nDCG@10
To compare rankings, you need one number per ranking. It should reward matching products, and exact ones more than partial ones, and count a product near the top more than one near the bottom, because shoppers read from the top. Normalized discounted cumulative gain at 10, nDCG@10, does all three.
In words: each of the top 10 results earns a gain equal to its grade: 2 for exact, 1 for partial, 0 for irrelevant. The gain is divided by a discount that grows with position: log2 of the position plus one, so the first result counts in full and the third counts half. Adding up the discounted gains gives the discounted cumulative gain, DCG. Dividing by the DCG of the best possible top 10 for that query, the ideal DCG, puts the score between 0 and 1, where 1 means no better top 10 exists. An unjudged product earns 0.
DCG@10 = sum over positions i from 1 to 10 of grade(i) / log2(i + 1)
nDCG@10 = DCG@10 / ideal DCG@10
The WANDS paper scores search with nDCG@10 too, with gains of 1.0, 0.5 and 0 for exact, partial and irrelevant (Chen et al., section 6). The book's grades are twice those gains, which doubles both DCG and ideal DCG, so nDCG doesn't change.
Take BM25's ranking of clear shower curtain: a partial product first, then exact and partial curtains, then three irrelevant hooks. Compute DCG, the ideal DCG and their ratio by hand, then compare with lib.metrics, the shared module of the book's metrics:
1"""Computes nDCG@10 for one test query step by step, then checks it against `lib.metrics`. The prose's point rests on the `ndcg@10` line and on the library line that prints the same value."""2import math34from lib.lexical import LexicalSearch5from lib.metrics import ndcg_at_k6from lib.store import load_products, load_queries, qrels78products = load_products()9query = load_queries().at[246, "query"] # "clear shower curtain"10judgments = qrels("test")[246] # {product_id: grade}11ranking = list(LexicalSearch(products).search(query, k=10).index)1213grades = [judgments.get(product_id, 0) for product_id in ranking] # unjudged counts as 014# positions count from 1, so the first result is discounted by log2(2) = 115dcg = sum(grade / math.log2(position + 1) for position, grade in enumerate(grades, start=1))16best = sorted(judgments.values(), reverse=True)[:10] # the best possible top 1017ideal = sum(grade / math.log2(position + 1) for position, grade in enumerate(best, start=1))1819print("grades:", grades)20print("best grades:", best)21print(f"dcg@10: {dcg:.3f}")22print(f"ideal dcg@10: {ideal:.3f}")23print(f"ndcg@10: {dcg / ideal:.3f}")24print(f"ndcg@10 from lib.metrics: {ndcg_at_k(ranking, judgments, 10):.3f}")
python ndcg-for-one-query.pygrades: [1, 2, 2, 2, 2, 1, 1, 0, 0, 0]
best grades: [2, 2, 2, 2, 2, 2, 2, 2, 2, 2]
dcg@10: 5.586
ideal dcg@10: 9.087
ndcg@10: 0.615
ndcg@10 from lib.metrics: 0.615
The ranking earns a DCG of 5.586. The best possible top 10 holds only exact products and earns 9.087. Their ratio, 0.615, matches what ndcg_at_k in lib.metrics returns: 0.615. The partial product at position 1 costs the most of any single result, because it holds the place with no discount. Each hook costs less, because it sits lower, but together the hooks cost more.
That one number sums up what the judgments showed: a ranking that found the right kind of product and made two kinds of mistake. You can compute it for any ranking of any judged query, and compare.
Working rule: Use nDCG@10 as your headline metric when your judgments are graded and the first page of results matters most, as it does in a store's search. When you have only yes-or-no labels, or when shoppers look at one result only, another metric fits better. Chapter 3 builds the alternatives and says when to choose each.
모든 테스트 쿼리 채점하기
쿼리 하나는 일화에 불과하다. 랭커를 판단하려면 고정된 쿼리 집합에서 점수를 매겨 평균을 내야 하며, 이 책에서는 그 집합이 테스트 쿼리이다. 한 집합의 모든 쿼리에 대한 순위를 묶어 run이라 하며, {query_id: [product_id, ...]} 형태로 쓰고 가장 좋은 상품을 앞에 둔다. lib.metrics.evaluate는 판정에 비추어 run을 채점하고 쿼리마다 한 행씩 반환한다.
evaluate는 nDCG@10과 함께 judged@10을 보고한다. 상위 10개 중 해당 쿼리에 대한 판정이 하나라도 있는 상품의 비율이다. 판정이 없는 상품은 등급 0으로 계산되므로, judged@10이 낮으면 점수의 일부는 추측이다. 이 지표로는 관련 없는 상품과, 관련은 있지만 아무도 판정하지 않은 상품을 구별할 수 없다.
이제 모든 테스트 쿼리에 BM25를 실행하고, 쿼리마다 상위 100개 상품을 남겨 run을 채점한다.
1"""Scores BM25 on every test query, keeping each query's top 100, and prints the averages and the spread. The prose's point rests on the mean and judged@10 lines and on the three lowest queries printed after them."""2from lib.lexical import LexicalSearch3from lib.metrics import evaluate4from lib.store import load_products, load_queries, qrels56LABELS = {2: "exact", 1: "partial", 0: "irrelevant"}78products, queries, judgments = load_products(), load_queries("test"), qrels("test")9run = LexicalSearch(products).run(queries, k=100) # {query_id: [product_id, ...]}10scores = evaluate(run, judgments) # one row per test query11scores["query"] = queries["query"]1213print("test queries:", len(scores))14print(f"mean ndcg@10: {scores['ndcg@10'].mean():.3f}")15print(f"mean judged@10: {scores['judged@10'].mean():.3f}")16print(f"lowest ndcg@10: {scores['ndcg@10'].min():.3f}")17print(f"median ndcg@10: {scores['ndcg@10'].median():.3f}")18print(f"highest ndcg@10: {scores['ndcg@10'].max():.3f}")19print("queries with ndcg@10 of 1:", (scores["ndcg@10"].round(3) == 1).sum())20print()21lowest = scores.nsmallest(3, "ndcg@10")22print(lowest[["query", "ndcg@10", "judged@10"]].round(3).to_string())23print("lowest query:", lowest["query"].iloc[0])24print(f"judged@10 of the lowest query: {lowest['judged@10'].iloc[0]:.3f}")25print()26# query 133, outdoor light fixtures, is the second-lowest scorer in the table above27print(f"judged@10 for {queries.at[133, 'query']}: {scores.at[133, 'judged@10']:.3f}")28print("bm25 top 5 for", queries.at[133, "query"])29for product_id in run[133][:5]:30 label = LABELS.get(judgments[133].get(product_id), "unjudged")31 print(f" {label:<10} {products.at[product_id, 'name'][:60]}")
python bm25-test-scores.pytest queries: 30
mean ndcg@10: 0.725
mean judged@10: 0.927
lowest ndcg@10: 0.212
median ndcg@10: 0.687
highest ndcg@10: 1.000
queries with ndcg@10 of 1: 4
query ndcg@10 judged@10
query_id
46 beach blue headboard 0.212 0.5
133 outdoor light fixtures 0.286 0.2
292 decorative white pillow 0.467 1.0
lowest query: beach blue headboard
judged@10 of the lowest query: 0.500
judged@10 for outdoor light fixtures: 0.200
bm25 top 5 for outdoor light fixtures
exact malikah 1 -bulb outdoor bulkhead light
unjudged 60 watt equivalent e12 led candle light bulb
unjudged 5w ( 40 watt equivalent ) , b11 led , dimmable light bulb e1
unjudged 60 watt equivalent , a19 led , non-dimmable light bulb , war
unjudged 40 watt equivalent e12 dimmable led edison candle light bulb
30개의 테스트 쿼리에서 BM25의 평균 nDCG@10은 0.725이고, 평균 judged@10은 0.927이다. 이 평균은 상점이 처음으로 측정한 결과이며, 이후의 모든 랭커는 이 값과 비교된다.
평균은 넓은 분포를 가린다. 점수는 0.212에서 1.000까지 퍼져 있고 중앙값은 0.687이며, 4개 쿼리가 만점인 1을 받는다.
점수가 가장 낮은 쿼리 beach blue headboard의 judged@10은 0.500이다. 점수가 낮은 또 다른 쿼리 outdoor light fixtures의 judged@10은 0.200이다. BM25의 상위 5개에는 정확히 맞는 옥외 조명이 하나 있고, 그다음에 쿼리의 단어를 공유하지만 이 쿼리에 대해 아무도 판정하지 않은 전구가 나온다. 전구는 조명 기구가 아니므로 이 경우 등급 0은 아마 타당하다. 그러나 지표는 그것을 알 수 없으며, 다른 랭커나 다른 쿼리에서는 판정되지 않은 상품이 좋은 상품일 수 있다.
작업 규칙: 테스트 쿼리의 평균 nDCG@10을 평균 judged@10과 나란히 보고하고, run을 채점할 때마다 점수가 가장 낮은 쿼리를 읽는다. 한 랭커의 judged@10이 다른 랭커보다 낮으면 nDCG를 비교하기 전에 그 이유를 찾는다. 차이는 더 나쁜 결과가 아니라 판정 누락에서 비롯되었을 수 있다.
이 부분은 아직 번역되지 않았으므로 영어로 표시된다.
Search by meaning with embeddings
Lexical retrieval fails when the shopper's words and the product's words differ, or when shared words mean different things, as with the hooks that BM25 ranks for clear shower curtain. Semantic retrieval compares the meanings of the query and the product instead of their words. It rests on embeddings. An embedding is a list of numbers, a vector, that a neural network computes for a piece of text. The network is trained so that texts with similar meanings get vectors pointing in similar directions. You embed every product once, ahead of time. At search time, you embed the query and return the products whose vectors are closest to it. With every vector scaled to length 1, closeness is the dot product of two vectors, which equals the cosine of the angle between them.
Embedding the products ahead of time is what makes this fast. In the paper that introduced Sentence-BERT, finding the most similar pair among 10,000 sentences took about 65 hours on a V100 GPU when BERT read every pair. Embedding each sentence once and comparing the embeddings took about 5 seconds, almost all of it spent computing the embeddings: the comparison itself took about 0.01 seconds (Reimers and Gurevych, 2019).
The store's embeddings come from the open model Snowflake/snowflake-arctic-embed-m-v1.5. Examples never call a model, so that every example runs offline and gives the same output each time. Instead, a recording, here the script recordings/embeddings.py, runs the model once and saves its vectors, and the examples read them offline through lib.recorded. We chose this model because it is open and under the Apache 2.0 licence, so anyone can download the pinned version and rerun the recording. The recording adds to each query the instruction that the model's card asks for ("Represent this sentence for searching relevant passages:") and embeds products as they are (model card). The book's semantic retrieval, built as lib.semantic.SemanticSearch, multiplies every product vector by the query vector and sorts the results. Chapter 6 builds it step by step.
Every product is a vector of 768 numbers, and semantic retrieval compares the query's vector with each one; here are the ten closest, with their judgments:
1"""Ranks the store's products for `clear shower curtain` by the similarity of their recorded embeddings to the query's, and prints the top 10 with each judgment. The prose's point rests on the top 10 counts and the similarity spread line."""2from lib.metrics import ndcg_at_k3from lib.recorded import manifest4from lib.semantic import SemanticSearch5from lib.store import load_products, load_queries, qrels67LABELS = {2: "exact", 1: "partial", 0: "irrelevant"}89products = load_products()10query_id = 24611judgments = qrels()[query_id]12semantic = SemanticSearch() # recorded embeddings of every product and query1314print("embedding model:", manifest("embeddings")["model"])15print("dimensions:", semantic.products.shape[1])16print("query:", load_queries().at[query_id, "query"])17print()18# the query's vector against every product's: unit-length vectors, so each dot product is a cosine19similarity = semantic.search(query_id, k=10)20for rank, (product_id, score) in enumerate(similarity.items(), start=1):21 label = LABELS.get(judgments.get(product_id), "unjudged")22 print(f"{rank:>2} {score:.3f} {label:<10} {products.at[product_id, 'name'][:50]}")2324print()25labels = [LABELS.get(judgments.get(product_id), "unjudged") for product_id in similarity.index]26for label in ["exact", "partial", "irrelevant", "unjudged"]:27 print(f"{label} in the top 10: {labels.count(label)}")28print(f"similarity spread in the top 10: {similarity.max() - similarity.min():.3f}")29print(f"ndcg@10: {ndcg_at_k(list(similarity.index), judgments, 10):.3f}")
python first-semantic-search.pyembedding model: Snowflake/snowflake-arctic-embed-m-v1.5
dimensions: 768
query: clear shower curtain
1 0.577 exact woolley vinyl solid color single shower curtain
2 0.572 exact woolley vinyl single shower curtain
3 0.569 partial newburg check 3d vinyl single shower curtain + hoo
4 0.556 partial rodas vinyl shower curtain
5 0.554 exact siniard single shower curtain
6 0.552 partial 71 '' x71 '' 4pcs shower curtain sets beach theme
7 0.550 exact almazan lightweight single shower curtain
8 0.549 exact simmon vinyl single shower curtain liner
9 0.546 partial vinyl striped shower curtain set
10 0.542 partial cindy window vinyl single shower curtain
exact in the top 10: 5
partial in the top 10: 5
irrelevant in the top 10: 0
unjudged in the top 10: 0
similarity spread in the top 10: 0.035
ndcg@10: 0.793
On this query, semantic retrieval does better than BM25. Its top 10 holds 5 exact products, against BM25's 4, and 0 irrelevant ones. Its nDCG@10 is 0.793, against BM25's 0.615. Notice how close the similarities are: the first and the tenth differ by 0.035. Semantic retrieval always returns something, ranked by small differences.
One query can mislead, so compare the two rankers on every test query:
1"""Scores BM25 and semantic retrieval on every test query and counts the queries each one is better on. The prose's point that the test set is too small to show a pattern rests on the `better` and `size in inches` lines."""2from lib.lexical import LexicalSearch3from lib.metrics import evaluate4from lib.semantic import SemanticSearch5from lib.store import load_products, load_queries, qrels67products, queries, judgments = load_products(), load_queries("test"), qrels("test")8bm25 = evaluate(LexicalSearch(products).run(queries, k=100), judgments)9semantic = evaluate(SemanticSearch().run(queries, k=100), judgments)1011for name, scores in [("bm25", bm25), ("semantic", semantic)]:12 print(f"{name} ndcg@10: {scores['ndcg@10'].mean():.3f}")13 print(f"{name} judged@10: {scores['judged@10'].mean():.3f}")1415# per test query, rounded to three decimals: queries that differ only beyond the third place count as tied16difference = (semantic["ndcg@10"] - bm25["ndcg@10"]).round(3)17print("semantic better:", (difference > 0).sum())18print("bm25 better:", (difference < 0).sum())19print("tied:", (difference == 0).sum())20sized = queries["query"].str.contains(r"\d+\s*(?:inch|in\b|\"|'')") # a size in inches21print("test queries with a size in inches:", sized.sum())22print("of those, bm25 better:", (difference[sized] < 0).sum())23print()24print("queries where bm25 is better:")25print(queries.loc[difference[difference < 0].index, "query"].to_string())
python compare-lexical-semantic.pybm25 ndcg@10: 0.725
bm25 judged@10: 0.927
semantic ndcg@10: 0.830
semantic judged@10: 0.990
semantic better: 21
bm25 better: 7
tied: 2
test queries with a size in inches: 2
of those, bm25 better: 2
queries where bm25 is better:
query_id
63 monthly calendar
72 fold in half table
94 kohler whitehaven farmhouse kitchen sink
285 48" sliding single track , barn door for laundry
292 decorative white pillow
293 wood floor protector mat
482 48 inch kitchen hood
Averaged over the test queries, semantic retrieval is ahead: an nDCG@10 of 0.830 against BM25's 0.725. Its judged@10, 0.990, is also higher than BM25's 0.927, so part of BM25's deficit could come from products nobody judged. For outdoor light fixtures, at least, those products are light bulbs, and a score of 0 is fair.
Semantic retrieval is better on 21 queries, BM25 on 7, and 2 are tied. The store has only 30 test queries, so chapter 3 tests whether a gap like this one is larger than chance.
BM25 is better on 2 of the 2 test queries that give a size in inches. Exact tokens such as sizes and model numbers may favour matching words, but a count this small can't show a pattern. On the train queries that contain a digit, BM25 is better on 1 of 5 (Find the queries each retriever wins), and the embedding model matches digits too, as text (Check queries that contain digits).
A theoretical result shows where single vectors fall short. Weller et al. prove that an embedding's dimension limits which combinations of top results a single vector can return. On LIMIT, a dataset they built to test that limit, BM25 finds 93.6% of the relevant items in its top 100, and the best single-vector model 18.9%. On a variant of LIMIT with words replaced by synonyms, BM25's score drops by more than 89% (Weller et al., 2025). Lexical and semantic retrieval fail in different places.
Working rule: Add semantic retrieval when shoppers describe what they want in their own words. Don't assume lexical retrieval wins on exact terms such as sizes, brands and model numbers: before you rely on it there, count which retriever wins the train queries that contain them. On this data, semantic retrieval alone scores highest so far, so it is the ranker to beat.
역순위 결합으로 두 순위 결합하기
어휘 기반 검색과 의미 기반 검색은 서로 다른 지점에서 실패하므로, 하이브리드 검색은 두 검색의 순위, 여기서는 BM25의 순위와 의미 기반 검색의 순위를 결합하여 양쪽의 강점을 모두 살리려 한다. 두 점수는 척도가 달라서 직접 더할 수 없다(7장에서 그 이유를 보인다). 역순위 결합(reciprocal rank fusion, RRF)은 순위만 사용하여 이 문제를 피한다. 각 상품은 자신이 포함된 순위마다 1을 '상수와 순위의 합'으로 나눈 값을 점수로 얻고, 이 점수들을 모두 더한다.
RRF(product) = sum over rankings r of 1 / (k + rank of product in r), with k = 60
상수 k는 상위권의 차이를 완만하게 한다. k = 60이면 1위는 1/61, 10위는 1/70을 얻는다. 따라서 두 순위의 상위권에 모두 든 상품이 한쪽 순위의 맨 위에만 있는 상품보다 높은 점수를 받는다. 저자들은 예비 연구에서 k = 60으로 정했고 "그 선택은 결정적이지 않았다"고 보고했다(Cormack et al., 2009). 이들은 두 종류의 실행 결과를 결합했다. 하나는 Text REtrieval Conference의 공동 평가인 TREC에 제출된 결과이고, 다른 하나는 한 검색 시스템의 설정 30가지이다. 이 실험에서 RRF는 평균 정밀도 평균(MAP)에서 가장 좋은 개별 실행보다 평균 4%에서 5% 높았다. MAP는 질의별로 관련 있는 결과가 나타난 각 위치의 정밀도를 평균한 값을 다시 질의에 걸쳐 평균한 것이다. 이는 어휘 기반 순위 하나와 의미 기반 순위 하나를 결합하는 상황과는 다르며, 하이브리드 검색을 다룬 이후 연구에서는 정규화한 점수의 가중합을 조정하면 RRF보다 좋은 결과를 얻는다고 보고했다(Bruch et al., 2023). 7장에서는 학습 질의로 두 방식을 모두 조정한다. 여기서는 RRF를 조정하지 않고 실행한다.
RRF는 두 함수로 직접 작성할 만큼 짧다. rrf_scores는 한 질의의 상품들에 점수를 매기고, reciprocal_rank_fusion는 모든 질의에 대해 상품을 정렬한다. 이때 점수가 같은 상품은 product_id 순서로 놓아, 어느 한 순위가 먼저 전달되었다는 이유로 동점에서 이기지 않게 한다. hybrid-rrf 예제는 모든 테스트 질의에 대해 BM25와 의미 기반 검색의 상위 100개를 결합하고, clear shower curtain에서 맨 앞에 놓인 두 상품의 점수를 확인한 다음, 하이브리드 실행의 점수를 계산한다.
BM25는 0.725, 의미 기반 검색은 0.830를 기록하므로, 더 약한 검색기가 같은 비중의 표를 얻는다. 그 표가 결합 결과를 끌어내릴까, 아니면 두 검색기가 서로의 누락을 충분히 메워 의미 기반 검색을 능가할까? 출력을 읽기 전에 세 가지 답 중 하나를 정하라. 의미 기반 검색보다 높다, 둘 사이이다, BM25보다 낮다.
1"""Fuses BM25 and semantic retrieval with reciprocal rank fusion, by hand for one query and then for every test query. The prose's point rests on the hybrid's `ndcg@10` line and on the two scores printed for the first product, which agree."""2from lib.lexical import LexicalSearch3from lib.metrics import evaluate4from lib.semantic import SemanticSearch5from lib.store import load_products, load_queries, qrels678def rrf_scores(rankings, k=60):9 """Each product scores the sum of 1 / (k + rank) over the rankings that contain it."""10 scores = {}11 for ranking in rankings:12 for rank, product_id in enumerate(ranking, start=1):13 scores[product_id] = scores.get(product_id, 0) + 1 / (k + rank)14 return scores151617def reciprocal_rank_fusion(runs, k=60, depth=100):18 """For each query, products in order of their RRF scores, best first; equal scores by product_id."""19 fused = {}20 for query_id in runs[0]:21 scores = rrf_scores([run[query_id] for run in runs], k)22 # equal scores fall back to product_id, so neither ranking wins ties by being passed first23 fused[query_id] = sorted(scores, key=lambda product_id: (-scores[product_id], product_id))[:depth]24 return fused252627products, queries, judgments = load_products(), load_queries("test"), qrels("test")28bm25, semantic = LexicalSearch(products).run(queries, k=100), SemanticSearch().run(queries, k=100)29hybrid = reciprocal_rank_fusion([bm25, semantic])3031for place, product_id in zip(["first", "second"], hybrid[246]): # query 246, "clear shower curtain"32 # list.index counts from 0, and a rank counts from 133 bm25_rank, semantic_rank = bm25[246].index(product_id) + 1, semantic[246].index(product_id) + 134 print(f"{place} product: {products.at[product_id, 'name']}")35 print(f"{place} product's bm25 rank: {bm25_rank}")36 print(f"{place} product's semantic rank: {semantic_rank}")37 # the formula with the two ranks in it, where k = 6038 print(f"{place} product's score by hand: {1 / (60 + bm25_rank) + 1 / (60 + semantic_rank):.5f}")39 print(f"{place} product's score from rrf_scores: {rrf_scores([bm25[246], semantic[246]])[product_id]:.5f}")4041hybrid_scores, semantic_scores = evaluate(hybrid, judgments), evaluate(semantic, judgments)42print(f"hybrid ndcg@10: {hybrid_scores['ndcg@10'].mean():.3f}")43print(f"hybrid judged@10: {hybrid_scores['judged@10'].mean():.3f}")44difference = (hybrid_scores["ndcg@10"] - semantic_scores["ndcg@10"]).round(3)45print("better than semantic:", (difference > 0).sum())46print("worse than semantic:", (difference < 0).sum())47print("tied with semantic:", (difference == 0).sum())
python hybrid-rrf.pyfirst product: woolley vinyl solid color single shower curtain
first product's bm25 rank: 3
first product's semantic rank: 1
first product's score by hand: 0.03227
first product's score from rrf_scores: 0.03227
second product: woolley vinyl single shower curtain
second product's bm25 rank: 2
second product's semantic rank: 2
second product's score by hand: 0.03226
second product's score from rrf_scores: 0.03226
hybrid ndcg@10: 0.812
hybrid judged@10: 0.980
better than semantic: 9
worse than semantic: 14
tied with semantic: 7
결합 결과의 첫 번째 상품 woolley vinyl solid color single shower curtain은 BM25 순위에서 3위, 의미 기반 검색 순위에서 1위이다. 이 상품의 점수는 각 순위에서 하나씩 항을 더한 1/(60 + 3) + 1/(60 + 1)이며, 직접 계산하면 0.03227이 되어 rrf_scores가 준 값과 같다.
두 번째 상품 woolley vinyl single shower curtain은 BM25 순위에서 2위, 의미 기반 검색 순위에서 2위이고 점수는 0.03226이므로 2위가 된다.
하이브리드 실행의 점수는 0.812이며, BM25보다 높고 의미 기반 검색 단독보다 낮다. 질의별로 보면 의미 기반 검색을 이기는 질의가 9개, 지는 질의가 14개, 비기는 질의가 7개이다. judged@10은 0.980이다.
따라서 여기서는 결합이 의미 기반 검색보다 나아지지 않았다. 다만 이것이 검색을 나쁘게 만들었다는 뜻은 아니다. 왜 도움이 되지 않았는지는 7장에서 다룬다. 7장은 결합하기 전에 각 검색기를 점검하고 학습 질의로 혼합 비율을 조정한다.
의미 기반 검색보다 높다고 답했다면, 얻어야 할 교훈은 두 순위를 결합한다고 해서 이득이 보장되지는 않는다는 점이다. 결합도 다른 변경과 마찬가지로 측정해야 한다.
작업 규칙: 검색기를 결합하기 전에 각각을 단독으로 측정한다. 조정하지 않은 기본값으로 k = 60의 RRF를 사용하고, 결합한 순위가 보류한 질의에서 가장 좋은 단일 검색기보다 나을 때만 유지한다. 한 검색기가 훨씬 약하다면 같은 비중의 표를 받아들이지 말고, 순위에 가중치를 주거나 학습 질의로 k를 조정해 본다. 7장은 이 상점의 데이터에서 두 방법을 모두 시도하며, 테스트 질의에서 보고한 조정된 혼합 중 의미 기반 검색을 이기는 것은 없다(테스트 질의에서 조정된 혼합 보고하기).
이 부분은 아직 번역되지 않았으므로 영어로 표시된다.
Rerank the top 50 with a cross-encoder
Every stage so far has been retrieval, which has to score every product in the catalog for every query and so must be cheap. Reranking reorders a short list from retrieval with a slower model, one that can afford to look at the query and each product together because it only sees a few dozen products.
One kind of reranking model is a cross-encoder, which reads the query and the product text together, as one input, and outputs one relevance score for the pair, scoring each pair independently (Nogueira and Cho, 2019). An embedding model reads the product alone, so its vectors can be computed ahead of time. A cross-encoder needs the query, so it can't compute anything ahead of time, and running it on every pair costs time: the slow side of the Sentence-BERT comparison was BERT reading every pair of sentences as a cross-encoder. So a cross-encoder reranks a shortlist that a cheaper first stage retrieves (Nogueira and Cho, section 2; Retrieve & Re-Rank). In return for that cost, it can relate every word of the query to every word of the product through attention, the part of the model that lets each word draw on the others in its input. Reimers and Gurevych (2019, section 4.3) note that BERT can use attention to compare the two sentences directly. In 2019, on the MS MARCO passage benchmark, reranking BM25's top 1,000 passages this way improved MRR@10, a measure of how high the first relevant passage lands, by 27% relative to the best earlier result (Nogueira and Cho).
A cross-encoder is a language model, but not a large language model (LLM). An LLM is a model trained on large amounts of text to generate text in response to a prompt. Later chapters use LLMs to judge, rewrite and rerank. A cross-encoder generates nothing: it outputs one number.
The store's cross-encoder is the open model Alibaba-NLP/gte-reranker-modernbert-base, chosen for the same reason as the embedding model: it is open and under the Apache 2.0 licence, so anyone can rerun the recording. Its recording, recordings/cross_encoder.py, saves the model's scores once, for every product in the union of BM25's and semantic retrieval's top 100 for each query, so every product in the hybrid's top 50 has a score. The rerank step reorders the top 50 of the hybrid run, BM25 and semantic retrieval fused with RRF, by those scores, keeps the rest in place, and scores the result, which the book calls the reranked hybrid:
1"""Reorders the top 50 of the hybrid run by cross-encoder score, keeps the rest in place, and scores the result. The prose's point rests on the reranked nDCG@10 line, set beside the hybrid's score, and on the better and worse counts."""2from lib.fusion import reciprocal_rank_fusion3from lib.lexical import LexicalSearch4from lib.metrics import evaluate5from lib.recorded import cross_encoder_scores, manifest6from lib.semantic import SemanticSearch7from lib.store import load_products, load_queries, qrels89products, queries, judgments = load_products(), load_queries("test"), qrels("test")10bm25 = LexicalSearch(products).run(queries, k=100)11semantic = SemanticSearch().run(queries, k=100)12hybrid = reciprocal_rank_fusion([bm25, semantic]) # the same function as in the hybrid section1314recorded = cross_encoder_scores() # one score per (query, product) pair15score = {(query_id, product_id): value for query_id, product_id, value in recorded.itertuples(index=False)}16print("cross-encoder model:", manifest("cross-encoder")["model"])1718depth = 50 # rerank the top 5019reranked = {}20for query_id, ranking in hybrid.items():21 # the top 50 are reordered by score; equal scores keep the hybrid's order22 top = sorted(ranking[:depth], key=lambda product_id: score[(query_id, product_id)], reverse=True)23 # products below the top 50 keep their hybrid order24 reranked[query_id] = top + ranking[depth:]2526before, after = evaluate(hybrid, judgments), evaluate(reranked, judgments)27print(f"reranked ndcg@10: {after['ndcg@10'].mean():.3f}")28print(f"reranked judged@10: {after['judged@10'].mean():.3f}")29difference = (after["ndcg@10"] - before["ndcg@10"]).round(3)30print("better than hybrid:", (difference > 0).sum())31print("worse than hybrid:", (difference < 0).sum())
python cross-encoder-rerank.pycross-encoder model: Alibaba-NLP/gte-reranker-modernbert-base
reranked ndcg@10: 0.798
reranked judged@10: 0.980
better than hybrid: 11
worse than hybrid: 12
Here the reranker doesn't help. It scores 0.798 against 0.812 for the hybrid run it started from. It improves 11 queries and hurts 12. Across the chapter, on the test queries:
| Ranker | nDCG@10 | judged@10 |
|---|---|---|
| BM25 | 0.725 | 0.927 |
| Semantic | 0.830 | 0.990 |
| Hybrid (RRF, k = 60) | 0.812 | 0.980 |
| Reranked hybrid (top 50) | 0.798 | 0.980 |
One query it made worse, refrigerator with ice an water in door, shows how a reranker loses ground:
1"""Shows the hybrid and reranked top 5 for one query whose nDCG@10 the reranker lowered, with each product's cross-encoder score and judgment. The prose's point rests on the two rank lines and the two nDCG@10 lines."""2import re34from lib.fusion import reciprocal_rank_fusion5from lib.lexical import LexicalSearch6from lib.metrics import ndcg_at_k7from lib.recorded import cross_encoder_scores8from lib.semantic import SemanticSearch9from lib.store import load_products, load_queries, product_text, qrels1011LABELS = {2: "exact", 1: "partial", 0: "irrelevant"}1213products = load_products()14query_id = 40615query = load_queries().at[query_id, "query"]16judgments = qrels()[query_id]17recorded = cross_encoder_scores()18score = recorded[recorded["query_id"] == query_id].set_index("product_id")["score"]1920bm25 = {query_id: list(LexicalSearch(products).search(query, k=100).index)}21semantic = {query_id: list(SemanticSearch().search(query_id, k=100).index)}22hybrid = reciprocal_rank_fusion([bm25, semantic])[query_id]23# the top 50 are reordered by score; the rest stay in the hybrid's order24reranked = sorted(hybrid[:50], key=score.get, reverse=True) + hybrid[50:]2526print("query:", query)27for name, ranking in [("hybrid", hybrid), ("reranked", reranked)]:28 first_exact = next(rank for rank, product_id in enumerate(ranking, start=1) if judgments.get(product_id) == 2)29 print()30 print(f"{name} ndcg@10: {ndcg_at_k(ranking, judgments, 10):.3f}")31 print(f"{name} rank of the first exact product: {first_exact}")32 for product_id in ranking[:5]:33 label = LABELS.get(judgments.get(product_id), "unjudged")34 print(f" {score[product_id]:.2f} {label:<10} {products.at[product_id, 'name']}")353637exact = next(product_id for product_id in hybrid if judgments.get(product_id) == 2)38words = set(re.findall(r"\w+", product_text(products).at[exact].lower())) # name, class, category, description39print()40print("hybrid's first exact product:", products.at[exact, "name"])41print("its text says ice or water:", "yes" if {"ice", "water"} & words else "no")
python rerank-one-query.pyquery: refrigerator with ice an water in door
hybrid ndcg@10: 0.660
hybrid rank of the first exact product: 3
1.95 partial apec water icemaker-kit-38-14-ro refrigerator/icemaker installation accessory
2.06 partial 36 '' french door 26.1 cu . ft. smart energy star refrigerator with interior ice maker
1.44 exact 36 '' french door refrigerator 27 cu . ft. energy star refrigerator
2.19 partial 36 '' energy star counter-depth french door 22.6 cu . ft. refrigerator with ice maker
2.19 partial 800 series 21 cu . ft. smart energy star counter depth french door refrigerator with internal water dispenser
reranked ndcg@10: 0.533
reranked rank of the first exact product: 9
2.20 partial smeg classic 36 '' french door 22 cu . ft. refrigerator with ice maker
2.19 partial 36 '' energy star counter-depth french door 22.6 cu . ft. refrigerator with ice maker
2.19 partial 800 series 21 cu . ft. smart energy star counter depth french door refrigerator with internal water dispenser
2.10 partial 28 '' energy star counter depth top freezer 12.56 cu . ft. refrigerator with auto ice maker
2.06 partial 36 '' french door 26.1 cu . ft. smart energy star refrigerator with interior ice maker
hybrid's first exact product: 36 '' french door refrigerator 27 cu . ft. energy star refrigerator
its text says ice or water: no
The cross-encoder buries the first exact product, moving it from rank 3 to rank 9, and drops this query's nDCG@10 from 0.660 to 0.533. The reranked top 5 are refrigerators whose names say ice maker or water dispenser, and the judges marked them partial: they match the thing asked for, a refrigerator, but not the modifier, ice and water in the door. The first exact product in the hybrid run, a French-door refrigerator, gets the lowest cross-encoder score of the hybrid's top 5.
The judges marked that refrigerator exact, so by the dataset's definition it fully matches the query. Yet its text says nothing about ice or water, so no model that reads only the text could rank it on that modifier. WANDS's judges saw each product's image and web page as well as its name (annotation guidelines), which probably let them see what the text doesn't say. A general-purpose reranker has never seen this store's judgments. Chapter 9 trains rankers on the train queries' judgments.
Working rule: Rerank a shortlist, here 50 products, because the cost grows with every pair. Adopt a reranker only when it beats the best retrieval alone on held-out queries. This one didn't improve on the hybrid run or on semantic retrieval, so the store's best pipeline so far is semantic retrieval alone. Before you give up on reranking, try a different depth, or a ranker trained on your own judgments. Neither a different depth nor LambdaMART beats semantic retrieval on the test queries in chapter 9 (Set the rerank depth from a budget, Report LambdaMART on the test queries).
이 부분은 아직 번역되지 않았으므로 영어로 표시된다.
What goes wrong in production
The tokenizer drops words without telling you. bm25s's default tokenizer keeps only tokens of two or more characters and doesn't stem words (bm25s tokenizer). Run it on a train query and a test query, and count the products it can't match:
1"""Tokenizes one train query, one test query and the products' text with bm25s, and counts the products that have `chairs` or only `chair`. The prose's points rest on the printed token lists and on the two product counts."""2import bm25s34from lib.store import load_products, product_text567def tokenize(texts):8 """The tokenizer lib.lexical uses for products and queries, with bm25s's default settings."""9 # stopwords="en" removes English words such as "and"; no stemmer is set, so "chairs" and "chair" stay different tokens10 return bm25s.tokenize(texts, stopwords="en", return_ids=False, show_progress=False)111213for query in ["ge top loading washer 4.5", "bistro table and chairs"]:14 print(f"{query!r} -> {tokenize([query])[0]}")1516products = tokenize(product_text(load_products()).tolist())17print()18print("products with the token chairs:", sum("chairs" in tokens for tokens in products))19print("products with chair but not chairs:", sum("chair" in tokens and "chairs" not in tokens for tokens in products))
python tokenizer-drops-terms.py'ge top loading washer 4.5' -> ['ge', 'top', 'loading', 'washer']
'bistro table and chairs' -> ['bistro', 'table', 'chairs']
products with the token chairs: 895
products with chair but not chairs: 359
The 4.5 in ge top loading washer 4.5 disappears, because 4 and 5 are one character each, so that number plays no part in the ranking. And chairs matches only products that say chairs: 895 products do, while 359 say chair and never chairs, and none of those can match that word of bistro table and chairs. Neither case raises an error. Chapter 5 shows both problems and adds stemming.
Unjudged results look like failures. A ranker that surfaces products nobody judged gets lower scores than it may deserve, and a ranker that surfaces the judged ones gets an advantage. Report judged@10 with every score, and have new products judged when it drops.
Scores from a sample are probably optimistic. The sample keeps 12,000 products, nearly all judged for some query. The full WANDS catalog has 42,994 (repository), and a store's catalog may be far larger, with more near misses competing for the top 10. Nothing in this chapter measures that, but expect lower scores on a full catalog. Don't compare the scores here with the WANDS paper's either: it ran the BM25 of Solr, a search server, over product titles and descriptions (Chen et al.).
Model outputs belong to one model version. Product and query embeddings must come from the same model at the same version, or their similarities aren't comparable. Change the model and you re-embed the whole catalog. Each folder in examples/data/recorded/ has a manifest.json that pins the model and its version.
Reranking costs time on every query. Product embeddings are computed once. A cross-encoder runs on every pair, for every query, while the shopper waits. In this chapter, a recording script computes the scores ahead of time on a GPU, which hides that cost. Chapter 9 counts what each rerank depth costs (Set the rerank depth from a budget).
Working rule: Check the tokens your queries become, track judged@10, and record the model version next to every vector you store.
이 장에서 내린 결정
이 기본값들은 이후의 장에서 train 질의로 하나를 시험하기 전까지 유지된다.
- test로 보고하고 train으로 조정한다. 이 장의 모든 점수는 30개의 test 질의에서 얻은 것이다. 이 장은 아무것도 조정하지 않으며, 이후의 장에서는 50개의 train 질의만으로 조정한다.
- 대표 지표는 nDCG@10이며 judged@10을 함께 제시한다. 등급을 이득값으로 사용하고, 판정되지 않은 상품은 등급 0으로 계산한다. judged@10은 이 가정이 결과에 큰 영향을 미치는 경우를 보여 준다.
- 기준선: bm25s의 기본값인 BM25, 즉
k1=1.5와b=0.75를 각 상품의 이름, 클래스, 카테고리, 설명에 적용한다. 조정은 train 질의로 측정할 수 있을 때까지 미룬다. - 기록된 임베딩을 이용한 의미 기반 검색. 공개 모델 하나의 임베딩을 사용하며, 단위 길이 벡터의 내적으로 비교한다. 따라서 모든 예제는 오프라인으로 실행되며 매번 같은 출력을 낸다. 모델 간의 선택은 이후의 문제로 남긴다.
- 하이브리드 검색: k = 60이고 가중치가 동일한 RRF이며, 조정하지 않는다. 이는 RRF를 제안한 저자들이 사용한 방식 그대로이다.
- 재순위화 깊이: 하이브리드의 상위 50개. 기록된 교차 인코더로 점수를 매기므로, 재순위화된 모든 상품에는 기록된 점수가 있다. 실제 상점에서 깊이는 품질의 문제인 만큼 비용의 문제이기도 하다.
- 이겨야 할 랭커는 의미 기반 검색 단독이다. 이 데이터에서는 융합도 이 재순위화기도 이를 개선하지 못했기 때문이다. 이와 같은 차이가 우연보다 큰지는 3장에서 평가 하니스를 통해 검증한다.
- 이 장이 보여 준 것: 도움이 되리라 기대한 단계인 융합이나 재순위화가 이 데이터에서는 의미 기반 검색을 넘어서지 못했다. 따라서 모든 변경은 유지하기 전에 측정한다.
이 판의 이 장
2026.11.10판예제 통과 2026년 11월 19이 장에서 11개 예제개 중 11개가 통과한다
영어판 2026.11.10판을 번역한 것이다.