의미 기반 검색
읽는 데 약 64분분 걸린다, 13절에 16개 예제이(가) 있음.
AI로 번역함. 영어 원문을 사람이 검증함. 우리가 책을 만드는 방식
영어판 2026.11.10판을 번역한 것이다. 원문 읽기
이 페이지의 내용
- Turn text into embeddings
- Compare embeddings with cosine similarity
- Find the exact nearest neighbours with NumPy
- Trade recall for speed with HNSW
- 테스트 쿼리로 인덱스 확인하기
- Check queries that contain digits
- 코드는 비중이 작을 것으로 예상한다
- 부정 표현에 대한 임베딩 시험
- 임베딩 모델 선택
- 벤치마크는 후보 목록으로 읽는다
- Truncate dimensions, then test the loss
- 운영 환경에서 무엇이 잘못되는가
- 이 장에서 내린 결정
이 장을 마치면 저장된 임베딩으로 카탈로그용 의미 기반 검색을 구축할 수 있다. 코사인 유사도로 벡터를 비교하고, numpy로 질의와 가장 가까운 제품을 정확하게 찾고, 카탈로그가 정확 검색의 한계를 넘어설 때 근사 색인으로 전환하며 그 색인이 놓치는 것을 측정하고, 임베딩이 오답을 내는 질의를 찾아낸다. 또한 자신의 판정 데이터로 시험하여 벡터 크기를 고르고, 같은 방식으로 모델을 비교할 수 있다.
48 inch kitchen hood에 대해 의미 기반 검색은 다른 너비의 후드를 반환하고, rug that is not round에 대해서는 둥근 러그를 반환한다. 그런데도 테스트 질의에서 평균 nDCG@10은 0.830이었다. BM25의 점수는 0.725이며, 의미 기반 검색이 BM25를 앞선다는 점은 3장에서 확립한 사실이다. 이 책은 이를 하나의 클래스 lib.semantic.SemanticSearch로 사용해 왔다. 1장은 소파부터 샤워 커튼까지 아우르는 카탈로그를 가진 상점을 위해 이 클래스를 만들었으며, 하나의 모델이 미리 한 번 계산해 둔 벡터를 사용했다. 상점이 이 검색에 의존하기 전에, 그 벡터가 무엇을 담고 있는지, 훨씬 큰 카탈로그에서 벡터를 검색하는 데 어떤 비용이 드는지, 어떤 질의에서 오답을 내는지 알아야 한다. 이 장은 정확 검색을 직접 구현하고, 근사 색인을 정확 검색과 비교하여 측정하며, 두 방식을 모두 상점의 질의와 판정 데이터로 시험한다.
이 장은 테스트 질의에서 2개의 비교를 계획하며, 두 비교 모두 색인을 시험하는 절에서 다룬다. 각 비교는 질의를 모든 제품과 비교하는 정확 검색을, 질의를 일부 제품하고만 비교하는 근사 색인인 HNSW와 맞세운다. 한 번은 faiss의 기본 설정에서, 한 번은 학습 질의에서 고른 설정에서 비교한다. 각 비교에는 이 책의 규칙이 적용되며, 여기서는 p-값이 0.025 미만이어야 한다. 이 장에서 테스트 질의를 살펴보는 다른 모든 작업은 탐색적이다.
이 부분은 아직 번역되지 않았으므로 영어로 표시된다.
Turn text into embeddings
An embedding places texts with similar meanings near each other; this section opens how. The model that computes an embedding is an encoder, trained so that texts with similar meanings get vectors that point in similar directions, whether or not they share words. Every text gets the same count of numbers, the embedding's dimension, whether it's a three-word query or a product's whole description. One number means nothing on its own; only comparisons between whole vectors do.
Search uses the encoder as a bi-encoder: it encodes each product by itself, once, ahead of time, and each query by itself when the shopper types it. Comparing a query with a product is then arithmetic on two vectors rather than a run of the model, which is what makes it cheap enough to compare one query with a whole catalog. Chapter 1 gives the size of the difference from the Sentence-BERT paper: about 65 hours for BERT to read every pair of 10,000 sentences, against about 5 seconds to embed each sentence once and compare. A model that reads a query and a product together is a cross-encoder, and chapter 1 uses one to rerank a shortlist.
The store's encoder is Snowflake/snowflake-arctic-embed-m-v1.5, an open model with 109 million parameters under the Apache-2.0 licence (model card). Merrick et al. (2024) report that its family's training kept only documents identified as mainly English. We chose it because anyone may use it commercially, it's small, and it was trained so that its vectors can be shortened, which the section on dimensions tests. Examples read a recording instead: vectors that the script recordings/embeddings.py computed once, at a pinned revision, and saved at unit length, so every example sees the same numbers. The model card asks for a prefix on queries only, so the script adds it to each query and embeds products as they are. Examples read the vectors through lib.recorded.embeddings():
1"""Loads the recorded vectors and shows the lengths lib.recorded stores and rescales, and how close two shopper queries are. The prose's point about rounding rests on the stored- and rescaled-length lines, and its point about nearby queries on the median and nearest-query lines."""2import re34import numpy as np56from lib.recorded import RECORDED, embeddings, manifest7from lib.store import load_queries89# unit-length rows, one per ID; the queries carry the model's query prefix10product_ids, product_vectors, query_ids, query_vectors = embeddings()11recording = manifest("embeddings")12print("embedding model:", recording["model"])13print("query prompt:", recording["prompts"]["query"])14print(f"products embedded: {len(product_ids):,}")15print("dimensions:", product_vectors.shape[1])16print("stored as:", recording["dtype"])17print("longest product text, tokens:", recording["longest_product_tokens"])18print("model's input limit, tokens:", recording["max_seq_length"])19# row 0 is product 1; [0, :4] takes its first four numbers20print(f"product {product_ids[0]}, first four numbers:", ", ".join(f"{x:.3f}" for x in product_vectors[0, :4]))21stored = np.load(RECORDED / "embeddings" / "products.npy").astype(np.float32) # as saved, before lib.recorded22stored_lengths, lengths = np.linalg.norm(stored, axis=1), np.linalg.norm(product_vectors, axis=1)23print(f"stored vector lengths: {stored_lengths.min():.5f} to {stored_lengths.max():.5f}")24print(f"lengths after lib.recorded rescales: {lengths.min():.5f} to {lengths.max():.5f}")2526# How close are two shopper queries? Compare every query with every other one.27queries = load_queries().loc[query_ids, "query"].tolist()28similarity = query_vectors @ query_vectors.T # unit length, so the dot product is the cosine29np.fill_diagonal(similarity, np.nan) # leave out each query's similarity to itself30print(f"median similarity of two queries: {np.nanmedian(similarity):.3f}")31for query in ["whimsical solar lights", "wall design shelf"]:32 row = queries.index(query)33 nearest = int(np.nanargmax(similarity[row]))34 # whole words as typed, with no stemming, so `lights` and `light` differ35 shared = set(re.findall(r"\w+", query)) & set(re.findall(r"\w+", queries[nearest]))36 print()37 print(f"{query}, nearest other query:", queries[nearest])38 print(f"{query}, similarity: {similarity[row, nearest]:.3f}")39 print(f"{query}, words in common:", len(shared))
python inspect-recorded-embeddings.pyembedding model: Snowflake/snowflake-arctic-embed-m-v1.5
query prompt: Represent this sentence for searching relevant passages:
products embedded: 12,000
dimensions: 768
stored as: float16
longest product text, tokens: 201
model's input limit, tokens: 512
product 1, first four numbers: -0.033, 0.051, 0.111, -0.012
stored vector lengths: 0.99994 to 1.00006
lengths after lib.recorded rescales: 1.00000 to 1.00000
median similarity of two queries: 0.291
whimsical solar lights, nearest other query: outdoor light fixtures
whimsical solar lights, similarity: 0.637
whimsical solar lights, words in common: 0
wall design shelf, nearest other query: kitchen storage shelves
wall design shelf, similarity: 0.679
wall design shelf, words in common: 0
Each of the 12,000 products is a vector of 768 numbers; product 1's begins -0.033, 0.051, 0.111, -0.012, and none of those numbers means anything alone. The recording stores each number as a float16, in 2 bytes, and lib.recorded widens them to 32-bit floats when it loads them. That rounding moves the stored lengths slightly off 1: they run from 0.99994 to 1.00006. So lib.recorded also scales each vector back to length 1, and the lengths it returns run from 1.00000 to 1.00000. The store's encoder reads text as tokens, word pieces from its own vocabulary, not the terms of chapter 5's tokenizer. The longest product text is 201 tokens, inside the model's limit of 512 tokens, so the model read every product in full.
The median similarity between two of the store's queries is 0.291, on a scale where 1 means two vectors point the same way (cosine similarity). Yet whimsical solar lights finds outdoor light fixtures at 0.637, and wall design shelf finds kitchen storage shelves at 0.679, with 0 and 0 words in common: lights and light are different strings, and so are shelves and shelf. The baseline BM25 doesn't stem words, so it sees nothing shared in either pair. Chapter 5's stemming would match lights with light, but not shelves with shelf.
Working rule: Embed products ahead of time and queries at search time, with the same model at the same revision, and follow the model card on prompts: here, a prefix on queries only. Record the model, revision and prompts next to the vectors, as each manifest.json in examples/data/recorded/ does. Check that your longest product fits the model's input limit before you trust its embedding.
이 부분은 아직 번역되지 않았으므로 영어로 표시된다.
Compare embeddings with cosine similarity
To rank products for a query, you need one number per pair that says how alike two vectors are. Cosine similarity is the cosine of the angle between two vectors: 1 when they point the same way, 0 at right angles and −1 when they point in opposite directions. It ignores how long the vectors are. The dot product multiplies two vectors number by number and adds up the results, so it equals the cosine times both lengths. Scaling every vector to length 1, called normalizing, makes the two the same number, as Sentence Transformers states. The dot product is then the cheaper choice, because it skips computing the lengths.
dot(q, d) = q1 × d1 + q2 × d2 + ... + qn × dn
length(q) = √(q1² + q2² + ... + qn²)
cosine(q, d) = dot(q, d) / (length(q) × length(d))
To see the formula at work, take the test query clear shower curtain and its most similar product: compute the dot product and the cosine by hand, and check the cosine with scikit-learn 1.9.1. Then break the one assumption that makes them agree. Keep the first 256 of each vector's 768 numbers, a shortcut that the section on dimensions tests, but skip the rescaling, and rank every query's products both ways:
1"""Computes the cosine of one test query and its best product by hand and with scikit-learn, then ranks every query's products by dot product and by cosine after the vectors are shortened. The prose's point rests on the line that counts queries whose top 10 changes."""2import numpy as np3from sklearn.metrics.pairwise import cosine_similarity45from lib.recorded import embeddings6from lib.store import load_products78products = load_products()9product_ids, product_vectors, query_ids, query_vectors = embeddings()10query = query_vectors[list(query_ids).index(246)] # clear shower curtain11best = int(np.argmax(product_vectors @ query))12product = product_vectors[best]13print("product:", products.at[product_ids[best], "name"])1415# cosine = dot product / (length of q * length of d)16dot = query @ product17cosine = dot / (np.linalg.norm(query) * np.linalg.norm(product))18print(f"dot product: {dot:.3f}")19print(f"cosine by hand: {cosine:.3f}")20print(f"cosine from scikit-learn: {cosine_similarity([query], [product])[0, 0]:.3f}")2122# Keep the first 256 numbers of each vector and don't rescale: the lengths are no longer 1.23short_products, short_queries = product_vectors[:, :256], query_vectors[:, :256]24lengths = np.linalg.norm(short_products, axis=1)25print(f"shortened product lengths: {lengths.min():.3f} to {lengths.max():.3f}")26# stable sort: tied scores keep their stored order27by_dot = np.argsort(-(short_queries @ short_products.T), axis=1, kind="stable")[:, :10]28# only the products are rescaled: a query's length scales all its scores alike, so it can't change their order29by_cosine = np.argsort(-(short_queries @ (short_products / lengths[:, None]).T), axis=1, kind="stable")[:, :10]30# compares the sets: a new order within the same 10 doesn't count31changed = sum(set(dot_top) != set(cosine_top) for dot_top, cosine_top in zip(by_dot, by_cosine))32print("queries:", len(query_ids))33print("queries whose top 10 changes:", changed)
python cosine-by-hand.pyproduct: woolley vinyl solid color single shower curtain
dot product: 0.577
cosine by hand: 0.577
cosine from scikit-learn: 0.577
shortened product lengths: 0.706 to 0.778
queries: 80
queries whose top 10 changes: 61
On the store's unit-length vectors, the dot product is 0.577, the cosine by hand is 0.577, and scikit-learn's cosine_similarity gives 0.577, so the three agree.
Shortening breaks that. The shortened vectors have lengths from 0.706 to 0.778, and for 61 of the 80 queries, ranking by dot product gives a different top 10 from ranking by cosine. A query's own length scales all of its scores by the same factor, so it can't change their order. A product's length can: it multiplies that product's score for every query, a boost the shopper never asked for.
Some libraries report a distance instead of a similarity. faiss's L2 indexes return the squared Euclidean distance, and its inner product equals cosine similarity only when the vectors are normalized, according to the faiss wiki. For unit vectors, the squared distance is 2 − 2 × cosine, so it ranks products in the same order as cosine, smallest distance first.
Working rule: Normalize every vector once, when you store it, and rank by dot product. Normalize again after anything that changes a vector, such as shortening it or averaging several. If a model's documentation says it was trained for the dot product of unnormalized vectors, follow the documentation instead, and never mix the two in one index.
이 부분은 아직 번역되지 않았으므로 영어로 표시된다.
Find the exact nearest neighbours with NumPy
Retrieval by embeddings is a nearest-neighbour search: find the k products whose vectors are most similar to the query's. Exact search compares the query with every product and keeps the best k, so it never misses one. With unit-length vectors, comparing one query with every product is one matrix–vector product, and comparing many queries is one matrix product, with a row per query and a column per product. What's left is to find the k highest scores in each row. Sorting every row does more work than needed. NumPy's argpartition moves the k highest scores to one side without ordering either side, and NumPy's documentation leaves that order undefined, so you sort only those k.
Chapter 1's semantic search took its rankings from lib.semantic.SemanticSearch. Build its core by hand for the 30 test queries with NumPy 2.5.5, then check it twice: against lib.semantic, and against faiss's IndexFlatIP, which computes the inner product with every vector. The faiss guidelines name two indexes that guarantee exact results, IndexFlatIP and IndexFlatL2:
1"""Builds exact search from one matrix product, then checks its top 10 against lib.semantic and faiss's IndexFlatIP. The prose's point rests on the two 'same top 10' lines, which show the hand-built search matches both."""2import faiss3import numpy as np45from lib.metrics import evaluate6from lib.recorded import embeddings7from lib.semantic import SemanticSearch8from lib.store import load_queries, qrels910product_ids, product_vectors, query_ids, query_vectors = embeddings()11test = load_queries("test").index12rows = np.isin(query_ids, test)131415def top_k(scores, k):16 """Column indices of the k highest scores in each row, best first."""17 # argpartition's kth is 0-based: with k - 1, the first k positions hold the k highest scores, in no order18 candidates = np.argpartition(-scores, k - 1, axis=1)[:, :k] # the top k, in no order19 # stable sort: tied scores keep their candidate order20 order = np.argsort(-np.take_along_axis(scores, candidates, axis=1), axis=1, kind="stable")21 return np.take_along_axis(candidates, order, axis=1)222324scores = query_vectors[rows] @ product_vectors.T # one row per test query, one column per product25top = top_k(scores, 10)26print("similarities computed:", f"{scores.size:,}")27print("similarities per query:", f"{scores.shape[1]:,}")2829semantic = SemanticSearch()30# two lists match only if the same products come in the same order31same = sum(list(product_ids[found]) == list(semantic.search(query_id, 10).index)32 for query_id, found in zip(query_ids[rows], top))33print("same top 10 as lib.semantic:", same, "of", rows.sum())3435flat = faiss.IndexFlatIP(product_vectors.shape[1]) # faiss's exact index: inner product with every product36flat.add(product_vectors)37_, found = flat.search(query_vectors[rows], 10)38# the same products in the same order, row by row39print("same top 10 as faiss IndexFlatIP:", (found == top).all(axis=1).sum(), "of", rows.sum())40run = {query_id: list(product_ids[row]) for query_id, row in zip(query_ids[rows], top)}41print(f"ndcg@10 of the exact run: {evaluate(run, qrels('test'))['ndcg@10'].mean():.3f}")
python exact-search-numpy.pysimilarities computed: 360,000
similarities per query: 12,000
same top 10 as lib.semantic: 30 of 30
same top 10 as faiss IndexFlatIP: 30 of 30
ndcg@10 of the exact run: 0.830
The hand-built search agrees with both checks: it returns the same top 10 as lib.semantic for 30 of 30 test queries, and as faiss for 30 of 30, so its run scores the same mean nDCG@10, 0.830. One matrix product computed every similarity: 360,000 of them, 12,000 per query. The book's shared module, lib.vectors, keeps this top_k for later examples and wraps it in exact_search.
Whether exact search is enough comes down to its cost. Its work grows in step with the catalog: one similarity per product, for every query. Its memory is the vectors themselves:
1"""Prints the bytes one product's float32 vector takes, and the memory the catalog's vectors use, for the store and for a million products. The prose's point rests on the MB line for the store's catalog and the GB line for a million products."""2from lib.recorded import embeddings34# Exact search keeps every product vector in memory, as 32-bit floats.5product_ids, product_vectors, _, _ = embeddings()6print(f"products: {len(product_ids):,}")7print(f"float32 bytes per product: {product_vectors[0].nbytes:,}")8# decimal MB and GB: a megabyte is a million bytes, a gigabyte a billion9print(f"product vectors in memory, MB: {product_vectors.nbytes / 1e6:.1f}")10print(f"the same for 1,000,000 products, GB: {1_000_000 * product_vectors[0].nbytes / 1e9:.1f}")
python exact-search-memory.pyproducts: 12,000
float32 bytes per product: 3,072
product vectors in memory, MB: 36.9
the same for 1,000,000 products, GB: 3.1
Each product takes 3,072 bytes as 32-bit floats, so the store's catalog takes 36.9 MB, and a million products would take 3.1 GB. Douze et al. (2024) advise deciding first whether an index is needed at all, because in some cases a brute-force search is the best option. We judge the store's catalog to be such a case.
Working rule: We recommend exact search until it no longer fits in memory or meets your latency budget, measured with your own catalog on your own hardware. It has no settings to tune and never misses a neighbour, so it is also the ground truth for any approximate index. Store the normalized vectors as one 32-bit matrix, take the top k with argpartition, and sort only those.
이 부분은 아직 번역되지 않았으므로 영어로 표시된다.
Trade recall for speed with HNSW
When exact search costs too much, an approximate nearest-neighbour (ANN) index skips most of its comparisons, and in return may miss some of the true nearest neighbours. An ANN index's quality is its recall@k against exact search: the share of exact search's top k that it also returns. That isn't the recall@k of chapter 3, which compares a ranking with judgments. An index with perfect recall against exact search returns what exact search returns, good or bad.
HNSW, short for hierarchical navigable small world, is a graph index. Each product is a node linked to some of its nearest neighbours, in a stack of layers: the bottom layer holds every product, and each layer above holds fewer. A search starts at the top, moves to whichever linked node is nearest the query, steps down a layer when it gets no closer, and at the bottom explores a list of candidates before it returns the best k (Malkov and Yashunin, 2018). In faiss, M sets the links per node. More links find more true neighbours, but use more memory (faiss wiki). efConstruction, the expansion factor at construction time, sets how many candidates the build considers, and efSearch, the expansion factor at search time, how many a search considers. With faiss 1.16.0, their defaults are 40 and 16, and its graph build is deterministic, because the only build add() uses is a deterministic one (HNSW.h; changelog). So the outputs here repeat exactly. Timings differ from machine to machine, so the examples count work instead: faiss counts the distances it computes in each search (HNSW.h), which for this index are similarities. The count leaves out following links and reading memory, so it is a guide to speed, not a measure.
The store doesn't need an index, but a store with millions of products would, and a catalog small enough to check every answer is the place to learn the trade-off. Build HNSW over the store's 12,000 products with hnsw_index(product_vectors, links=32), lib.vectors's name for M = 32. Then sweep efSearch from 16 to 256 on the train queries:
1"""Builds HNSW with M = 32 and sweeps efSearch on the train queries against exact search, printing recall, the worst query and the work per query. The prose's point rests on the efSearch 16 lines and the two smallest-efSearch lines printed after the table."""2import faiss3import numpy as np45from lib.recorded import embeddings6from lib.store import load_products, load_queries7# EF_SEARCH_GRID runs 16 to 256 in steps of 16; WORST_RECALL is 0.9: at least 9 of the true top 108from lib.vectors import EF_SEARCH_GRID, WORST_RECALL, exact_search, hnsw_index, hnsw_search, recall_against_exact910product_ids, product_vectors, query_ids, query_vectors = embeddings()11names = load_products().loc[product_ids, "name"].str[:36].to_numpy() # by row number, cut to 36 characters12rows = np.isin(query_ids, load_queries("train").index) # tune on train queries only13queries = load_queries().loc[query_ids[rows], "query"].to_numpy()1415index = hnsw_index(product_vectors, links=32) # M = 32, and faiss's default efConstruction of 4016levels = faiss.vector_to_array(index.hnsw.levels) # how many layers each product reaches17for layer in range(1, levels.max() + 1):18 print(f"products in layer {layer}: {(levels >= layer).sum():,}")1920exact = exact_search(query_vectors[rows], product_vectors, 10)21print("\nefSearch mean recall@10 worst queries at 0 similarities per query")22sweep = {}23for ef_search in EF_SEARCH_GRID:24 # similarities is faiss's count of the distances each search computed, per query, not a score25 found, similarities = hnsw_search(index, query_vectors[rows], 10, ef_search)26 # share of exact search's top 10 that the approximate search also found, order ignored27 recall = recall_against_exact(found, exact)28 sweep[ef_search] = (recall, similarities)29 print(f"{ef_search:>8} {recall.mean():>14.3f} {recall.min():>5.1f} {(recall == 0).sum():>12} {similarities:>22.0f}")3031recall, similarities = sweep[16] # faiss's default32print(f"\nefSearch 16, recall@10 against exact: {recall.mean():.3f}")33print(f"efSearch 16, similarities computed per query: {similarities:.0f}")34print(f"efSearch 16, train queries with recall@10 of 0: {(recall == 0).sum()}")35print("efSearch 16, those queries:", ", ".join(queries[recall == 0]))36print("smallest efSearch with no query at 0:", min(ef for ef, (r, _) in sweep.items() if r.min() > 0))37chosen = min(ef for ef, (r, _) in sweep.items() if r.min() >= WORST_RECALL)38print(f"smallest efSearch with every query at {WORST_RECALL} or more:", chosen)39print(f"chosen efSearch, recall@10 against exact: {sweep[chosen][0].mean():.3f}")40print(f"chosen efSearch, similarities computed per query: {sweep[chosen][1]:.0f}\n")4142# the top 3 at the default efSearch, for the query that fails43found, _ = hnsw_search(index, query_vectors[rows], 3, 16)44row = list(queries).index("foutains with brick look")45for rank, (approximate, nearest) in enumerate(zip(found[row], exact[row]), start=1):46 print(f"{rank} hnsw: {names[approximate]:<36} exact: {names[nearest]}")
python hnsw-efsearch-sweep.pyproducts in layer 1: 12,000
products in layer 2: 369
products in layer 3: 10
efSearch mean recall@10 worst queries at 0 similarities per query
16 0.934 0.0 2 282
32 0.936 0.0 2 346
48 0.940 0.0 2 404
64 0.940 0.0 2 463
80 0.956 0.0 1 530
96 0.978 0.3 0 611
112 0.980 0.3 0 682
128 0.994 0.8 0 767
144 0.998 0.9 0 835
160 0.998 0.9 0 924
176 0.998 0.9 0 1020
192 0.998 0.9 0 1115
208 0.998 0.9 0 1191
224 1.000 1.0 0 1271
240 1.000 1.0 0 1349
256 1.000 1.0 0 1441
efSearch 16, recall@10 against exact: 0.934
efSearch 16, similarities computed per query: 282
efSearch 16, train queries with recall@10 of 0: 2
efSearch 16, those queries: foutains with brick look, sheffield home bath set
smallest efSearch with no query at 0: 96
smallest efSearch with every query at 0.9 or more: 144
chosen efSearch, recall@10 against exact: 0.998
chosen efSearch, similarities computed per query: 835
1 hnsw: dayse outdoor square cushion with fi exact: tribeca 2 '' x 10 '' porcelain brick
2 hnsw: 3 piece wall decor set exact: barbados 5 '' x 10 '' ceramic brick
3 hnsw: rustic autumn ivy branch exact: kimary 0.75 '' x 1.5 '' glass brick
The graph thins fast: the bottom layer holds all 12,000 products, the next 369 and the top 10.
The mean hides a failure. At the default efSearch of 16, a query computes 282 similarities instead of 12,000, and mean recall@10 against exact search is 0.934. Yet 2 train queries get none of their true top 10.
Those queries are foutains with brick look, sheffield home bath set. Look at foutains with brick look, one of them: HNSW returns a cushion, a wall decor set and an ivy branch, while exact search's top 3 are brick-look tiles.
The smallest efSearch in the sweep at which no train query is at 0 is 96, but the sweep's table shows that its worst query still misses most of its true top 10. So choose by a stricter rule: the smallest efSearch at which every train query finds at least 9 of its true top 10. That is 144, with a mean recall of 0.998 and 835 similarities per query.
Working rule: Use an ANN index only when exact search is too slow or too large. Tune it on train queries by its recall against exact search, at the k your pipeline uses, and choose by the worst query, not the mean. We recommend the smallest efSearch at which every query keeps at least 9 of its true top 10, with M = 32. This chapter tunes only efSearch; M and efConstruction change the graph itself, and deserve the same sweep on a large catalog. Aumüller et al. (2018) find that very different ANN methods reach comparable quality-performance trade-offs, so measure on your own data, not by reputation.
테스트 쿼리로 인덱스 확인하기
훈련 쿼리로 조정한 인덱스는 본 적 없는 쿼리에서도 확인해야 한다. 확인 기준은 정확 검색 대비 재현율과 판정 점수에 미치는 영향이다. 이 장에서 계획한 비교는 2개이며, 각 비교는 쌍체 순열 검정으로 하고 t-검정을 함께 본다. 비교 대상은 faiss 기본값인 efSearch 16의 HNSW 인덱스, 그리고 훈련 쿼리에서 고른 설정의 HNSW 인덱스이며, 둘 다 정확 검색과 비교한다. 검색은 k = 100도 요청하는데, 하이브리드 검색이 각 검색기의 상위 100개를 융합하기 때문이다.
1"""Compares HNSW at faiss's default efSearch, and at the setting chosen on train queries, with exact search on the test queries, using paired tests at the planned threshold. The prose's point rests on the default setting's permutation and interval lines, and on the chosen setting's recall line."""2import numpy as np34from lib.metrics import evaluate5from lib.recorded import embeddings6from lib.significance import compare, verdict7from lib.store import load_queries, qrels8from lib.vectors import as_run, choose_ef_search, exact_search, hnsw_index, hnsw_search, recall_against_exact910# Planned before scoring: HNSW at faiss's default efSearch, and at the setting chosen on train, each against exact.11PLANNED = 212threshold = 0.05 / PLANNED # the book's rule splits 0.05 across the planned comparisons13product_ids, product_vectors, query_ids, query_vectors = embeddings()14rows = np.isin(query_ids, load_queries("test").index)15index = hnsw_index(product_vectors, links=32)16# the smallest efSearch at which every train query keeps WORST_RECALL of its true top 1017chosen = choose_ef_search(index, query_vectors[np.isin(query_ids, load_queries("train").index)], product_vectors)18# k = 100 because hybrid retrieval fuses each retriever's top 10019exact = exact_search(query_vectors[rows], product_vectors, 100)20exact_ndcg = evaluate(as_run(exact, query_ids[rows], product_ids), qrels("test"))["ndcg@10"]21print("planned test comparisons:", PLANNED)22print(f"p-value threshold: {threshold:.3f}")23print("efSearch chosen on train queries:", chosen)24print(f"exact ndcg@10: {exact_ndcg.mean():.3f}")2526for name, ef_search in [("default", 16), ("chosen", chosen)]:27 found, _ = hnsw_search(index, query_vectors[rows], 100, ef_search)28 # the first 10 of the 10029 recall = recall_against_exact(found[:, :10], exact[:, :10])30 ndcg = evaluate(as_run(found, query_ids[rows], product_ids), qrels("test"))["ndcg@10"]31 result = compare(ndcg, exact_ndcg, confidence=1 - threshold) # a 97.5% interval, at the test's level32 print(f"\n{name}, recall@10 against exact: {recall.mean():.3f}")33 print(f"{name}, queries with recall@10 of 0: {(recall == 0).sum()}")34 print(f"{name}, those queries:", ", ".join(load_queries().loc[query_ids[rows][recall == 0], "query"]) or "none")35 print(f"{name}, recall@100 against exact: {recall_against_exact(found, exact).mean():.3f}")36 print(f"{name}, hnsw ndcg@10: {ndcg.mean():.3f}")37 # compare rounds each difference to 3 decimals: wins and losses are the nonzero ones38 print(f"{name}, queries whose ndcg@10 changes: {result['wins'] + result['losses']}")39 print(f"{name}, permutation p: {result['permutation p']:.3f}")40 print(f"{name}, t-test p: {result['t-test p']:.3f}")41 print(f"{name}, {1 - threshold:.1%} interval: {result['ci low']:.4f} to {result['ci high']:.4f}")42 for check, passed in verdict(result, threshold).items(): # established, t-test agrees, interval agrees43 print(f"{name}, {check}: {'yes' if passed else 'no'}")
python hnsw-test-recall.pyplanned test comparisons: 2
p-value threshold: 0.025
efSearch chosen on train queries: 144
exact ndcg@10: 0.830
default, recall@10 against exact: 0.930
default, queries with recall@10 of 0: 2
default, those queries: liberty hardware francisco, 48" sliding single track , barn door for laundry
default, recall@100 against exact: 0.849
default, hnsw ndcg@10: 0.791
default, queries whose ndcg@10 changes: 3
default, permutation p: 0.256
default, t-test p: 0.154
default, 97.5% interval: -0.1544 to -0.0006
default, established: no
default, t-test agrees: yes
default, interval agrees: no
chosen, recall@10 against exact: 1.000
chosen, queries with recall@10 of 0: 0
chosen, those queries: none
chosen, recall@100 against exact: 0.994
chosen, hnsw ndcg@10: 0.830
chosen, queries whose ndcg@10 changes: 0
chosen, permutation p: 1.000
chosen, t-test p: 1.000
chosen, 97.5% interval: 0.0000 to 0.0000
chosen, established: no
chosen, t-test agrees: yes
chosen, interval agrees: yes
훈련 쿼리에서 고른 설정은 잘 유지된다. 정확 검색 대비 recall@10은 1.000이고, 평균 nDCG@10은 0.830으로 정확 검색의 0.830과 같으며, 점수가 달라지는 쿼리는 0개이다.
faiss 기본 설정에서는 테스트 쿼리 2개가 실제 상위 10개 중 하나도 얻지 못하고(liberty hardware francisco, 48" sliding single track , barn door for laundry), 평균 nDCG@10은 0.791로 떨어진다. 그러나 점수가 달라지는 쿼리는 3개뿐이며 이 하락은 입증되지 않는다. 순열 검정은 p = 0.256을 주고, t-검정도 p = 0.154로 같은 결론을 낸다. -0.1544 to -0.0006에서 시작하는 97.5% 구간은 0을 제외하므로 검정 결과와 어긋난다. 차이를 만드는 쿼리가 이처럼 적으므로, 여기서 얻은 결론은 정확 검색 대비 잃은 재현율이지 측정된 품질 손실이 아니다.
더 많은 결과를 요청한다고 해서 작은 efSearch가 보완되지는 않는다. 기본 설정에서 recall@100은 0.849이고, 선택한 설정에서는 0.994이다.
그래프는 벡터 외에 메모리도 추가로 사용한다.
1"""Counts the links per product in the HNSW graph, and the bytes per product of the exact and the HNSW index. The prose's point rests on the two bytes-per-product lines, set against faiss's estimate."""2import faiss34from lib.recorded import embeddings5from lib.vectors import hnsw_index67product_ids, product_vectors, _, _ = embeddings()8dimensions = product_vectors.shape[1]9flat = faiss.IndexFlatIP(dimensions) # exact search: the vectors only10flat.add(product_vectors)11index = hnsw_index(product_vectors, links=32) # the vectors, plus the graph's links1213# layer 0 is the bottom layer: 2 × M links there, and M on each upper layer14print("links per product, upper layers:", index.hnsw.nb_neighbors(1))15print("links per product, bottom layer:", index.hnsw.nb_neighbors(0))16# 8M with M = 32: 4 bytes for each of the 2M bottom-layer links17print(f"faiss's estimate, 4d + 8M bytes per product: {4 * dimensions + 8 * 32:,}")18# the saved index's size, spread over the products19for name, built in [("exact index", flat), ("hnsw index", index)]:20 print(f"bytes per product, {name}: {faiss.serialize_index(built).nbytes / len(product_ids):,.0f}")
python hnsw-index-memory.pylinks per product, upper layers: 32
links per product, bottom layer: 64
faiss's estimate, 4d + 8M bytes per product: 3,328
bytes per product, exact index: 3,072
bytes per product, hnsw index: 3,344
각 제품은 상위 계층에 링크 32개, 바닥 계층에 64개를 가지며, 이는 M의 두 배이다. faiss는 벡터당 4d + 8M바이트를 추정한다(faiss wiki). d개의 숫자마다 4바이트이고, 바닥 계층의 링크 2M개마다 4바이트이며, 여기서는 3,328이다. 저장된 인덱스는 제품당 3,344바이트로 이 추정에 가깝고, 정확 인덱스는 3,072바이트이다.
작업 규칙: 조정한 인덱스는 보류한 쿼리에서 정확 검색 대비 재현율로 확인하고, 평균 점수보다 먼저 상위 k개를 모두 잃은 쿼리를 살펴라. 소수의 쿼리에 기댄 nDCG@10 하락은 쌍체 검정으로 입증되기 전까지 확인되지 않은 것으로 다뤄라. 메모리는 벡터뿐 아니라 링크 몫까지 예산에 넣어라.
이 부분은 아직 번역되지 않았으므로 영어로 표시된다.
Check queries that contain digits
An index can only return what the embeddings put near a query, so the next question is which queries the embeddings get wrong. An encoder squeezes a whole query into one vector, so a short token that changes the meaning, such as a size, a model number or a negation, may move that vector very little. Deng et al. (2026) measured this for numbers, in a study of 13 embedding models on synthetic financial text. Their accuracy at telling texts apart by a number was 0.54, only slightly above the 0.5 of random guessing, and four-digit numbers were at the level of guessing. Products aren't financial reports, so check the store's queries.
Few of the store's queries hold a digit, so compare BM25 and semantic retrieval on each that does, from both splits. Nothing is tuned on these queries, and a handful can't establish anything:
1"""Scores BM25 and semantic retrieval on each query with a digit, in both splits, and counts the products whose text writes 7qt or 7 qt. The prose's point rests on the three lines that count which ranker wins and the two product counts."""2import pandas as pd34from lib.lexical import LexicalSearch5from lib.metrics import evaluate6from lib.semantic import SemanticSearch7from lib.store import load_products, load_queries, product_text, qrels89products, queries = load_products(), load_queries()10with_digit = queries[queries["query"].str.contains(r"\d")] # both splits: a diagnosis, nothing is tuned11judgments = {query_id: grades for query_id, grades in qrels().items() if query_id in with_digit.index}12# each ranker returns its top 100; nDCG@10 scores the first 10 of them13scores = pd.DataFrame({14 "split": with_digit["split"],15 "bm25": evaluate(LexicalSearch(products).run(with_digit, k=100), judgments)["ndcg@10"],16 "semantic": evaluate(SemanticSearch().run(with_digit, k=100), judgments)["ndcg@10"],17 "query": with_digit["query"],18}).round(3)1920print("queries with a digit:", len(scores))21print(scores.to_string())22print("semantic better:", (scores["semantic"] > scores["bm25"]).sum())23print("bm25 better:", (scores["bm25"] > scores["semantic"]).sum())24# a gap under 0.03 counts as a near tie: the book's choice, not a measured limit25print("gaps under 0.03:", ((scores["semantic"] - scores["bm25"]).abs() < 0.03).sum())2627text = product_text(products).str.lower()28print("products whose text has the word 7qt:", text.str.contains(r"\b7qt\b").sum())29# '7 qt', but not '17 qt' or '3.7 qt'30print("products whose text has 7 qt:", text.str.contains(r"(?<![\d.])7 qt\b").sum())
python digits-in-queries.pyqueries with a digit: 7
split bm25 semantic query
query_id
45 train 0.288 0.316 70s inspired furniture
62 train 0.554 0.930 7qt slow cooker
101 train 0.569 0.770 48 in entry table with side by side drawer
203 train 0.606 0.740 ge top loading washer 4.5
205 train 0.935 0.930 ines shag rug w003096809
285 test 0.592 0.555 48" sliding single track , barn door for laundry
482 test 1.000 0.921 48 inch kitchen hood
semantic better: 4
bm25 better: 3
gaps under 0.03: 2
products whose text has the word 7qt: 0
products whose text has 7 qt: 8
Neither ranker owns the 7 queries with a digit. Semantic retrieval scores higher on 4 of them and BM25 on 3, and 2 of the gaps are under 0.03. BM25's clearest loss is 7qt slow cooker: the word 7qt appears in 0 products, and 8 write 7 qt.
On the test query 48 inch kitchen hood, BM25 scores higher, so here is semantic retrieval's top 10:
1"""Lists semantic retrieval's top 10 for 48 inch kitchen hood with each hood's width, read from the attributes column the embeddings never saw. The prose's point rests on the two width-48 lines, one for each ranker."""2from lib.lexical import LexicalSearch3from lib.semantic import SemanticSearch4from lib.store import load_products, load_queries, qrels56LABELS = {2: "exact", 1: "partial", 0: "irrelevant"}7products = load_products()8query_id = 482 # 48 inch kitchen hood, a test query9query = load_queries().at[query_id, "query"]10judgments = qrels()[query_id]11# The width sits in the attributes column, which the embeddings never saw: product_text leaves it out.12width = products["attributes"].str.extract(r"\bwidth: ([\d.]+)", expand=False).astype(float)1314semantic = SemanticSearch().search(query_id, k=10).index15bm25 = LexicalSearch(products).search(query, k=10).index16print("query:", query)17print(" judgment width name")18for rank, product_id in enumerate(semantic, start=1):19 label = LABELS.get(judgments.get(product_id), "unjudged")20 print(f"{rank:>2} {label:<10} {width[product_id]:>5g} {products.at[product_id, 'name'][:50]}")2122print()23# exact: a 48-inch hood is 48 wide, where probe-pair-overlap allows half an inch for hoods listed at 35.7524for name, top in [("semantic", semantic), ("bm25", bm25)]:25 print(f"{name}, width 48 in the top 10:", (width[top] == 48).sum())26 print(f"{name}, exact in the top 10:", sum(judgments.get(product_id) == 2 for product_id in top))
python hood-query-digits.pyquery: 48 inch kitchen hood
judgment width name
1 exact 48 48 '' 1200 cfm ducted wall mount range hood in bru
2 exact 48 48 '' 400 cfm ducted wall mount range hood in brus
3 exact 48 48 '' 400 cfm convertible wall mount range hood
4 partial 35.75 36 '' ws-48 series 500 cfm ducted under cabinet ra
5 exact 48 48 '' hand crafted designer wood island hood serie
6 exact 48 48 '' island range hood
7 exact 48 48 inch wall mount chimney range hood with 600 cfm
8 exact 48 48 '' ducted island range hood
9 exact 48 48 '' professional 900cfm ducted island range hood
10 partial 29.8 30 '' 480 cfm convertible under cabinet range hood
semantic, width 48 in the top 10: 8
semantic, exact in the top 10: 8
bm25, width 48 in the top 10: 10
bm25, exact in the top 10: 10
For this query, the embedding matched the digits as text, not the width they stand for. Of semantic retrieval's top 10, 8 hoods have a width of 48. The others are a 36-inch hood from a series named ws-48 and a 30-inch hood rated at 480 cfm, cubic feet per minute, the airflow of its fan. Both contain the digits 48, neither is 48 wide, and the judges marked both partial. BM25's top 10 holds 10 hoods of width 48.
Each hood's width comes from the attributes column, which holds curated fields from WANDS's product features, such as width, shape and material. product_text, the text the model embedded, leaves that column out, so the embeddings never saw it, and a size the shopper types can be checked against it exactly.
One query can't show a pattern. To see how the model handles sizes in general, use probe queries: pairs of queries written for the book that differ only in a size, a code or a negation, which recordings/probes.py embeds with the same model. One of them repeats the test query 48 inch kitchen hood word for word. Probes have no judgments, so the example counts their top 10s and never scores them. The right size comes from the width in attributes for the hoods, and from the name for the rest:
1"""Counts, for each probe pair, the products the two top 10s share and the top-10 products of the right size, without scoring anything. The prose's point rests on the right-size lines for the hood, slow-cooker and bed probes, and on the seat-count and loveseat lines for the sofa probe."""2from lib.recorded import probes3from lib.semantic import SemanticSearch4from lib.store import load_products56# Probe queries written for the book, embedded by the same model. They have no judgments,7# so this example counts what each top 10 holds; it doesn't score anything.8probe_queries, probe_vectors = probes()9products = load_products()10names = products["name"]11width = products["attributes"].str.extract(r"\bwidth: ([\d.]+)", expand=False).astype(float)12semantic = SemanticSearch()131415def states(size, unit):16 """Names that state this size, such as "7 qt" or "7-quart", but not "3.7 qt"."""17 return names.str.contains(rf"(?<![\d.]){size}[ -]?(?:{unit})")181920RIGHT_SIZE = { # the size each probe asks for: the width attribute, or the name21 # within half an inch of 36 or 48, so a 35.75-inch hood counts as 3622 "36 inch kitchen hood": (width - 36).abs() <= 0.5, "48 inch kitchen hood": (width - 48).abs() <= 0.5,23 "4 qt slow cooker": states(4, "qt|quart"), "7 qt slow cooker": states(7, "qt|quart"),24 # the whole word, which also matches 'twin over full' bunk beds25 "twin bed": names.str.contains(r"\btwin\b"), "king bed": names.str.contains(r"\bking\b"),26 "2 seat sofa": names.str.contains(r"\bloveseat\b"), # a two-seat sofa is a loveseat27}28DIGIT_ONLY = {"2 seat sofa": states(2, " in 1| piece"), "4 seat sofa": states(4, " in 1| piece")}2930for pair in ["hood", "cooker", "bed", "seats"]:31 first, second = probe_queries.index[probe_queries["pair"] == pair] # each pair is two probes, in file order32 tops = {row: semantic.search_vector(probe_vectors[row], k=10).index for row in (first, second)}33 print(f"{pair}, similarity of the two queries: {probe_vectors[first] @ probe_vectors[second]:.3f}")34 print(f"{pair}, products in both top 10s:", len(set(tops[first]) & set(tops[second])))35 for row, top in tops.items():36 query = probe_queries.at[row, "query"]37 if query in RIGHT_SIZE:38 print(f"{query}, right size in the top 10:", RIGHT_SIZE[query][top].sum())39 if query in DIGIT_ONLY:40 print(f"{query}, same digit but a piece count or 2 in 1:", DIGIT_ONLY[query][top].sum())41 print()42# the two counts the sofa probes ask for43print("catalog names that give a seat count:", (states(2, "seat") | states(4, "seat")).sum())44print("catalog products named loveseat:", names.str.contains(r"\bloveseat\b").sum())
python probe-pair-overlap.pyhood, similarity of the two queries: 0.795
hood, products in both top 10s: 0
36 inch kitchen hood, right size in the top 10: 8
48 inch kitchen hood, right size in the top 10: 8
cooker, similarity of the two queries: 0.841
cooker, products in both top 10s: 1
4 qt slow cooker, right size in the top 10: 6
7 qt slow cooker, right size in the top 10: 7
bed, similarity of the two queries: 0.697
bed, products in both top 10s: 0
twin bed, right size in the top 10: 10
king bed, right size in the top 10: 10
seats, similarity of the two queries: 0.759
seats, products in both top 10s: 1
2 seat sofa, right size in the top 10: 0
2 seat sofa, same digit but a piece count or 2 in 1: 4
4 seat sofa, same digit but a piece count or 2 in 1: 4
catalog names that give a seat count: 0
catalog products named loveseat: 14
For these probes, the model mostly finds the right size. The two hood probes share 0 products, yet each top 10 holds hoods of the right width, 8 for 36 inches and 8 for 48; the slow-cooker probes find 6 and 7 of the right capacity, and the bed probes 10 and 10 of the right size. Where the catalog has no number to match, the digit takes over. The number of product names that give a seat count is 0, so the top 10 for 2 seat sofa holds 4 products that share only the digit, two-piece sets and "2 in 1" kids' sofas, and 0 of the catalog's 14 loveseats.
코드는 비중이 작을 것으로 예상한다
상품 코드는 정확히 한 상품에만 일치해야 한다. 인코더에게 코드는 글자와 숫자로 이루어진 드문 문자열일 뿐이다. 탐침 질의 중 두 쌍이 코드를 시험한다. 한 쌍은 어떤 상품도 갖고 있지 않은 코드를 바탕으로 하며, 이 코드는 학습 질의 ines shag rug w003096809에서 가져왔다. 다른 한 쌍은 정확히 한 상품의 텍스트에만 들어 있는 화면 해상도를 바탕으로 한다.
1"""Compares the top 10s of two probe queries that differ only in a code no product holds, and shows where one product ranks for two resolutions, semantically and with BM25. The prose's point rests on the shared-top-10 line for the code and on the rank lines for the resolutions."""2from lib.lexical import LexicalSearch3from lib.recorded import probes4from lib.semantic import SemanticSearch5from lib.store import load_products, product_text67# Probe queries written for the book, embedded by the same model; no judgments, so nothing is scored.8probe_queries, probe_vectors = probes()9products = load_products()10text = product_text(products).str.lower()11semantic, bm25 = SemanticSearch(), LexicalSearch(products)1213# A code no product holds: does changing it change the results?14first, second = probe_queries.index[probe_queries["pair"] == "code"]15tops = [set(semantic.search_vector(probe_vectors[row], k=10).index) for row in (first, second)]16print("products whose text contains w003096809:", text.str.contains("w003096809").sum())17print(f"code, similarity of the two queries: {probe_vectors[first] @ probe_vectors[second]:.3f}")18print("code, products in both top 10s:", len(tops[0] & tops[1]))1920# A code one product holds: where does that product rank, with its code and with another?21holder = text.index[text.str.contains("1920x1080p")]22print()23print("products whose text contains 1920x1080p:", len(holder))24print("that product:", products.at[holder[0], "name"])25print("products in the Projectors class:", (products["product_class"] == "Projectors").sum())26for row in probe_queries.index[probe_queries["pair"] == "resolution"]:27 query = probe_queries.at[row, "query"]28 ranked = semantic.search_vector(probe_vectors[row], k=len(products)) # ranks every product, so the rank is exact29 lexical = list(bm25.search(query, k=100).index) # BM25 ranks are read from its top 100 only30 print(f"{query}, its semantic rank: {list(ranked.index).index(holder[0]) + 1}") # ranks count from 131 print(f"{query}, its similarity: {ranked[holder[0]]:.3f}")32 # the best product other than this one33 print(f"{query}, similarity of the next product: {ranked.drop(holder[0]).iloc[0]:.3f}")34 print(f"{query}, its bm25 rank:", lexical.index(holder[0]) + 1 if holder[0] in lexical else "not in the top 100")
python code-probes.pyproducts whose text contains w003096809: 0
code, similarity of the two queries: 0.967
code, products in both top 10s: 10
products whose text contains 1920x1080p: 1
that product: 5000 portable transmissive projector
products in the Projectors class: 4
1920x1080p projector, its semantic rank: 1
1920x1080p projector, its similarity: 0.562
1920x1080p projector, similarity of the next product: 0.496
1920x1080p projector, its bm25 rank: 1
2560x1440p projector, its semantic rank: 1
2560x1440p projector, its similarity: 0.509
2560x1440p projector, similarity of the next product: 0.508
2560x1440p projector, its bm25 rank: not in the top 100
알 수 없는 코드의 마지막 숫자들을 바꿔도 두 질의의 유사도는 0.967로 유지되고, 두 질의의 상위 10개에는 같은 상품이 10개 들어 있다. 어떤 것에도 일치하지 않는 코드는 아무것도 바꾸지 못하며, BM25도 일치시킬 대상이 없다.
한 상품만 갖고 있는 코드는 BM25의 순위를 결정하지만 임베딩의 순위는 약간 움직일 뿐이다. 해상도 1920x1080p는 5000 portable transmissive projector의 텍스트에만 나타난다. BM25는 1920x1080p projector 질의에서 이 상품을 1위로, 2560x1440p projector 질의에서는 not in the top 100위로 매긴다. Sciavolino 외(2021)는 희귀한 개체에 관한 질문에서 밀집(임베딩 기반) 검색기가 희소 방식보다 훨씬 나쁜 성능을 보인다고 보고했고, Karpukhin 외(2020)는 BM25가 선택성이 높은 키워드와 구에 민감하게 반응한다고 설명한다.
임베딩은 이 상품을 자신의 코드가 있을 때 1위, 다른 코드가 있을 때 1위로 매기는데, 이 상점에 프로젝터가 4개뿐이기 때문이다. 코드는 다음 상품과의 격차에서만 드러난다. 코드가 있을 때는 0.562과 0.496이고, 없을 때는 0.509과 0.508이다.
작업 규칙: 로그에서 크기, 수량, 코드가 들어 있는 질의를 골라 목록으로 만들고, 어느 한쪽 순위 모델을 신뢰하기 전에 두 모델을 모두 그 질의들에 대해 비교한다. 코드가 중요하면 어휘적으로 일치시킨다. 정확한 크기가 중요하면 attributes의 너비 같은 상품 필드에 대해 일치시키거나, 토크나이저가 숫자를 유지하고 단위가 한 가지 방식으로 표기된다면 어휘적으로 일치시킨다(5장, 8장). 질의의 나머지는 의미 기반 검색에 맡긴다. 7장에서는 두 방식을 결합하는 방법을 시험한다.
부정 표현에 대한 임베딩 시험
부정은 짧은 단어 하나로 쇼핑객이 원하는 바를 뒤집는다. 예를 들어 둥글지 않은 러그가 그렇다. 질의 전체를 임베딩하면 그 단어를 뺀 같은 질의와 가깝게 남을 수 있다. 부정 하나로 차이가 나는 문서 쌍으로 구성된 벤치마크 NevIR은 어떤 바이 인코더 아키텍처도 쌍 정확도(paired accuracy)가 12%를 넘지 못한다는 것을 보였다. 무작위로 순위를 매겨도 25%가 나오는 시험이다(Weller et al., 2024). 2025년의 재현 연구에서 van den Elsen et al.은 70억 파라미터 바이 인코더인 GritLM-7B 하나만 39.0%에 도달했고, 다른 바이 인코더는 거의 나아지지 않았으며, 두 가지 부정 과제를 모두 어느 정도 잘 처리한 것은 크로스 인코더와 리스트와이즈 LLM 재순위기뿐이라는 것을 발견했다.
이 스토어의 질의 중 "no", "not", "without", "non"을 포함하는 것은 0이므로, 판정 데이터로는 부정을 시험할 수 없다. 대신 탐침 질의 3쌍을 사용한다. 각 쌍은 질의 하나와 그 부정형으로 이루어지며, 각각의 상위 10개에 무엇이 들어 있는지 센다.
카탈로그는 부정형 탐침을 모두 만족시킬 수 있다. armless라는 이름의 제품이 42개, 유리를 언급하지 않는 커피 테이블이 62개, 이름에 round가 없는 면적 러그가 90개 있다.
실행하기 전에, rug that is not round에 대한 시맨틱 검색의 상위 10개 중 이름에 round가 들어 있는 것이 몇 개일지 추측해 보라.
1"""Counts what each negation probe's top 10 holds for semantic retrieval and for BM25, without scoring it. The prose's point rests on the semantic top 10 lines for the rug, coffee-table and sofa probes."""2import bm25s34from lib.lexical import LexicalSearch5from lib.recorded import probes6from lib.semantic import SemanticSearch7from lib.store import load_products, load_queries, product_text89# Probe queries written for the book, with no judgments: count what the top 10 holds, never score it.10probe_queries, probe_vectors = probes()11products = load_products()12names, text = products["name"].str.lower(), product_text(products).str.lower()13semantic, bm25 = SemanticSearch(), LexicalSearch(products)14# armless and round are matched in product names, glass in the full text15FEATURES = {"arms": ("armless", names), "glass": ("glass", text), "round": ("round", names)}1617negated = load_queries()["query"].str.contains(r"\b(?:no|not|without|non)\b")18print("store queries with no, not, without or non:", negated.sum())19# the tokenizer settings LexicalSearch uses, so these are the terms BM25 searches20tokens = bm25s.tokenize(["rug that is not round"], stopwords="en", return_ids=False, show_progress=False)21print("bm25 terms of rug that is not round:", " ".join(tokens[0]))22print("products named armless:", names.str.contains(r"\barmless\b").sum())23coffee_tables = products["product_class"] == "Coffee & Cocktail Tables"24area_rugs = products["product_class"] == "Area Rugs"25print("coffee tables that never mention glass:", (coffee_tables & ~text.str.contains(r"\bglass\b")).sum())26print("area rugs not named round:", (area_rugs & ~names.str.contains(r"\bround\b")).sum())2728negation = probe_queries[probe_queries["kind"] == "negation"]29print("negation probe pairs:", negation["pair"].nunique())30for row, probe in negation.iterrows():31 word, field = FEATURES[probe["pair"]]32 has_feature = field.str.contains(rf"\b{word}\b")33 semantic_top = semantic.search_vector(probe_vectors[row], k=10).index34 bm25_top = bm25.search(probe["query"], k=10).index35 print()36 print(f"{probe['query']}, semantic top 10 with {word}:", has_feature[semantic_top].sum())37 print(f"{probe['query']}, bm25 top 10 with {word}:", has_feature[bm25_top].sum())
python negation-probes.pystore queries with no, not, without or non: 0
bm25 terms of rug that is not round: rug round
products named armless: 42
coffee tables that never mention glass: 62
area rugs not named round: 90
negation probe pairs: 3
sofa with arms, semantic top 10 with armless: 1
sofa with arms, bm25 top 10 with armless: 0
sofa without arms, semantic top 10 with armless: 8
sofa without arms, bm25 top 10 with armless: 0
coffee table with glass top, semantic top 10 with glass: 10
coffee table with glass top, bm25 top 10 with glass: 9
coffee table not glass, semantic top 10 with glass: 10
coffee table not glass, bm25 top 10 with glass: 10
round rug, semantic top 10 with round: 10
round rug, bm25 top 10 with round: 10
rug that is not round, semantic top 10 with round: 10
rug that is not round, bm25 top 10 with round: 10
적다고 추측했다면, 쇼핑객처럼 "not"을 읽은 것이고 임베딩은 그렇게 읽지 않았다. 러그와 커피 테이블 탐침에서 임베딩은 속성을 유지하고 "not"을 놓쳤다. rug that is not round에 대한 시맨틱 검색의 상위 10개에는 이름에 round가 들어간 제품이 10개 있고, round rug의 상위 10개에는 10개 있다. coffee table not glass의 경우 상위 10개 중 10개가 glass를 언급하며, coffee table with glass top은 10개이다.
sofa without arms는 반대로 움직인다. 상위 10개 중 8개가 armless라는 이름이고, sofa with arms는 1개이다. 카탈로그에는 이 부정을 나타내는 단어가 armless 하나뿐이며, 모델은 "without arms"를 그 단어 근처에 놓는다.
BM25도 이 탐침에서 더 낫지 않다. 토크나이저가 "not", "that", "is"를 불용어로 제거하므로 rug that is not round를 rug round로 검색한다. 상위 10개에는 둥근 러그가 10개, armless 소파가 0개 들어 있다.
이 탐침은 측정이 아니라 몇 개의 쌍일 뿐이며, 소파 쌍은 반대로 움직인다. 그래도 러그와 커피 테이블 쌍은 NevIR과 일치한다. 질의 벡터는 질의가 지명한 속성 근처에 머문다.
작업 규칙: 어휘 기반이든 시맨틱이든 1단계 검색기가 부정을 처리하리라 기대하지 마라. 쇼핑객이 카탈로그에 저장된 속성, 예컨대 모양, 소재, 팔걸이를 제외하려 할 때는 그 제외를 감지해 해당 속성으로 필터링한다. 그렇지 않으면 재순위를 한다. van den Elsen et al.의 재현 연구에서 크로스 인코더와 LLM 재순위기가 부정을 훨씬 더 잘 처리했기 때문이다. 9장은 크로스 인코더와 LLM으로 스토어의 시맨틱 검색을 재순위하는데, 부정을 시험할 수 없는 스토어의 판정 질의에서는 어떤 재순위기도 시맨틱 검색을 앞서지 못한다. 이와 같은 탐침은 모델을 바꾸거나 재순위기를 추가할 때마다 다시 실행하는 점검으로 보관하라.
임베딩 모델 선택
모델은 벡터가 무엇을 알 수 있는지를 결정하며, 나중에 모델을 바꾸면 카탈로그 전체를 다시 임베딩해야 한다. 공개 벤치마크는 후보의 범위를 좁혀 준다. 최종 결정은 스토어 자체의 판단이 내린다.
벤치마크는 후보 목록으로 읽는다
MTEB(Massive Text Embedding Benchmark)는 여러 과제에서 모델의 점수를 매긴다. 이 벤치마크의 저자들은 모든 과제에서 우위를 차지하는 단일 방법이 없다는 점을 발견했다(Muennighoff et al., 2023). 또한 평균 점수가 데이터셋이 많은 과제, 특히 검색, 분류, 군집화 쪽으로 기울어 있다는 점도 발견했으므로, 자신이 다루는 과제의 점수를 읽어야 한다.
공개 점수는 모델이 보지 못한 데이터에서의 성능을 과대평가할 수도 있다. 검색 벤치마크인 RTEB의 제작자들은 공개 벤치마크에서 보고된 모델의 점수와 새로운 미공개 데이터에서의 실제 성능 사이에 차이가 있다고 설명한다(Liu et al., 2025). RTEB는 48개 데이터셋 가운데 28개를 비공개로 유지한다. Verma et al. (2026)은 RTEB의 모델 순위가 MTEB 및 BEIR(Thakur et al., 2021)의 순위와 ρ = 0.38의 상관만 보인다는 것을 발견했다. BEIR는 여러 도메인에 걸친 검색을 다루는 이전 벤치마크이다. 여기서 ρ는 순위 상관이며, 1은 순서가 같다는 뜻이다. 크기가 곧 품질인 것도 아니다. Enevoldsen et al. (2025)는 MTEB의 다국어 후속인 MMTEB에서 공개적으로 이용할 수 있는 모델 중 가장 성능이 좋은 것이 multilingual-e5-large-instruct이며, 이 모델의 매개변수가 5억 6천만 개에 불과하다고 보고한다.
상점은 네 개의 오픈 모델을 검토했으며, 그 모델 카드 중 어느 것도 WANDS에서의 결과를 보고하지 않는다. 그 모델은 상점의 인코더와 Qwen/Qwen3-Embedding-0.6B(카드)이다. 나머지 둘은 BAAI/bge-base-en-v1.5(카드)와 ibm-granite/granite-embedding-english-r2(카드)이다. ESCI와 같은 상품 검색 벤치마크는 질의–상품 쌍을 정확, 대체, 보완, 무관으로 분류하며 약 13만 개의 질의와 260만 개의 판정을 담고 있다(Reddy et al., 2022). 이런 벤치마크는 상점의 과제에 더 가깝지만, 그래도 상점의 카탈로그는 아니다. 그래서 상점의 후보 목록은 리더보드가 확인할 수 없는 기준에 근거한다. 상업적 사용을 허용하는 라이선스, 영어 지원, 특수한 하드웨어 없이 실행되는 크기, 더 짧은 벡터에 대한 문서화된 지원이 그것이다. 상점이 선택한 모델의 카드는 MTEB 검색 과제에서 nDCG@10 55.14를 보고하는데, 이는 상품이 아닌 일반 텍스트에 대한 제작사의 측정값이다(모델 카드). 후보 중에서 고르려면 각 후보를 같은 상품 텍스트와 질의에서 기록하고, 학습 질의에서 각각 점수를 매긴 다음, 이 책의 쌍체 검정으로 비교한다. 다음 소절이 하나의 모델을 여러 크기에서 비교하며 그렇게 한다.
이 부분은 아직 번역되지 않았으므로 영어로 표시된다.
Truncate dimensions, then test the loss
Every dimension costs 4 bytes per product as a 32-bit float, and every similarity multiplies all of them, so a shorter vector is cheaper to store and to search. Matryoshka training teaches a model to put the most useful information in its first dimensions, so that the start of a vector works as a smaller embedding on its own. Kusupati et al. (2022) report embeddings up to 14 times smaller for ImageNet-1K classification at the same level of accuracy. Shortening saves storage and search, not encoding: a Matryoshka model is no faster to run, as Sentence Transformers' documentation notes. Only models trained this way are built to survive it, and a shortened vector isn't unit length, so normalize it again, as the model card's own example does.
The store's encoder has a model card that documents shortening to 256 dimensions. lib.recorded.embeddings(dims=...) keeps the first dims numbers of each vector and normalizes again. Score each length on the train queries:
1"""Scores semantic retrieval on the train queries at 768, 256 and 128 dimensions, with each product's float32 size. The prose's point rests on the nDCG@10 line for each length, set against 768 dimensions."""2from lib.metrics import evaluate3from lib.semantic import SemanticSearch4from lib.store import load_queries, qrels56train, judgments = load_queries("train"), qrels("train") # choose on train queries78# dims keeps the first dims numbers of every vector and rescales each to length 1 (lib.recorded).9for dims in [768, 256, 128]:10 semantic = SemanticSearch(dims=dims)11 scores = evaluate(semantic.run(train, k=100), judgments)12 # semantic.products holds the product vectors, not the product table13 print(f"{dims} dimensions, float32 bytes per product: {semantic.products[0].nbytes:,}")14 print(f"{dims} dimensions, ndcg@10: {scores['ndcg@10'].mean():.3f}")15 print(f"{dims} dimensions, judged@10: {scores['judged@10'].mean():.3f}")16 print()
python truncate-embeddings.py768 dimensions, float32 bytes per product: 3,072
768 dimensions, ndcg@10: 0.772
768 dimensions, judged@10: 0.964
256 dimensions, float32 bytes per product: 1,024
256 dimensions, ndcg@10: 0.762
256 dimensions, judged@10: 0.952
128 dimensions, float32 bytes per product: 512
128 dimensions, ndcg@10: 0.680
128 dimensions, judged@10: 0.874
Shortening to 256 dimensions cuts a product from 3,072 bytes to 1,024, and moves mean nDCG@10 from 0.772 to 0.762.
At 128 dimensions the mean is 0.680, with judged@10 at 0.874 against 0.964, so part of that fall could come from products nobody judged.
Is the change real, or noise among 50 queries? Test it with lib.significance.compare, which runs chapter 3's permutation test and its paired t-test. It also reports a bootstrap interval. These comparisons use train queries, so they tune the vector size; they aren't evidence about the test queries:
1"""Tests the change in nDCG@10 from shortening to 256 and 128 dimensions against the full vectors, on the train queries, with a permutation test, a paired t-test and a bootstrap interval. The prose's point rests on the permutation, interval and effect-size lines for each length."""2from lib.metrics import evaluate3from lib.semantic import SemanticSearch4from lib.significance import compare5from lib.store import load_queries, qrels67# Train queries: these comparisons tune the vector size. They are not evidence about the test queries.8train, judgments = load_queries("train"), qrels("train")91011def ndcg(dims):12 return evaluate(SemanticSearch(dims=dims).run(train, k=100), judgments)["ndcg@10"]131415def p_value(p):16 return "below 0.001" if p < 0.001 else f"{p:.3f}"171819full = ndcg(768)20for dims in [256, 128]:21 # compare's default interval is 95%22 result = compare(ndcg(dims), full) # paired by train query: shortened minus full23 print(f"{dims} vs 768, mean difference in ndcg@10: {result['mean difference']:.3f}")24 print(f"{dims} vs 768, queries worse: {result['losses']}")25 print(f"{dims} vs 768, permutation test p: {p_value(result['permutation p'])}")26 print(f"{dims} vs 768, paired t-test p: {p_value(result['t-test p'])}")27 print(f"{dims} vs 768, 95% bootstrap interval: {result['ci low']:.4f} to {result['ci high']:.4f}")28 print(f"{dims} vs 768, interval excludes 0: {'yes' if result['ci high'] < 0 or result['ci low'] > 0 else 'no'}")29 print(f"{dims} vs 768, effect size: {result['effect size']:.3f}")30 print()
python truncation-paired-test.py256 vs 768, mean difference in ndcg@10: -0.010
256 vs 768, queries worse: 24
256 vs 768, permutation test p: 0.076
256 vs 768, paired t-test p: 0.080
256 vs 768, 95% bootstrap interval: -0.0211 to -0.0002
256 vs 768, interval excludes 0: yes
256 vs 768, effect size: -0.253
128 vs 768, mean difference in ndcg@10: -0.092
128 vs 768, queries worse: 33
128 vs 768, permutation test p: below 0.001
128 vs 768, paired t-test p: below 0.001
128 vs 768, 95% bootstrap interval: -0.1569 to -0.0565
128 vs 768, interval excludes 0: yes
128 vs 768, effect size: -0.547
At 128 dimensions the loss is clear. The mean difference is -0.092, and 33 train queries get worse. The permutation test gives p below 0.001, and the 95% interval, from -0.1569 to -0.0565, excludes 0.
At 256 dimensions the evidence is borderline, because the tests and the interval disagree. The mean difference is -0.010; the permutation test gives p = 0.076 and the t-test p = 0.080, yet the interval, from -0.0211 to -0.0002, excludes 0 by a hair. The effect size, -0.253 at 256 dimensions against -0.547 at 128, says any loss at 256 is the smaller of the two.
For the store, the decision follows from its size, not from the test. Its full vectors take 36.9 MB, so there is little to save; it keeps all 768 dimensions, and its test score doesn't change. A catalog of hundreds of millions of products would weigh a cut from 3,072 to 1,024 bytes per product against a loss that this many train queries can't pin down, and would judge more queries before it decided.
Working rule: Shortlist models on licence, language, size and domain, and treat benchmark ranks as a hint. Decide between models, or between vector sizes, on your own train queries with a paired test, and read the interval as well as the p-value. Shorten vectors only for a model trained for it, normalize after shortening, and only when memory or search time requires it.
운영 환경에서 무엇이 잘못되는가
유사도는 신뢰도가 아니다. 가장 높은 유사도가 낮으면 아무것도 보여 주지 않고 싶어질 수 있다. 그러나 유사도는 질의마다 같은 척도에 놓여 있지 않다.
1"""Finds the test query with the lowest best similarity and the one with the highest 10th similarity, and counts the products above a 0.5 cutoff. The prose's point rests on the two similarity lines and the cutoff counts."""2import numpy as np34from lib.recorded import embeddings5from lib.semantic import SemanticSearch6from lib.store import load_queries, qrels78product_ids, product_vectors, query_ids, query_vectors = embeddings()9test = load_queries("test")10rows = np.isin(query_ids, test.index)11similarity = np.sort(query_vectors[rows] @ product_vectors.T, axis=1)[:, ::-1] # best first, per query12queries = test.loc[query_ids[rows], "query"].to_numpy()1314weakest = similarity[:, 0].argmin() # the query whose best match is least similar15strongest = similarity[:, 9].argmax() # the query whose 10th match is most similar16print("first query, lowest best similarity:", queries[weakest])17print(f"its best similarity: {similarity[weakest, 0]:.3f}")18judgments = qrels()[query_ids[rows][weakest]]19top = SemanticSearch().search(query_ids[rows][weakest], k=10).index20print("exact products in its top 10:", sum(judgments.get(product_id) == 2 for product_id in top))21print("second query, highest 10th similarity:", queries[strongest])22print(f"its 10th similarity: {similarity[strongest, 9]:.3f}")2324cutoff = 0.5 # an illustrative cutoff; nothing about the model sets it25print()26print(f"products above {cutoff} for the first query:", (similarity[weakest] > cutoff).sum())27print(f"products above {cutoff} for the second query:", (similarity[strongest] > cutoff).sum())28print(f"test queries with no product above {cutoff}:", (similarity[:, 0] <= cutoff).sum())
python similarity-scale.pyfirst query, lowest best similarity: liberty hardware francisco
its best similarity: 0.436
exact products in its top 10: 5
second query, highest 10th similarity: kohler whitehaven farmhouse kitchen sink
its 10th similarity: 0.647
products above 0.5 for the first query: 0
products above 0.5 for the second query: 103
test queries with no product above 0.5: 2
테스트 질의 중 양쪽 끝의 두 질의가 이를 보여 준다. liberty hardware francisco는 최고 일치 유사도가 0.436로 가장 약하지만, 상위 10개에는 정확히 일치하는 제품이 5개 들어 있다. 반면 kohler whitehaven farmhouse kitchen sink는 10번째 일치의 유사도가 0.647로 가장 높다.
0.5를 기준값으로 삼으면 첫 번째 질의에서는 제품 0개가 남고 두 번째 질의에서는 103개가 남으며, 테스트 질의 2개에서는 아무것도 남지 않는다. 또한 의미 기반 검색은 결과가 비어 돌아오는 법이 없다. 어떤 제품도 갖고 있지 않은 코드를 질의해도 상위 10개가 가득 채워져 나온다. 질의에 일치하는 제품이 없다는 판단은 다른 방법으로 내려야 한다. 예를 들어 어휘 기반 검색이 그 질의의 드문 용어를 하나도 찾지 못했을 때 그렇게 판단할 수 있다.
일치하지 않는 벡터는 조용히 실패한다. 질의 벡터와 제품 벡터는 같은 모델, 같은 리비전, 같은 프롬프트에서 나와야 한다. 서로 다른 두 모델의 벡터는 서로 다른 공간의 숫자이며, 두 모델의 차원이 같으면 비교해도 오류가 발생하지 않는다. Sentence Transformers의 encode_query는 모델에 미리 정의된 질의 프롬프트가 있으면 이를 추가한다(API 레퍼런스). 실시간 질의를 임베딩하는 코드가 색인 코드와 같은 일을 하는지 확인해야 한다.
색인은 카탈로그를 저절로 따라가지 않는다. faiss 위키에 따르면 faiss의 HNSW는 벡터 삭제를 지원하지 않으므로, 단종된 제품은 결과에서 걸러 내거나 색인을 다시 만들 때까지 그래프에 남는다. Douze 외(2024)는 faiss가 데이터베이스가 아니며 동시 쓰기, 샤딩, 트랜잭션이 없다고 말한다. 정확 검색 대비 재현율은 하나의 카탈로그와 하나의 설정에 속하는 값이므로, 다시 만들 때마다 재측정해야 한다.
긴 텍스트는 잘린다. 모델은 최대 512개의 토큰까지만 읽으며, Sentence Transformers의 API 레퍼런스는 더 긴 입력이 잘린다고 설명한다. 이 상점에서 가장 긴 제품 텍스트는 201개 토큰이지만, 설명이 긴 카탈로그에서는 한계를 넘는 부분을 모델이 전혀 읽지 못한다. recordings/embeddings.py처럼 임베딩하기 전에 토큰 수를 세어야 한다.
오타는 벡터를 움직인다. Zhuang과 Zuccon(2021)은 오타가 포함된 데이터로 학습하지 않은 BERT 기반 검색기에서 오타가 검색과 순위 효과를 크게 떨어뜨릴 수 있음을 발견했다. 이 상점의 질의 중 4개에 오타가 있고, 각각에 철자가 올바른 탐침이 있다. 교정된 탐침은 원래 질의와 같은 것을 요청하므로, 그 질의의 판정으로 점수를 매길 수 있다.
1"""Scores each correctly spelled probe with the judgments of the query it corrects, for semantic retrieval and BM25, as typed and as corrected. The prose's point rests on the semantic nDCG@10 lines for each spelling and on the smallest-p line."""2from lib.lexical import LexicalSearch3from lib.metrics import ndcg_at_k4from lib.recorded import probes5from lib.semantic import SemanticSearch6from lib.store import load_products, load_queries, qrels78# Each spelling probe is the correctly spelled text of one store query (of_query_id), embedded by the same9# model. It asks for the same thing, so it is scored with that query's judgments. A diagnosis, not a test.10probe_queries, probe_vectors = probes()11products, queries, judgments = load_products(), load_queries(), qrels()12semantic, bm25 = SemanticSearch(), LexicalSearch(products)1314spelling = probe_queries[probe_queries["kind"] == "spelling"]15for row, probe in spelling.iterrows():16 query_id, pair = int(probe["of_query_id"]), probe["pair"]17 typed, corrected = semantic.search(query_id, k=10).index, semantic.search_vector(probe_vectors[row], k=10).index18 print(f"{pair}: {queries.at[query_id, 'query']} | {probe['query']} ({queries.at[query_id, 'split']} query)")19 print(f"{pair}, similarity of the two spellings: {semantic.query_vectors[query_id] @ probe_vectors[row]:.3f}")20 print(f"{pair}, products in both semantic top 10s:", len(set(typed) & set(corrected)))21 print(f"{pair}, semantic ndcg@10 as typed: {ndcg_at_k(list(typed), judgments[query_id], 10):.3f}")22 print(f"{pair}, semantic ndcg@10 corrected: {ndcg_at_k(list(corrected), judgments[query_id], 10):.3f}")23 for name, text in [("as typed", queries.at[query_id, "query"]), ("corrected", probe["query"])]:24 print(f"{pair}, bm25 ndcg@10 {name}: {ndcg_at_k(list(bm25.search(text, k=10).index), judgments[query_id], 10):.3f}")25 print()26print("spelling probes:", len(spelling))27# a sign-flip test on n queries has 2 ** n outcomes, and its two most extreme give the smallest two-sided p28print(f"smallest two-sided permutation p-value possible with {len(spelling)} queries: {2 / 2 ** len(spelling):.3f}")
python spelling-probes.pyligth: ligth bulb | light bulb (train query)
ligth, similarity of the two spellings: 0.710
ligth, products in both semantic top 10s: 5
ligth, semantic ndcg@10 as typed: 1.000
ligth, semantic ndcg@10 corrected: 1.000
ligth, bm25 ndcg@10 as typed: 0.945
ligth, bm25 ndcg@10 corrected: 0.945
foutains: foutains with brick look | fountains with brick look (train query)
foutains, similarity of the two spellings: 0.668
foutains, products in both semantic top 10s: 0
foutains, semantic ndcg@10 as typed: 0.353
foutains, semantic ndcg@10 corrected: 0.645
foutains, bm25 ndcg@10 as typed: 0.356
foutains, bm25 ndcg@10 corrected: 0.507
desins: twin over full bunk beds cool desins | twin over full bunk beds cool designs (train query)
desins, similarity of the two spellings: 0.829
desins, products in both semantic top 10s: 6
desins, semantic ndcg@10 as typed: 0.565
desins, semantic ndcg@10 corrected: 0.616
desins, bm25 ndcg@10 as typed: 0.488
desins, bm25 ndcg@10 corrected: 0.488
ice-an: refrigerator with ice an water in door | refrigerator with ice and water in door (test query)
ice-an, similarity of the two spellings: 0.975
ice-an, products in both semantic top 10s: 9
ice-an, semantic ndcg@10 as typed: 0.807
ice-an, semantic ndcg@10 corrected: 0.829
ice-an, bm25 ndcg@10 as typed: 0.501
ice-an, bm25 ndcg@10 corrected: 0.501
spelling probes: 4
smallest two-sided permutation p-value possible with 4 queries: 0.125
오타 하나가 질의의 의미 기반 상위 10개 전체를 바꿀 수 있다. 학습 질의 foutains with brick look의 경우, 입력한 철자와 교정한 철자가 공유하는 제품은 0개이고 nDCG@10은 0.353에서 0.645로 바뀐다. BM25도 0.356에서 0.507로 움직인다.
나머지 탐침에서는 점수가 덜 움직인다. 그중 유일한 테스트 질의인 refrigerator with ice an water in door는 0.807에서 0.829로 바뀐다. 이 결과 중 어느 것도 일정한 양상을 입증하지는 못한다. 탐침이 4개일 때 가능한 가장 작은 양측 순열 검정 p값은 0.125이다.
작업 규칙: 고정된 유사도 값으로 걸러 내지 않는다. 모든 벡터와 함께 모델, 리비전, 프롬프트를 기록하고, 질의를 임베딩하는 곳에서 이를 확인한다. 카탈로그가 바뀌면 색인을 다시 만들고 다시 측정하며, 임베딩하기 전에 토큰 수를 센다. 로그에 오타가 보이면 질의를 임베딩하기 전에, 각 단어의 이웃 단어를 확인하는 교정기로 철자를 교정한다(8장).
이 장에서 내린 결정
훈련 질의로 선택을 측정할 수 있는 곳에서는 측정했다.
- 모델:
Snowflake/snowflake-arctic-embed-m-v1.5이다.recordings/embeddings.py가 고정된 리비전에서 한 번 기록한다. 이 모델은 공개되어 있고 Apache-2.0 라이선스를 따르며, 크기가 작고 더 짧은 벡터를 위해 학습되었다. 질의에는 모델 카드가 요구하는 접두사를 붙이고, 제품에는 붙이지 않는다. - 유사도: 단위 길이 벡터의 내적이며, 이는 코사인 유사도이다. 벡터는 한 번 정규화하고, 변경이 있을 때마다 다시 정규화한다.
- 검색: NumPy를 사용하는 정확 검색이며
lib.vectors.exact_search을 통해 수행한다. 저장소의 벡터가 차지하는 크기가 36.9 MB이기 때문이다. 정확 검색은 모든 인덱스의 기준 정답이기도 하다. - 카탈로그가 정확 검색으로 감당할 수 있는 규모를 넘어설 경우:
M = 32와efSearch값 144을 쓰는 faiss HNSW를 사용한다. 이 값은 훈련 스윕에서 모든 질의가 실제 상위 10개 중 최소 9개를 유지하는 가장 작은 설정이다. 테스트 질의에서 정확 검색에 대한 이 인덱스의 recall@10은 1.000이다. - 인덱스의 재현율은 정확 검색을 기준으로 측정한다. 훈련 질의로, 파이프라인이 사용하는 k에서 측정하며, 평균뿐 아니라 가장 나쁜 질의도 함께 살펴본다.
- 차원: 768을 모두 사용한다. 256으로 줄이면 제품당 3,072바이트에서 1,024바이트로 줄어들지만, 그에 따른 손실은 훈련 질의로 확정할 수 없으며, 저장소에는 절약할 여지가 적다.
- 탐침 질의는 판정으로 검증할 수 없는 것을 검사한다. 부정, 코드, 철자 같은 것이 그 예이다. 탐침 질의는 판정된 질의와 별도로 상위 10개 결과를 비교한다. 같은 것을 묻는 교정된 철자만 해당 질의의 판정으로 채점한다.
- 알려진 약점은 다른 단계로 넘긴다. 코드는 어휘 검색으로, 정확한 크기는 속성 필드로 넘기며, 토크나이저가 숫자를 유지하게 되면 어휘 검색으로 넘긴다(5장, 8장). 둘 모두 결합이 아니라 시험해 볼 필터로 다룬다(7장). 부정은 속성 필터와 재순위화로 넘긴다.
- 이 장이 보여 준 것: 인덱스의 품질은 실제로 배포하는 카탈로그와 설정에서 정확 검색에 대해 측정한 재현율만큼만 보장된다. 탐침 질의 결과, 임베딩은 대부분의 크기를 처리했지만 코드와, 러그 및 커피 테이블 쌍의 부정은 처리하지 못했으며, 몇몇 쌍은 아무것도 입증하지 못한다.
이 판의 이 장
2026.11.10판예제 통과 2026년 11월 19이 장에서 16개 예제개 중 16개가 통과한다
영어판 2026.11.10판을 번역한 것이다.