セマンティック検索

読了まで約75 分分、13 節 節の 16 例 例。

第2026.11.10版 2026年11月19日時点で例が通過 132 例中 132 例が合格 変更点

AIによる翻訳。英語版は人による検証済みである。 当社の書籍の作り方

英語版の書籍(版2026.11.10)から翻訳した。 原文を読む

この章を終えると、保存済みの埋め込みから、カタログ向けのセマンティック検索を構築できるようになる。コサイン類似度でベクトルを比較し、NumPy でクエリに最も近い商品を厳密に求め、カタログが厳密検索の手に負えない規模に育った場合は近似インデックスに切り替え、そのインデックスが取りこぼすものを測定し、さらに埋め込みが誤るクエリを見つける。また、自分の判定データでテストしてベクトルの次元数を選び、同じ方法でモデルを比較することもできる。

48 inch kitchen hood に対して、セマンティック検索は他の幅のフードを返し、rug that is not round に対しては丸いラグを返す。それでもテスト用クエリでは、平均 nDCG@10 は 0.830 であった。BM25 の 0.725 に対するこの優位は、第3章で確立されるものである。本書はこれを一つのクラス lib.semantic.SemanticSearch として使ってきた。第1章では、ソファからシャワーカーテンまでを扱うストアのカタログ向けに、あるモデルが事前に一度だけ計算したベクトルの上にこれを構築した。ストアがこれに依存する前に、次の三点を知っておく必要がある。それらのベクトルが何を保持しているか、はるかに大きなカタログでそれらを検索すると何がかかるか、そしてどのクエリで誤るか、である。この章では厳密検索を手作業で構築し、近似インデックスをそれと比較して測定し、両者をストアのクエリと判定データでテストする。

テスト用クエリについては、この章で 2 件の比較を計画しており、いずれもインデックスのテストを扱う節で行う。各比較では、クエリをすべての商品と比較する厳密検索と、その一部とだけ比較する近似インデックスである HNSW を対照する。HNSW は faiss の既定設定のものと、訓練用クエリで選んだ設定のものとの二通りを、それぞれ厳密検索と比較する。各比較には本書の規則が課され、ここでは p 値が 0.025 を下回ることが条件となる。この章でテスト用クエリを見るそれ以外の場面は、すべて探索的なものである。

この部分はまだ翻訳されていないため、英語で表示している。

Turn text into embeddings

最終変更:版2026.11.10

An embedding places texts with similar meanings near each other; this section opens how. The model that computes an embedding is an encoder, trained so that texts with similar meanings get vectors that point in similar directions, whether or not they share words. Every text gets the same count of numbers, the embedding's dimension, whether it's a three-word query or a product's whole description. One number means nothing on its own; only comparisons between whole vectors do.

Search uses the encoder as a bi-encoder: it encodes each product by itself, once, ahead of time, and each query by itself when the shopper types it. Comparing a query with a product is then arithmetic on two vectors rather than a run of the model, which is what makes it cheap enough to compare one query with a whole catalog. Chapter 1 gives the size of the difference from the Sentence-BERT paper: about 65 hours for BERT to read every pair of 10,000 sentences, against about 5 seconds to embed each sentence once and compare. A model that reads a query and a product together is a cross-encoder, and chapter 1 uses one to rerank a shortlist.

The store's encoder is Snowflake/snowflake-arctic-embed-m-v1.5, an open model with 109 million parameters under the Apache-2.0 licence (model card). Merrick et al. (2024) report that its family's training kept only documents identified as mainly English. We chose it because anyone may use it commercially, it's small, and it was trained so that its vectors can be shortened, which the section on dimensions tests. Examples read a recording instead: vectors that the script recordings/embeddings.py computed once, at a pinned revision, and saved at unit length, so every example sees the same numbers. The model card asks for a prefix on queries only, so the script adds it to each query and embeds products as they are. Examples read the vectors through lib.recorded.embeddings():

inspect-recorded-embeddings.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 合格
"""Loads the recorded vectors and shows the lengths lib.recorded stores and rescales, and how close two shopper queries are. The prose's point about rounding rests on the stored- and rescaled-length lines, and its point about nearby queries on the median and nearest-query lines."""import reimport numpy as npfrom lib.recorded import RECORDED, embeddings, manifestfrom lib.store import load_queries# unit-length rows, one per ID; the queries carry the model's query prefixproduct_ids, product_vectors, query_ids, query_vectors = embeddings()recording = manifest("embeddings")print("embedding model:", recording["model"])print("query prompt:", recording["prompts"]["query"])print(f"products embedded: {len(product_ids):,}")print("dimensions:", product_vectors.shape[1])print("stored as:", recording["dtype"])print("longest product text, tokens:", recording["longest_product_tokens"])print("model's input limit, tokens:", recording["max_seq_length"])# row 0 is product 1; [0, :4] takes its first four numbersprint(f"product {product_ids[0]}, first four numbers:", ", ".join(f"{x:.3f}" for x in product_vectors[0, :4]))stored = np.load(RECORDED / "embeddings" / "products.npy").astype(np.float32)   # as saved, before lib.recordedstored_lengths, lengths = np.linalg.norm(stored, axis=1), np.linalg.norm(product_vectors, axis=1)print(f"stored vector lengths: {stored_lengths.min():.5f} to {stored_lengths.max():.5f}")print(f"lengths after lib.recorded rescales: {lengths.min():.5f} to {lengths.max():.5f}")# How close are two shopper queries? Compare every query with every other one.queries = load_queries().loc[query_ids, "query"].tolist()similarity = query_vectors @ query_vectors.T  # unit length, so the dot product is the cosinenp.fill_diagonal(similarity, np.nan)               # leave out each query's similarity to itselfprint(f"median similarity of two queries: {np.nanmedian(similarity):.3f}")for query in ["whimsical solar lights", "wall design shelf"]:    row = queries.index(query)    nearest = int(np.nanargmax(similarity[row]))    # whole words as typed, with no stemming, so `lights` and `light` differ    shared = set(re.findall(r"\w+", query)) & set(re.findall(r"\w+", queries[nearest]))    print()    print(f"{query}, nearest other query:", queries[nearest])    print(f"{query}, similarity: {similarity[row, nearest]:.3f}")    print(f"{query}, words in common:", len(shared))
出力python inspect-recorded-embeddings.py
embedding model: Snowflake/snowflake-arctic-embed-m-v1.5
query prompt: Represent this sentence for searching relevant passages:
products embedded: 12,000
dimensions: 768
stored as: float16
longest product text, tokens: 201
model's input limit, tokens: 512
product 1, first four numbers: -0.033, 0.051, 0.111, -0.012
stored vector lengths: 0.99994 to 1.00006
lengths after lib.recorded rescales: 1.00000 to 1.00000
median similarity of two queries: 0.291

whimsical solar lights, nearest other query: outdoor light fixtures
whimsical solar lights, similarity: 0.637
whimsical solar lights, words in common: 0

wall design shelf, nearest other query: kitchen storage shelves
wall design shelf, similarity: 0.679
wall design shelf, words in common: 0

Each of the 12,000 products is a vector of 768 numbers; product 1's begins -0.033, 0.051, 0.111, -0.012, and none of those numbers means anything alone. The recording stores each number as a float16, in 2 bytes, and lib.recorded widens them to 32-bit floats when it loads them. That rounding moves the stored lengths slightly off 1: they run from 0.99994 to 1.00006. So lib.recorded also scales each vector back to length 1, and the lengths it returns run from 1.00000 to 1.00000. The store's encoder reads text as tokens, word pieces from its own vocabulary, not the terms of chapter 5's tokenizer. The longest product text is 201 tokens, inside the model's limit of 512 tokens, so the model read every product in full.

The median similarity between two of the store's queries is 0.291, on a scale where 1 means two vectors point the same way (cosine similarity). Yet whimsical solar lights finds outdoor light fixtures at 0.637, and wall design shelf finds kitchen storage shelves at 0.679, with 0 and 0 words in common: lights and light are different strings, and so are shelves and shelf. The baseline BM25 doesn't stem words, so it sees nothing shared in either pair. Chapter 5's stemming would match lights with light, but not shelves with shelf.

Working rule: Embed products ahead of time and queries at search time, with the same model at the same revision, and follow the model card on prompts: here, a prefix on queries only. Record the model, revision and prompts next to the vectors, as each manifest.json in examples/data/recorded/ does. Check that your longest product fits the model's input limit before you trust its embedding.

この部分はまだ翻訳されていないため、英語で表示している。

Compare embeddings with cosine similarity

最終変更:版2026.11.10

To rank products for a query, you need one number per pair that says how alike two vectors are. Cosine similarity is the cosine of the angle between two vectors: 1 when they point the same way, 0 at right angles and −1 when they point in opposite directions. It ignores how long the vectors are. The dot product multiplies two vectors number by number and adds up the results, so it equals the cosine times both lengths. Scaling every vector to length 1, called normalizing, makes the two the same number, as Sentence Transformers states. The dot product is then the cheaper choice, because it skips computing the lengths.

dot(q, d)    = q1 × d1 + q2 × d2 + ... + qn × dn
length(q)    = √(q1² + q2² + ... + qn²)
cosine(q, d) = dot(q, d) / (length(q) × length(d))

To see the formula at work, take the test query clear shower curtain and its most similar product: compute the dot product and the cosine by hand, and check the cosine with scikit-learn 1.9.1. Then break the one assumption that makes them agree. Keep the first 256 of each vector's 768 numbers, a shortcut that the section on dimensions tests, but skip the rescaling, and rank every query's products both ways:

cosine-by-hand.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 · scikit-learn 1.9.1 · scipy 1.18.1 合格
"""Computes the cosine of one test query and its best product by hand and with scikit-learn, then ranks every query's products by dot product and by cosine after the vectors are shortened. The prose's point rests on the line that counts queries whose top 10 changes."""import numpy as npfrom sklearn.metrics.pairwise import cosine_similarityfrom lib.recorded import embeddingsfrom lib.store import load_productsproducts = load_products()product_ids, product_vectors, query_ids, query_vectors = embeddings()query = query_vectors[list(query_ids).index(246)]           # clear shower curtainbest = int(np.argmax(product_vectors @ query))product = product_vectors[best]print("product:", products.at[product_ids[best], "name"])# cosine = dot product / (length of q * length of d)dot = query @ productcosine = dot / (np.linalg.norm(query) * np.linalg.norm(product))print(f"dot product: {dot:.3f}")print(f"cosine by hand: {cosine:.3f}")print(f"cosine from scikit-learn: {cosine_similarity([query], [product])[0, 0]:.3f}")# Keep the first 256 numbers of each vector and don't rescale: the lengths are no longer 1.short_products, short_queries = product_vectors[:, :256], query_vectors[:, :256]lengths = np.linalg.norm(short_products, axis=1)print(f"shortened product lengths: {lengths.min():.3f} to {lengths.max():.3f}")# stable sort: tied scores keep their stored orderby_dot = np.argsort(-(short_queries @ short_products.T), axis=1, kind="stable")[:, :10]# only the products are rescaled: a query's length scales all its scores alike, so it can't change their orderby_cosine = np.argsort(-(short_queries @ (short_products / lengths[:, None]).T), axis=1, kind="stable")[:, :10]# compares the sets: a new order within the same 10 doesn't countchanged = sum(set(dot_top) != set(cosine_top) for dot_top, cosine_top in zip(by_dot, by_cosine))print("queries:", len(query_ids))print("queries whose top 10 changes:", changed)
出力python cosine-by-hand.py
product: woolley vinyl solid color single shower curtain
dot product: 0.577
cosine by hand: 0.577
cosine from scikit-learn: 0.577
shortened product lengths: 0.706 to 0.778
queries: 80
queries whose top 10 changes: 61

On the store's unit-length vectors, the dot product is 0.577, the cosine by hand is 0.577, and scikit-learn's cosine_similarity gives 0.577, so the three agree.

Shortening breaks that. The shortened vectors have lengths from 0.706 to 0.778, and for 61 of the 80 queries, ranking by dot product gives a different top 10 from ranking by cosine. A query's own length scales all of its scores by the same factor, so it can't change their order. A product's length can: it multiplies that product's score for every query, a boost the shopper never asked for.

Some libraries report a distance instead of a similarity. faiss's L2 indexes return the squared Euclidean distance, and its inner product equals cosine similarity only when the vectors are normalized, according to the faiss wiki. For unit vectors, the squared distance is 2 − 2 × cosine, so it ranks products in the same order as cosine, smallest distance first.

Working rule: Normalize every vector once, when you store it, and rank by dot product. Normalize again after anything that changes a vector, such as shortening it or averaging several. If a model's documentation says it was trained for the dot product of unnormalized vectors, follow the documentation instead, and never mix the two in one index.

この部分はまだ翻訳されていないため、英語で表示している。

Find the exact nearest neighbours with NumPy

最終変更:版2026.11.9

Retrieval by embeddings is a nearest-neighbour search: find the k products whose vectors are most similar to the query's. Exact search compares the query with every product and keeps the best k, so it never misses one. With unit-length vectors, comparing one query with every product is one matrix–vector product, and comparing many queries is one matrix product, with a row per query and a column per product. What's left is to find the k highest scores in each row. Sorting every row does more work than needed. NumPy's argpartition moves the k highest scores to one side without ordering either side, and NumPy's documentation leaves that order undefined, so you sort only those k.

Chapter 1's semantic search took its rankings from lib.semantic.SemanticSearch. Build its core by hand for the 30 test queries with NumPy 2.5.5, then check it twice: against lib.semantic, and against faiss's IndexFlatIP, which computes the inner product with every vector. The faiss guidelines name two indexes that guarantee exact results, IndexFlatIP and IndexFlatL2:

exact-search-numpy.py Python 3.12.15 · faiss-cpu 1.16.0 · numpy 2.5.5 · pandas 3.1.1 合格
"""Builds exact search from one matrix product, then checks its top 10 against lib.semantic and faiss's IndexFlatIP. The prose's point rests on the two 'same top 10' lines, which show the hand-built search matches both."""import faissimport numpy as npfrom lib.metrics import evaluatefrom lib.recorded import embeddingsfrom lib.semantic import SemanticSearchfrom lib.store import load_queries, qrelsproduct_ids, product_vectors, query_ids, query_vectors = embeddings()test = load_queries("test").indexrows = np.isin(query_ids, test)def top_k(scores, k):    """Column indices of the k highest scores in each row, best first."""    # argpartition's kth is 0-based: with k - 1, the first k positions hold the k highest scores, in no order    candidates = np.argpartition(-scores, k - 1, axis=1)[:, :k]          # the top k, in no order    # stable sort: tied scores keep their candidate order    order = np.argsort(-np.take_along_axis(scores, candidates, axis=1), axis=1, kind="stable")    return np.take_along_axis(candidates, order, axis=1)scores = query_vectors[rows] @ product_vectors.T      # one row per test query, one column per producttop = top_k(scores, 10)print("similarities computed:", f"{scores.size:,}")print("similarities per query:", f"{scores.shape[1]:,}")semantic = SemanticSearch()# two lists match only if the same products come in the same ordersame = sum(list(product_ids[found]) == list(semantic.search(query_id, 10).index)           for query_id, found in zip(query_ids[rows], top))print("same top 10 as lib.semantic:", same, "of", rows.sum())flat = faiss.IndexFlatIP(product_vectors.shape[1])     # faiss's exact index: inner product with every productflat.add(product_vectors)_, found = flat.search(query_vectors[rows], 10)# the same products in the same order, row by rowprint("same top 10 as faiss IndexFlatIP:", (found == top).all(axis=1).sum(), "of", rows.sum())run = {query_id: list(product_ids[row]) for query_id, row in zip(query_ids[rows], top)}print(f"ndcg@10 of the exact run: {evaluate(run, qrels('test'))['ndcg@10'].mean():.3f}")
出力python exact-search-numpy.py
similarities computed: 360,000
similarities per query: 12,000
same top 10 as lib.semantic: 30 of 30
same top 10 as faiss IndexFlatIP: 30 of 30
ndcg@10 of the exact run: 0.830

The hand-built search agrees with both checks: it returns the same top 10 as lib.semantic for 30 of 30 test queries, and as faiss for 30 of 30, so its run scores the same mean nDCG@10, 0.830. One matrix product computed every similarity: 360,000 of them, 12,000 per query. The book's shared module, lib.vectors, keeps this top_k for later examples and wraps it in exact_search.

Whether exact search is enough comes down to its cost. Its work grows in step with the catalog: one similarity per product, for every query. Its memory is the vectors themselves:

exact-search-memory.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 合格
"""Prints the bytes one product's float32 vector takes, and the memory the catalog's vectors use, for the store and for a million products. The prose's point rests on the MB line for the store's catalog and the GB line for a million products."""from lib.recorded import embeddings# Exact search keeps every product vector in memory, as 32-bit floats.product_ids, product_vectors, _, _ = embeddings()print(f"products: {len(product_ids):,}")print(f"float32 bytes per product: {product_vectors[0].nbytes:,}")# decimal MB and GB: a megabyte is a million bytes, a gigabyte a billionprint(f"product vectors in memory, MB: {product_vectors.nbytes / 1e6:.1f}")print(f"the same for 1,000,000 products, GB: {1_000_000 * product_vectors[0].nbytes / 1e9:.1f}")
出力python exact-search-memory.py
products: 12,000
float32 bytes per product: 3,072
product vectors in memory, MB: 36.9
the same for 1,000,000 products, GB: 3.1

Each product takes 3,072 bytes as 32-bit floats, so the store's catalog takes 36.9 MB, and a million products would take 3.1 GB. Douze et al. (2024) advise deciding first whether an index is needed at all, because in some cases a brute-force search is the best option. We judge the store's catalog to be such a case.

Working rule: We recommend exact search until it no longer fits in memory or meets your latency budget, measured with your own catalog on your own hardware. It has no settings to tune and never misses a neighbour, so it is also the ground truth for any approximate index. Store the normalized vectors as one 32-bit matrix, take the top k with argpartition, and sort only those.

この部分はまだ翻訳されていないため、英語で表示している。

Trade recall for speed with HNSW

最終変更:版2026.11.10

When exact search costs too much, an approximate nearest-neighbour (ANN) index skips most of its comparisons, and in return may miss some of the true nearest neighbours. An ANN index's quality is its recall@k against exact search: the share of exact search's top k that it also returns. That isn't the recall@k of chapter 3, which compares a ranking with judgments. An index with perfect recall against exact search returns what exact search returns, good or bad.

HNSW, short for hierarchical navigable small world, is a graph index. Each product is a node linked to some of its nearest neighbours, in a stack of layers: the bottom layer holds every product, and each layer above holds fewer. A search starts at the top, moves to whichever linked node is nearest the query, steps down a layer when it gets no closer, and at the bottom explores a list of candidates before it returns the best k (Malkov and Yashunin, 2018). In faiss, M sets the links per node. More links find more true neighbours, but use more memory (faiss wiki). efConstruction, the expansion factor at construction time, sets how many candidates the build considers, and efSearch, the expansion factor at search time, how many a search considers. With faiss 1.16.0, their defaults are 40 and 16, and its graph build is deterministic, because the only build add() uses is a deterministic one (HNSW.h; changelog). So the outputs here repeat exactly. Timings differ from machine to machine, so the examples count work instead: faiss counts the distances it computes in each search (HNSW.h), which for this index are similarities. The count leaves out following links and reading memory, so it is a guide to speed, not a measure.

The store doesn't need an index, but a store with millions of products would, and a catalog small enough to check every answer is the place to learn the trade-off. Build HNSW over the store's 12,000 products with hnsw_index(product_vectors, links=32), lib.vectors's name for M = 32. Then sweep efSearch from 16 to 256 on the train queries:

hnsw-efsearch-sweep.py Python 3.12.15 · faiss-cpu 1.16.0 · numpy 2.5.5 · pandas 3.1.1 合格
"""Builds HNSW with M = 32 and sweeps efSearch on the train queries against exact search, printing recall, the worst query and the work per query. The prose's point rests on the efSearch 16 lines and the two smallest-efSearch lines printed after the table."""import faissimport numpy as npfrom lib.recorded import embeddingsfrom lib.store import load_products, load_queries# EF_SEARCH_GRID runs 16 to 256 in steps of 16; WORST_RECALL is 0.9: at least 9 of the true top 10from lib.vectors import EF_SEARCH_GRID, WORST_RECALL, exact_search, hnsw_index, hnsw_search, recall_against_exactproduct_ids, product_vectors, query_ids, query_vectors = embeddings()names = load_products().loc[product_ids, "name"].str[:36].to_numpy()   # by row number, cut to 36 charactersrows = np.isin(query_ids, load_queries("train").index)                 # tune on train queries onlyqueries = load_queries().loc[query_ids[rows], "query"].to_numpy()index = hnsw_index(product_vectors, links=32)          # M = 32, and faiss's default efConstruction of 40levels = faiss.vector_to_array(index.hnsw.levels)      # how many layers each product reachesfor layer in range(1, levels.max() + 1):    print(f"products in layer {layer}: {(levels >= layer).sum():,}")exact = exact_search(query_vectors[rows], product_vectors, 10)print("\nefSearch  mean recall@10  worst  queries at 0  similarities per query")sweep = {}for ef_search in EF_SEARCH_GRID:    # similarities is faiss's count of the distances each search computed, per query, not a score    found, similarities = hnsw_search(index, query_vectors[rows], 10, ef_search)    # share of exact search's top 10 that the approximate search also found, order ignored    recall = recall_against_exact(found, exact)    sweep[ef_search] = (recall, similarities)    print(f"{ef_search:>8}  {recall.mean():>14.3f}  {recall.min():>5.1f}  {(recall == 0).sum():>12}  {similarities:>22.0f}")recall, similarities = sweep[16]                       # faiss's defaultprint(f"\nefSearch 16, recall@10 against exact: {recall.mean():.3f}")print(f"efSearch 16, similarities computed per query: {similarities:.0f}")print(f"efSearch 16, train queries with recall@10 of 0: {(recall == 0).sum()}")print("efSearch 16, those queries:", ", ".join(queries[recall == 0]))print("smallest efSearch with no query at 0:", min(ef for ef, (r, _) in sweep.items() if r.min() > 0))chosen = min(ef for ef, (r, _) in sweep.items() if r.min() >= WORST_RECALL)print(f"smallest efSearch with every query at {WORST_RECALL} or more:", chosen)print(f"chosen efSearch, recall@10 against exact: {sweep[chosen][0].mean():.3f}")print(f"chosen efSearch, similarities computed per query: {sweep[chosen][1]:.0f}\n")# the top 3 at the default efSearch, for the query that failsfound, _ = hnsw_search(index, query_vectors[rows], 3, 16)row = list(queries).index("foutains with brick look")for rank, (approximate, nearest) in enumerate(zip(found[row], exact[row]), start=1):    print(f"{rank}  hnsw: {names[approximate]:<36}  exact: {names[nearest]}")
出力python hnsw-efsearch-sweep.py
products in layer 1: 12,000
products in layer 2: 369
products in layer 3: 10

efSearch  mean recall@10  worst  queries at 0  similarities per query
      16           0.934    0.0             2                     282
      32           0.936    0.0             2                     346
      48           0.940    0.0             2                     404
      64           0.940    0.0             2                     463
      80           0.956    0.0             1                     530
      96           0.978    0.3             0                     611
     112           0.980    0.3             0                     682
     128           0.994    0.8             0                     767
     144           0.998    0.9             0                     835
     160           0.998    0.9             0                     924
     176           0.998    0.9             0                    1020
     192           0.998    0.9             0                    1115
     208           0.998    0.9             0                    1191
     224           1.000    1.0             0                    1271
     240           1.000    1.0             0                    1349
     256           1.000    1.0             0                    1441

efSearch 16, recall@10 against exact: 0.934
efSearch 16, similarities computed per query: 282
efSearch 16, train queries with recall@10 of 0: 2
efSearch 16, those queries: foutains with brick look, sheffield home bath set
smallest efSearch with no query at 0: 96
smallest efSearch with every query at 0.9 or more: 144
chosen efSearch, recall@10 against exact: 0.998
chosen efSearch, similarities computed per query: 835

1  hnsw: dayse outdoor square cushion with fi  exact: tribeca 2 '' x 10 '' porcelain brick
2  hnsw: 3 piece wall decor set                exact: barbados 5 '' x 10 '' ceramic brick
3  hnsw: rustic autumn ivy branch              exact: kimary 0.75 '' x 1.5 '' glass brick

The graph thins fast: the bottom layer holds all 12,000 products, the next 369 and the top 10.

The mean hides a failure. At the default efSearch of 16, a query computes 282 similarities instead of 12,000, and mean recall@10 against exact search is 0.934. Yet 2 train queries get none of their true top 10.

Those queries are foutains with brick look, sheffield home bath set. Look at foutains with brick look, one of them: HNSW returns a cushion, a wall decor set and an ivy branch, while exact search's top 3 are brick-look tiles.

The smallest efSearch in the sweep at which no train query is at 0 is 96, but the sweep's table shows that its worst query still misses most of its true top 10. So choose by a stricter rule: the smallest efSearch at which every train query finds at least 9 of its true top 10. That is 144, with a mean recall of 0.998 and 835 similarities per query.

Working rule: Use an ANN index only when exact search is too slow or too large. Tune it on train queries by its recall against exact search, at the k your pipeline uses, and choose by the worst query, not the mean. We recommend the smallest efSearch at which every query keeps at least 9 of its true top 10, with M = 32. This chapter tunes only efSearch; M and efConstruction change the graph itself, and deserve the same sweep on a large catalog. Aumüller et al. (2018) find that very different ANN methods reach comparable quality-performance trade-offs, so measure on your own data, not by reputation.

テストクエリでインデックスを検証する

最終変更:版2026.11.8

train クエリで調整したインデックスは、見たことのないクエリでも検証しなければならない。検証は、完全探索に対する再現率と、評価済みスコアへの影響の両面から行う。本章で計画した比較は2件であり、それぞれが対応のある並べ替え検定と、それに並べて行うt 検定からなる。比較の対象は、faiss の既定のefSearchである 16 を使った HNSW インデックスと、train クエリで選んだ設定の HNSW インデックスであり、いずれも完全探索と比べる。検索ではk = 100も要求する。ハイブリッド検索が各検索器の上位 100 件を融合するためである。

hnsw-test-recall.py Python 3.12.15 · faiss-cpu 1.16.0 · numpy 2.5.5 · pandas 3.1.1 · scipy 1.18.1 合格
"""Compares HNSW at faiss's default efSearch, and at the setting chosen on train queries, with exact search on the test queries, using paired tests at the planned threshold. The prose's point rests on the default setting's permutation and interval lines, and on the chosen setting's recall line."""import numpy as npfrom lib.metrics import evaluatefrom lib.recorded import embeddingsfrom lib.significance import compare, verdictfrom lib.store import load_queries, qrelsfrom lib.vectors import as_run, choose_ef_search, exact_search, hnsw_index, hnsw_search, recall_against_exact# Planned before scoring: HNSW at faiss's default efSearch, and at the setting chosen on train, each against exact.PLANNED = 2threshold = 0.05 / PLANNED  # the book's rule splits 0.05 across the planned comparisonsproduct_ids, product_vectors, query_ids, query_vectors = embeddings()rows = np.isin(query_ids, load_queries("test").index)index = hnsw_index(product_vectors, links=32)# the smallest efSearch at which every train query keeps WORST_RECALL of its true top 10chosen = choose_ef_search(index, query_vectors[np.isin(query_ids, load_queries("train").index)], product_vectors)# k = 100 because hybrid retrieval fuses each retriever's top 100exact = exact_search(query_vectors[rows], product_vectors, 100)exact_ndcg = evaluate(as_run(exact, query_ids[rows], product_ids), qrels("test"))["ndcg@10"]print("planned test comparisons:", PLANNED)print(f"p-value threshold: {threshold:.3f}")print("efSearch chosen on train queries:", chosen)print(f"exact ndcg@10: {exact_ndcg.mean():.3f}")for name, ef_search in [("default", 16), ("chosen", chosen)]:    found, _ = hnsw_search(index, query_vectors[rows], 100, ef_search)    # the first 10 of the 100    recall = recall_against_exact(found[:, :10], exact[:, :10])    ndcg = evaluate(as_run(found, query_ids[rows], product_ids), qrels("test"))["ndcg@10"]    result = compare(ndcg, exact_ndcg, confidence=1 - threshold)  # a 97.5% interval, at the test's level    print(f"\n{name}, recall@10 against exact: {recall.mean():.3f}")    print(f"{name}, queries with recall@10 of 0: {(recall == 0).sum()}")    print(f"{name}, those queries:", ", ".join(load_queries().loc[query_ids[rows][recall == 0], "query"]) or "none")    print(f"{name}, recall@100 against exact: {recall_against_exact(found, exact).mean():.3f}")    print(f"{name}, hnsw ndcg@10: {ndcg.mean():.3f}")    # compare rounds each difference to 3 decimals: wins and losses are the nonzero ones    print(f"{name}, queries whose ndcg@10 changes: {result['wins'] + result['losses']}")    print(f"{name}, permutation p: {result['permutation p']:.3f}")    print(f"{name}, t-test p: {result['t-test p']:.3f}")    print(f"{name}, {1 - threshold:.1%} interval: {result['ci low']:.4f} to {result['ci high']:.4f}")    for check, passed in verdict(result, threshold).items():      # established, t-test agrees, interval agrees        print(f"{name}, {check}: {'yes' if passed else 'no'}")
出力python hnsw-test-recall.py
planned test comparisons: 2
p-value threshold: 0.025
efSearch chosen on train queries: 144
exact ndcg@10: 0.830

default, recall@10 against exact: 0.930
default, queries with recall@10 of 0: 2
default, those queries: liberty hardware francisco, 48" sliding single track , barn door for laundry
default, recall@100 against exact: 0.849
default, hnsw ndcg@10: 0.791
default, queries whose ndcg@10 changes: 3
default, permutation p: 0.256
default, t-test p: 0.154
default, 97.5% interval: -0.1544 to -0.0006
default, established: no
default, t-test agrees: yes
default, interval agrees: no

chosen, recall@10 against exact: 1.000
chosen, queries with recall@10 of 0: 0
chosen, those queries: none
chosen, recall@100 against exact: 0.994
chosen, hnsw ndcg@10: 0.830
chosen, queries whose ndcg@10 changes: 0
chosen, permutation p: 1.000
chosen, t-test p: 1.000
chosen, 97.5% interval: 0.0000 to 0.0000
chosen, established: no
chosen, t-test agrees: yes
chosen, interval agrees: yes

train クエリで選んだ設定は持ちこたえている。完全探索に対する recall@10 は1.000、平均 nDCG@10 は0.830で、完全探索の0.830と同じであり、スコアが変わるクエリは0件である。

faiss の既定の設定では、真の上位 10 件を 1 件も得られないテストクエリが2件ある(liberty hardware francisco, 48" sliding single track , barn door for laundry)。平均 nDCG@10 も0.791に下がる。ただしスコアが変わるクエリは3件にとどまり、この低下は確立されていない。並べ替え検定は p = 0.256を与え、t 検定も p = 0.154で一致する。-0.1544 to -0.0006である 97.5% 区間は 0 を含まず、検定とは食い違う。差を担うクエリがこれほど少ないため、ここでの知見は完全探索に対して失われた再現率であり、品質の低下を測定したものではない。

より多くの結果を要求しても、小さなefSearchを埋め合わせることはできない。既定では recall@100 は0.849であり、選んだ設定では0.994である。

グラフはベクトルに加えてメモリも消費する。

hnsw-index-memory.py Python 3.12.15 · faiss-cpu 1.16.0 · numpy 2.5.5 · pandas 3.1.1 合格
"""Counts the links per product in the HNSW graph, and the bytes per product of the exact and the HNSW index. The prose's point rests on the two bytes-per-product lines, set against faiss's estimate."""import faissfrom lib.recorded import embeddingsfrom lib.vectors import hnsw_indexproduct_ids, product_vectors, _, _ = embeddings()dimensions = product_vectors.shape[1]flat = faiss.IndexFlatIP(dimensions)                    # exact search: the vectors onlyflat.add(product_vectors)index = hnsw_index(product_vectors, links=32)           # the vectors, plus the graph's links# layer 0 is the bottom layer: 2 × M links there, and M on each upper layerprint("links per product, upper layers:", index.hnsw.nb_neighbors(1))print("links per product, bottom layer:", index.hnsw.nb_neighbors(0))# 8M with M = 32: 4 bytes for each of the 2M bottom-layer linksprint(f"faiss's estimate, 4d + 8M bytes per product: {4 * dimensions + 8 * 32:,}")# the saved index's size, spread over the productsfor name, built in [("exact index", flat), ("hnsw index", index)]:    print(f"bytes per product, {name}: {faiss.serialize_index(built).nbytes / len(product_ids):,.0f}")
出力python hnsw-index-memory.py
links per product, upper layers: 32
links per product, bottom layer: 64
faiss's estimate, 4d + 8M bytes per product: 3,328
bytes per product, exact index: 3,072
bytes per product, hnsw index: 3,344

各商品は、上位の層に32本、最下層に64本のリンクを持つ。最下層のリンク数はMの 2 倍である。faiss はベクトル 1 本あたり 4d + 8M バイトと見積もる(faiss wiki)。これは d 個の数値それぞれに 4 バイト、最下層の 2M 本のリンクそれぞれに 4 バイトという内訳であり、ここでは3,328になる。保存したインデックスはこれに近く、1 商品あたり3,344バイトである。完全探索のインデックスは3,072バイトである。

実践ルール: 調整したインデックスは、保持しておいたクエリで、完全探索に対する再現率によって検証する。平均スコアを読む前に、上位 k 件をすべて失うクエリを確認する。nDCG@10 の低下がごく少数のクエリに依存しているなら、対応のある検定で確立されるまでは未証明として扱う。メモリは、ベクトルだけでなくリンクの分も見積もる。

この部分はまだ翻訳されていないため、英語で表示している。

Check queries that contain digits

最終変更:版2026.11.10

An index can only return what the embeddings put near a query, so the next question is which queries the embeddings get wrong. An encoder squeezes a whole query into one vector, so a short token that changes the meaning, such as a size, a model number or a negation, may move that vector very little. Deng et al. (2026) measured this for numbers, in a study of 13 embedding models on synthetic financial text. Their accuracy at telling texts apart by a number was 0.54, only slightly above the 0.5 of random guessing, and four-digit numbers were at the level of guessing. Products aren't financial reports, so check the store's queries.

Few of the store's queries hold a digit, so compare BM25 and semantic retrieval on each that does, from both splits. Nothing is tuned on these queries, and a handful can't establish anything:

digits-in-queries.py Python 3.12.15 · bm25s 0.3.16 · numpy 2.5.5 · pandas 3.1.1 · scipy 1.18.1 合格
"""Scores BM25 and semantic retrieval on each query with a digit, in both splits, and counts the products whose text writes 7qt or 7 qt. The prose's point rests on the three lines that count which ranker wins and the two product counts."""import pandas as pdfrom lib.lexical import LexicalSearchfrom lib.metrics import evaluatefrom lib.semantic import SemanticSearchfrom lib.store import load_products, load_queries, product_text, qrelsproducts, queries = load_products(), load_queries()with_digit = queries[queries["query"].str.contains(r"\d")]       # both splits: a diagnosis, nothing is tunedjudgments = {query_id: grades for query_id, grades in qrels().items() if query_id in with_digit.index}# each ranker returns its top 100; nDCG@10 scores the first 10 of themscores = pd.DataFrame({    "split": with_digit["split"],    "bm25": evaluate(LexicalSearch(products).run(with_digit, k=100), judgments)["ndcg@10"],    "semantic": evaluate(SemanticSearch().run(with_digit, k=100), judgments)["ndcg@10"],    "query": with_digit["query"],}).round(3)print("queries with a digit:", len(scores))print(scores.to_string())print("semantic better:", (scores["semantic"] > scores["bm25"]).sum())print("bm25 better:", (scores["bm25"] > scores["semantic"]).sum())# a gap under 0.03 counts as a near tie: the book's choice, not a measured limitprint("gaps under 0.03:", ((scores["semantic"] - scores["bm25"]).abs() < 0.03).sum())text = product_text(products).str.lower()print("products whose text has the word 7qt:", text.str.contains(r"\b7qt\b").sum())# '7 qt', but not '17 qt' or '3.7 qt'print("products whose text has 7 qt:", text.str.contains(r"(?<![\d.])7 qt\b").sum())
出力python digits-in-queries.py
queries with a digit: 7
          split   bm25  semantic                                             query
query_id
45        train  0.288     0.316                            70s inspired furniture
62        train  0.554     0.930                                   7qt slow cooker
101       train  0.569     0.770        48 in entry table with side by side drawer
203       train  0.606     0.740                         ge top loading washer 4.5
205       train  0.935     0.930                          ines shag rug w003096809
285        test  0.592     0.555  48" sliding single track , barn door for laundry
482        test  1.000     0.921                              48 inch kitchen hood
semantic better: 4
bm25 better: 3
gaps under 0.03: 2
products whose text has the word 7qt: 0
products whose text has 7 qt: 8

Neither ranker owns the 7 queries with a digit. Semantic retrieval scores higher on 4 of them and BM25 on 3, and 2 of the gaps are under 0.03. BM25's clearest loss is 7qt slow cooker: the word 7qt appears in 0 products, and 8 write 7 qt.

On the test query 48 inch kitchen hood, BM25 scores higher, so here is semantic retrieval's top 10:

hood-query-digits.py Python 3.12.15 · bm25s 0.3.16 · numpy 2.5.5 · pandas 3.1.1 · scipy 1.18.1 合格
"""Lists semantic retrieval's top 10 for 48 inch kitchen hood with each hood's width, read from the attributes column the embeddings never saw. The prose's point rests on the two width-48 lines, one for each ranker."""from lib.lexical import LexicalSearchfrom lib.semantic import SemanticSearchfrom lib.store import load_products, load_queries, qrelsLABELS = {2: "exact", 1: "partial", 0: "irrelevant"}products = load_products()query_id = 482  # 48 inch kitchen hood, a test queryquery = load_queries().at[query_id, "query"]judgments = qrels()[query_id]# The width sits in the attributes column, which the embeddings never saw: product_text leaves it out.width = products["attributes"].str.extract(r"\bwidth: ([\d.]+)", expand=False).astype(float)semantic = SemanticSearch().search(query_id, k=10).indexbm25 = LexicalSearch(products).search(query, k=10).indexprint("query:", query)print("     judgment   width  name")for rank, product_id in enumerate(semantic, start=1):    label = LABELS.get(judgments.get(product_id), "unjudged")    print(f"{rank:>2}   {label:<10} {width[product_id]:>5g}  {products.at[product_id, 'name'][:50]}")print()# exact: a 48-inch hood is 48 wide, where probe-pair-overlap allows half an inch for hoods listed at 35.75for name, top in [("semantic", semantic), ("bm25", bm25)]:    print(f"{name}, width 48 in the top 10:", (width[top] == 48).sum())    print(f"{name}, exact in the top 10:", sum(judgments.get(product_id) == 2 for product_id in top))
出力python hood-query-digits.py
query: 48 inch kitchen hood
     judgment   width  name
 1   exact         48  48 '' 1200 cfm ducted wall mount range hood in bru
 2   exact         48  48 '' 400 cfm ducted wall mount range hood in brus
 3   exact         48  48 '' 400 cfm convertible wall mount range hood
 4   partial    35.75  36 '' ws-48 series 500 cfm ducted under cabinet ra
 5   exact         48  48 '' hand crafted designer wood island hood serie
 6   exact         48  48 '' island range hood
 7   exact         48  48 inch wall mount chimney range hood with 600 cfm
 8   exact         48  48 '' ducted island range hood
 9   exact         48  48 '' professional 900cfm ducted island range hood
10   partial     29.8  30 '' 480 cfm convertible under cabinet range hood

semantic, width 48 in the top 10: 8
semantic, exact in the top 10: 8
bm25, width 48 in the top 10: 10
bm25, exact in the top 10: 10

For this query, the embedding matched the digits as text, not the width they stand for. Of semantic retrieval's top 10, 8 hoods have a width of 48. The others are a 36-inch hood from a series named ws-48 and a 30-inch hood rated at 480 cfm, cubic feet per minute, the airflow of its fan. Both contain the digits 48, neither is 48 wide, and the judges marked both partial. BM25's top 10 holds 10 hoods of width 48.

Each hood's width comes from the attributes column, which holds curated fields from WANDS's product features, such as width, shape and material. product_text, the text the model embedded, leaves that column out, so the embeddings never saw it, and a size the shopper types can be checked against it exactly.

One query can't show a pattern. To see how the model handles sizes in general, use probe queries: pairs of queries written for the book that differ only in a size, a code or a negation, which recordings/probes.py embeds with the same model. One of them repeats the test query 48 inch kitchen hood word for word. Probes have no judgments, so the example counts their top 10s and never scores them. The right size comes from the width in attributes for the hoods, and from the name for the rest:

probe-pair-overlap.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 合格
"""Counts, for each probe pair, the products the two top 10s share and the top-10 products of the right size, without scoring anything. The prose's point rests on the right-size lines for the hood, slow-cooker and bed probes, and on the seat-count and loveseat lines for the sofa probe."""from lib.recorded import probesfrom lib.semantic import SemanticSearchfrom lib.store import load_products# Probe queries written for the book, embedded by the same model. They have no judgments,# so this example counts what each top 10 holds; it doesn't score anything.probe_queries, probe_vectors = probes()products = load_products()names = products["name"]width = products["attributes"].str.extract(r"\bwidth: ([\d.]+)", expand=False).astype(float)semantic = SemanticSearch()def states(size, unit):    """Names that state this size, such as "7 qt" or "7-quart", but not "3.7 qt"."""    return names.str.contains(rf"(?<![\d.]){size}[ -]?(?:{unit})")RIGHT_SIZE = {                       # the size each probe asks for: the width attribute, or the name    # within half an inch of 36 or 48, so a 35.75-inch hood counts as 36    "36 inch kitchen hood": (width - 36).abs() <= 0.5, "48 inch kitchen hood": (width - 48).abs() <= 0.5,    "4 qt slow cooker": states(4, "qt|quart"), "7 qt slow cooker": states(7, "qt|quart"),    # the whole word, which also matches 'twin over full' bunk beds    "twin bed": names.str.contains(r"\btwin\b"), "king bed": names.str.contains(r"\bking\b"),    "2 seat sofa": names.str.contains(r"\bloveseat\b"),        # a two-seat sofa is a loveseat}DIGIT_ONLY = {"2 seat sofa": states(2, " in 1| piece"), "4 seat sofa": states(4, " in 1| piece")}for pair in ["hood", "cooker", "bed", "seats"]:    first, second = probe_queries.index[probe_queries["pair"] == pair]  # each pair is two probes, in file order    tops = {row: semantic.search_vector(probe_vectors[row], k=10).index for row in (first, second)}    print(f"{pair}, similarity of the two queries: {probe_vectors[first] @ probe_vectors[second]:.3f}")    print(f"{pair}, products in both top 10s:", len(set(tops[first]) & set(tops[second])))    for row, top in tops.items():        query = probe_queries.at[row, "query"]        if query in RIGHT_SIZE:            print(f"{query}, right size in the top 10:", RIGHT_SIZE[query][top].sum())        if query in DIGIT_ONLY:            print(f"{query}, same digit but a piece count or 2 in 1:", DIGIT_ONLY[query][top].sum())    print()# the two counts the sofa probes ask forprint("catalog names that give a seat count:", (states(2, "seat") | states(4, "seat")).sum())print("catalog products named loveseat:", names.str.contains(r"\bloveseat\b").sum())
出力python probe-pair-overlap.py
hood, similarity of the two queries: 0.795
hood, products in both top 10s: 0
36 inch kitchen hood, right size in the top 10: 8
48 inch kitchen hood, right size in the top 10: 8

cooker, similarity of the two queries: 0.841
cooker, products in both top 10s: 1
4 qt slow cooker, right size in the top 10: 6
7 qt slow cooker, right size in the top 10: 7

bed, similarity of the two queries: 0.697
bed, products in both top 10s: 0
twin bed, right size in the top 10: 10
king bed, right size in the top 10: 10

seats, similarity of the two queries: 0.759
seats, products in both top 10s: 1
2 seat sofa, right size in the top 10: 0
2 seat sofa, same digit but a piece count or 2 in 1: 4
4 seat sofa, same digit but a piece count or 2 in 1: 4

catalog names that give a seat count: 0
catalog products named loveseat: 14

For these probes, the model mostly finds the right size. The two hood probes share 0 products, yet each top 10 holds hoods of the right width, 8 for 36 inches and 8 for 48; the slow-cooker probes find 6 and 7 of the right capacity, and the bed probes 10 and 10 of the right size. Where the catalog has no number to match, the digit takes over. The number of product names that give a seat count is 0, so the top 10 for 2 seat sofa holds 4 products that share only the digit, two-piece sets and "2 in 1" kids' sofas, and 0 of the catalog's 14 loveseats.

商品コードの重みは小さいと見込む

最終変更:版2026.11.8

商品コードは、1つの商品とだけ完全に一致するべきものである。エンコーダにとって、それは文字と数字からなる珍しい文字列にすぎない。プローブクエリのうち2組が、コードを調べるものである。1組は、どの商品にも存在しないコードを使ったもので、学習用クエリ ines shag rug w003096809 から取っている。もう1組は、ちょうど1つの商品のテキストにだけ現れる画面解像度を使ったものである。

code-probes.py Python 3.12.15 · bm25s 0.3.16 · numpy 2.5.5 · pandas 3.1.1 · scipy 1.18.1 合格
"""Compares the top 10s of two probe queries that differ only in a code no product holds, and shows where one product ranks for two resolutions, semantically and with BM25. The prose's point rests on the shared-top-10 line for the code and on the rank lines for the resolutions."""from lib.lexical import LexicalSearchfrom lib.recorded import probesfrom lib.semantic import SemanticSearchfrom lib.store import load_products, product_text# Probe queries written for the book, embedded by the same model; no judgments, so nothing is scored.probe_queries, probe_vectors = probes()products = load_products()text = product_text(products).str.lower()semantic, bm25 = SemanticSearch(), LexicalSearch(products)# A code no product holds: does changing it change the results?first, second = probe_queries.index[probe_queries["pair"] == "code"]tops = [set(semantic.search_vector(probe_vectors[row], k=10).index) for row in (first, second)]print("products whose text contains w003096809:", text.str.contains("w003096809").sum())print(f"code, similarity of the two queries: {probe_vectors[first] @ probe_vectors[second]:.3f}")print("code, products in both top 10s:", len(tops[0] & tops[1]))# A code one product holds: where does that product rank, with its code and with another?holder = text.index[text.str.contains("1920x1080p")]print()print("products whose text contains 1920x1080p:", len(holder))print("that product:", products.at[holder[0], "name"])print("products in the Projectors class:", (products["product_class"] == "Projectors").sum())for row in probe_queries.index[probe_queries["pair"] == "resolution"]:    query = probe_queries.at[row, "query"]    ranked = semantic.search_vector(probe_vectors[row], k=len(products))  # ranks every product, so the rank is exact    lexical = list(bm25.search(query, k=100).index)  # BM25 ranks are read from its top 100 only    print(f"{query}, its semantic rank: {list(ranked.index).index(holder[0]) + 1}")  # ranks count from 1    print(f"{query}, its similarity: {ranked[holder[0]]:.3f}")    # the best product other than this one    print(f"{query}, similarity of the next product: {ranked.drop(holder[0]).iloc[0]:.3f}")    print(f"{query}, its bm25 rank:", lexical.index(holder[0]) + 1 if holder[0] in lexical else "not in the top 100")
出力python code-probes.py
products whose text contains w003096809: 0
code, similarity of the two queries: 0.967
code, products in both top 10s: 10

products whose text contains 1920x1080p: 1
that product: 5000 portable transmissive projector
products in the Projectors class: 4
1920x1080p projector, its semantic rank: 1
1920x1080p projector, its similarity: 0.562
1920x1080p projector, similarity of the next product: 0.496
1920x1080p projector, its bm25 rank: 1
2560x1440p projector, its semantic rank: 1
2560x1440p projector, its similarity: 0.509
2560x1440p projector, similarity of the next product: 0.508
2560x1440p projector, its bm25 rank: not in the top 100

存在しないコードの末尾の数字を変えても、2つのクエリの類似度は 0.967 であり、上位10件に共通する商品は 10 件である。何にも一致しないコードは何も変えず、BM25 にも照合できるものがない。

1つの商品だけが持つコードは、BM25 の順位を決めるが、埋め込みの順位をわずかに動かすにとどまる。解像度 1920x1080p は、5000 portable transmissive projector のテキストにしか現れない。BM25 での順位は、1920x1080p projector に対して 1 であり、2560x1440p projector に対しては not in the top 100 である。Sciavolino ら (2021) は、稀なエンティティに関する質問では、密な(埋め込みに基づく)検索器が疎な手法より大幅に劣ることを見出した。一方、Karpukhin ら (2020) は、BM25 を、選択性の高いキーワードやフレーズによく反応するものとして説明している。

埋め込みでの順位は、自身のコードを使った場合が 1、もう一方のコードを使った場合が 1 である。ストアにあるプロジェクターが 4 件しかないためである。コードの影響は、次点の商品との差にだけ表れる。自身のコードを使うと 0.562 対 0.496 であり、使わないと 0.509 対 0.508 である。

実践ルール: ログの中から、サイズ、数量、コードを含むクエリを洗い出し、どちらの検索器も信頼する前に、両方でそれらを比較すること。コードが重要なときは、字句的に照合する。正確なサイズが重要なときは、attributes の幅のような商品フィールドと照合するか、トークナイザーが数字を保持し単位の書き方が統一されてから字句的に照合する(5章、8章)。クエリの残りの部分は、意味的検索に任せる。両者の組み合わせは7章で検証する。

否定表現に対する埋め込みを検証する

最終変更:版2026.11.8

否定は、短い一語で購入者の求めるものを正反対にする。たとえば、丸くないラグがその一例である。クエリ全体の埋め込みは、その一語を除いた同じクエリに近いままであることがある。否定の有無だけが異なる文書ペアからなるベンチマーク NevIR では、どのバイエンコーダのアーキテクチャもペア正解率が 12% を超えなかった。ランダムに順位を付けた場合は 25% である(Weller et al., 2024)。2025 年の再現実験で van den Elsen et al. は、70 億パラメータのバイエンコーダである GritLM-7B が 39.0% に達したものの、他のバイエンコーダはほとんど改善せず、両方の否定タスクをおおむね適切に処理できたのはクロスエンコーダとリストワイズ LLM リランカーだけであったと報告している。

ストアのクエリのうち「no」「not」「without」「non」を含むものは 0 件であり、判定データでは否定を検証できない。そこで代わりに、3 組のプローブクエリを用いる。各組は、あるクエリとその否定形からなり、それぞれの上位 10 件に何が含まれるかを数える。

このカタログは、否定を含むどのプローブにも応えられる。armless と名付けられた商品が 42 件、ガラスに一切言及しないコーヒーテーブルが 62 件、round と名付けられていないエリアラグが 90 件ある。

実行する前に、rug that is not round に対するセマンティック検索の上位 10 件のうち、round と名付けられたものが何件かを予想してほしい。

negation-probes.py Python 3.12.15 · bm25s 0.3.16 · numpy 2.5.5 · pandas 3.1.1 · scipy 1.18.1 合格
"""Counts what each negation probe's top 10 holds for semantic retrieval and for BM25, without scoring it. The prose's point rests on the semantic top 10 lines for the rug, coffee-table and sofa probes."""import bm25sfrom lib.lexical import LexicalSearchfrom lib.recorded import probesfrom lib.semantic import SemanticSearchfrom lib.store import load_products, load_queries, product_text# Probe queries written for the book, with no judgments: count what the top 10 holds, never score it.probe_queries, probe_vectors = probes()products = load_products()names, text = products["name"].str.lower(), product_text(products).str.lower()semantic, bm25 = SemanticSearch(), LexicalSearch(products)# armless and round are matched in product names, glass in the full textFEATURES = {"arms": ("armless", names), "glass": ("glass", text), "round": ("round", names)}negated = load_queries()["query"].str.contains(r"\b(?:no|not|without|non)\b")print("store queries with no, not, without or non:", negated.sum())# the tokenizer settings LexicalSearch uses, so these are the terms BM25 searchestokens = bm25s.tokenize(["rug that is not round"], stopwords="en", return_ids=False, show_progress=False)print("bm25 terms of rug that is not round:", " ".join(tokens[0]))print("products named armless:", names.str.contains(r"\barmless\b").sum())coffee_tables = products["product_class"] == "Coffee & Cocktail Tables"area_rugs = products["product_class"] == "Area Rugs"print("coffee tables that never mention glass:", (coffee_tables & ~text.str.contains(r"\bglass\b")).sum())print("area rugs not named round:", (area_rugs & ~names.str.contains(r"\bround\b")).sum())negation = probe_queries[probe_queries["kind"] == "negation"]print("negation probe pairs:", negation["pair"].nunique())for row, probe in negation.iterrows():    word, field = FEATURES[probe["pair"]]    has_feature = field.str.contains(rf"\b{word}\b")    semantic_top = semantic.search_vector(probe_vectors[row], k=10).index    bm25_top = bm25.search(probe["query"], k=10).index    print()    print(f"{probe['query']}, semantic top 10 with {word}:", has_feature[semantic_top].sum())    print(f"{probe['query']}, bm25 top 10 with {word}:", has_feature[bm25_top].sum())
出力python negation-probes.py
store queries with no, not, without or non: 0
bm25 terms of rug that is not round: rug round
products named armless: 42
coffee tables that never mention glass: 62
area rugs not named round: 90
negation probe pairs: 3

sofa with arms, semantic top 10 with armless: 1
sofa with arms, bm25 top 10 with armless: 0

sofa without arms, semantic top 10 with armless: 8
sofa without arms, bm25 top 10 with armless: 0

coffee table with glass top, semantic top 10 with glass: 10
coffee table with glass top, bm25 top 10 with glass: 9

coffee table not glass, semantic top 10 with glass: 10
coffee table not glass, bm25 top 10 with glass: 10

round rug, semantic top 10 with round: 10
round rug, bm25 top 10 with round: 10

rug that is not round, semantic top 10 with round: 10
rug that is not round, bm25 top 10 with round: 10

少ないと予想したなら、購入者と同じように「not」を読んだことになる。しかし埋め込みはそう読まなかった。ラグとコーヒーテーブルのプローブでは、埋め込みは属性を保持し、「not」を失った。rug that is not round に対するセマンティック検索の上位 10 件には、round と名付けられた商品が 10 件含まれ、round rug の上位 10 件では 10 件である。coffee table not glass では、上位 10 件のうちガラスに言及するものが 10 件で、coffee table with glass top の 10 件と比べてもほとんど減っていない。

sofa without arms は逆の結果になり、上位 10 件のうち armless と名付けられたものは 8 件で、sofa with arms の場合の 1 件を上回る。カタログにはこの否定を表す語として armless があり、モデルは「without arms」をそれに近い位置に置く。

BM25 もこれらのプローブでは成績が良くない。トークナイザは「not」「that」「is」をストップワードとして除去するため、rug that is not round を rug round として検索する。上位 10 件には、丸いラグが 10 件、armless のソファが 0 件含まれる。

これらのプローブはわずかな組にすぎず、測定と呼べるものではない。ソファの組は逆の結果にもなっている。しかしラグとコーヒーテーブルの組は NevIR と一致しており、クエリベクトルはクエリが挙げる属性の近くにとどまる。

実践ルール: 否定の処理を、語彙ベースかセマンティックかを問わず、第一段階の検索器に頼ってはならない。購入者が、カタログに格納された属性(形状、素材、アームなど)を除外したいときは、その除外を検出し、その属性でフィルタリングする。それ以外の場合はリランキングを行う。van den Elsen et al. の再現実験では、クロスエンコーダと LLM リランカーのほうが否定をはるかにうまく処理したからである。第 9 章では、ストアのセマンティック検索をクロスエンコーダと LLM でリランキングするが、否定を検証できないストアの判定付きクエリでは、どのリランカーもセマンティック検索を上回らない。こうしたプローブは、モデルを変更するとき、あるいはリランカーを追加するたびに再実行する検査として残しておく。

埋め込みモデルを選ぶ

ベクトルが何を表現できるかはモデルによって決まり、後からモデルを変更するにはカタログ全体を埋め込み直す必要がある。公開ベンチマークは候補を絞り込む助けになる。決め手となるのは、そのストア自身の判定である。

ベンチマークは候補を絞る材料として読む

最終変更:版2026.11.8

MTEB(Massive Text Embedding Benchmark)は、多数のタスクでモデルを採点する。その著者らは、すべてのタスクで他を上回る単一の手法は存在しないことを見いだした(Muennighoff et al., 2023)。また、そのスコアの平均はデータセット数の多いタスク、とりわけ検索、分類、クラスタリングに偏ることも見いだしているため、スコアは自分が扱うタスクのものを読むべきである。

公開スコアは、モデルが見たことのないデータでの性能を過大に見せることもある。検索ベンチマークであるRTEBの作成者らは、公開ベンチマークで報告されるモデルのスコアと、新しい未知のデータでの実際の性能との間に差があると述べている(Liu et al., 2025)。RTEBは48個のデータセットのうち28個を非公開にしている。Verma et al. (2026)は、RTEBにおけるモデルの順位が、MTEBおよびBEIR(Thakur et al., 2021)、すなわち多くの領域にまたがる検索を対象とした先行のベンチマークにおける順位と、ρ = 0.38 の相関しか示さないことを明らかにしている。ここでρは順位相関であり、1は順序が同一であることを意味する。大きさも品質を保証するものではない。Enevoldsen et al. (2025)は、MTEBの多言語対応の後継であるMMTEBにおいて、公開されているモデルのうち最も性能が高いモデルはmultilingual-e5-large-instructであり、そのパラメータ数は5億6000万にすぎないと報告している。

このストアは4つのオープンモデルを検討したが、いずれのモデルカードにもWANDSでの結果は載っていない。そのうち2つは、ストアのエンコーダとQwen/Qwen3-Embedding-0.6B(カード)である。残る2つはBAAI/bge-base-en-v1.5(カード)とibm-granite/granite-embedding-english-r2(カード)である。ESCIのような商品検索のベンチマークは、クエリと商品の組を完全一致、代替、補完、無関係のいずれかに分類し、約13万件のクエリと260万件の判定を含む(Reddy et al., 2022)。これはストアのタスクに近いが、それでもストア自身のカタログではない。そのため、ストアの候補の絞り込みは、リーダーボードでは確かめられない条件に基づいている。商用利用を許すライセンスであること、英語に対応していること、特別なハードウェアなしで動くサイズであること、そしてより短いベクトルへの対応が文書化されていることである。ストアが選んだモデルのカードは、MTEBの検索タスクでnDCG@10が55.14と報告しているが、これはベンダーによる測定であり、対象は商品ではなく一般的なテキストである(モデルカード)。候補の中から選ぶには、それぞれについて同じ商品テキストと同じクエリで埋め込みを記録し、学習用クエリで採点し、本書のペア検定で比較する。次の小節では、1つのモデルをいくつかのサイズで比較する際に、この手順を用いている。

この部分はまだ翻訳されていないため、英語で表示している。

Truncate dimensions, then test the loss

最終変更:版2026.11.8

Every dimension costs 4 bytes per product as a 32-bit float, and every similarity multiplies all of them, so a shorter vector is cheaper to store and to search. Matryoshka training teaches a model to put the most useful information in its first dimensions, so that the start of a vector works as a smaller embedding on its own. Kusupati et al. (2022) report embeddings up to 14 times smaller for ImageNet-1K classification at the same level of accuracy. Shortening saves storage and search, not encoding: a Matryoshka model is no faster to run, as Sentence Transformers' documentation notes. Only models trained this way are built to survive it, and a shortened vector isn't unit length, so normalize it again, as the model card's own example does.

The store's encoder has a model card that documents shortening to 256 dimensions. lib.recorded.embeddings(dims=...) keeps the first dims numbers of each vector and normalizes again. Score each length on the train queries:

truncate-embeddings.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 合格
"""Scores semantic retrieval on the train queries at 768, 256 and 128 dimensions, with each product's float32 size. The prose's point rests on the nDCG@10 line for each length, set against 768 dimensions."""from lib.metrics import evaluatefrom lib.semantic import SemanticSearchfrom lib.store import load_queries, qrelstrain, judgments = load_queries("train"), qrels("train")       # choose on train queries# dims keeps the first dims numbers of every vector and rescales each to length 1 (lib.recorded).for dims in [768, 256, 128]:    semantic = SemanticSearch(dims=dims)    scores = evaluate(semantic.run(train, k=100), judgments)    # semantic.products holds the product vectors, not the product table    print(f"{dims} dimensions, float32 bytes per product: {semantic.products[0].nbytes:,}")    print(f"{dims} dimensions, ndcg@10: {scores['ndcg@10'].mean():.3f}")    print(f"{dims} dimensions, judged@10: {scores['judged@10'].mean():.3f}")    print()
出力python truncate-embeddings.py
768 dimensions, float32 bytes per product: 3,072
768 dimensions, ndcg@10: 0.772
768 dimensions, judged@10: 0.964

256 dimensions, float32 bytes per product: 1,024
256 dimensions, ndcg@10: 0.762
256 dimensions, judged@10: 0.952

128 dimensions, float32 bytes per product: 512
128 dimensions, ndcg@10: 0.680
128 dimensions, judged@10: 0.874

Shortening to 256 dimensions cuts a product from 3,072 bytes to 1,024, and moves mean nDCG@10 from 0.772 to 0.762.

At 128 dimensions the mean is 0.680, with judged@10 at 0.874 against 0.964, so part of that fall could come from products nobody judged.

Is the change real, or noise among 50 queries? Test it with lib.significance.compare, which runs chapter 3's permutation test and its paired t-test. It also reports a bootstrap interval. These comparisons use train queries, so they tune the vector size; they aren't evidence about the test queries:

truncation-paired-test.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 · scipy 1.18.1 合格
"""Tests the change in nDCG@10 from shortening to 256 and 128 dimensions against the full vectors, on the train queries, with a permutation test, a paired t-test and a bootstrap interval. The prose's point rests on the permutation, interval and effect-size lines for each length."""from lib.metrics import evaluatefrom lib.semantic import SemanticSearchfrom lib.significance import comparefrom lib.store import load_queries, qrels# Train queries: these comparisons tune the vector size. They are not evidence about the test queries.train, judgments = load_queries("train"), qrels("train")def ndcg(dims):    return evaluate(SemanticSearch(dims=dims).run(train, k=100), judgments)["ndcg@10"]def p_value(p):    return "below 0.001" if p < 0.001 else f"{p:.3f}"full = ndcg(768)for dims in [256, 128]:    # compare's default interval is 95%    result = compare(ndcg(dims), full)                    # paired by train query: shortened minus full    print(f"{dims} vs 768, mean difference in ndcg@10: {result['mean difference']:.3f}")    print(f"{dims} vs 768, queries worse: {result['losses']}")    print(f"{dims} vs 768, permutation test p: {p_value(result['permutation p'])}")    print(f"{dims} vs 768, paired t-test p: {p_value(result['t-test p'])}")    print(f"{dims} vs 768, 95% bootstrap interval: {result['ci low']:.4f} to {result['ci high']:.4f}")    print(f"{dims} vs 768, interval excludes 0: {'yes' if result['ci high'] < 0 or result['ci low'] > 0 else 'no'}")    print(f"{dims} vs 768, effect size: {result['effect size']:.3f}")    print()
出力python truncation-paired-test.py
256 vs 768, mean difference in ndcg@10: -0.010
256 vs 768, queries worse: 24
256 vs 768, permutation test p: 0.076
256 vs 768, paired t-test p: 0.080
256 vs 768, 95% bootstrap interval: -0.0211 to -0.0002
256 vs 768, interval excludes 0: yes
256 vs 768, effect size: -0.253

128 vs 768, mean difference in ndcg@10: -0.092
128 vs 768, queries worse: 33
128 vs 768, permutation test p: below 0.001
128 vs 768, paired t-test p: below 0.001
128 vs 768, 95% bootstrap interval: -0.1569 to -0.0565
128 vs 768, interval excludes 0: yes
128 vs 768, effect size: -0.547

At 128 dimensions the loss is clear. The mean difference is -0.092, and 33 train queries get worse. The permutation test gives p below 0.001, and the 95% interval, from -0.1569 to -0.0565, excludes 0.

At 256 dimensions the evidence is borderline, because the tests and the interval disagree. The mean difference is -0.010; the permutation test gives p = 0.076 and the t-test p = 0.080, yet the interval, from -0.0211 to -0.0002, excludes 0 by a hair. The effect size, -0.253 at 256 dimensions against -0.547 at 128, says any loss at 256 is the smaller of the two.

For the store, the decision follows from its size, not from the test. Its full vectors take 36.9 MB, so there is little to save; it keeps all 768 dimensions, and its test score doesn't change. A catalog of hundreds of millions of products would weigh a cut from 3,072 to 1,024 bytes per product against a loss that this many train queries can't pin down, and would judge more queries before it decided.

Working rule: Shortlist models on licence, language, size and domain, and treat benchmark ranks as a hint. Decide between models, or between vector sizes, on your own train queries with a paired test, and read the interval as well as the p-value. Shorten vectors only for a model trained for it, normalize after shortening, and only when memory or search time requires it.

本番環境で起きる問題

最終変更:版2026.11.8

類似度は確信度ではない。 最良の類似度が低いときは何も表示しない、という方針は魅力的に見える。しかし、類似度はクエリごとに同じ尺度にはなっていない。

similarity-scale.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 合格
"""Finds the test query with the lowest best similarity and the one with the highest 10th similarity, and counts the products above a 0.5 cutoff. The prose's point rests on the two similarity lines and the cutoff counts."""import numpy as npfrom lib.recorded import embeddingsfrom lib.semantic import SemanticSearchfrom lib.store import load_queries, qrelsproduct_ids, product_vectors, query_ids, query_vectors = embeddings()test = load_queries("test")rows = np.isin(query_ids, test.index)similarity = np.sort(query_vectors[rows] @ product_vectors.T, axis=1)[:, ::-1]   # best first, per queryqueries = test.loc[query_ids[rows], "query"].to_numpy()weakest = similarity[:, 0].argmin()                    # the query whose best match is least similarstrongest = similarity[:, 9].argmax()                  # the query whose 10th match is most similarprint("first query, lowest best similarity:", queries[weakest])print(f"its best similarity: {similarity[weakest, 0]:.3f}")judgments = qrels()[query_ids[rows][weakest]]top = SemanticSearch().search(query_ids[rows][weakest], k=10).indexprint("exact products in its top 10:", sum(judgments.get(product_id) == 2 for product_id in top))print("second query, highest 10th similarity:", queries[strongest])print(f"its 10th similarity: {similarity[strongest, 9]:.3f}")cutoff = 0.5  # an illustrative cutoff; nothing about the model sets itprint()print(f"products above {cutoff} for the first query:", (similarity[weakest] > cutoff).sum())print(f"products above {cutoff} for the second query:", (similarity[strongest] > cutoff).sum())print(f"test queries with no product above {cutoff}:", (similarity[:, 0] <= cutoff).sum())
出力python similarity-scale.py
first query, lowest best similarity: liberty hardware francisco
its best similarity: 0.436
exact products in its top 10: 5
second query, highest 10th similarity: kohler whitehaven farmhouse kitchen sink
its 10th similarity: 0.647

products above 0.5 for the first query: 0
products above 0.5 for the second query: 103
test queries with no product above 0.5: 2

テストクエリのうち両極端の二つがそれを示している。liberty hardware francisco は最良の一致が最も弱く 0.436 であるが、上位 10 件には正解の商品が 5 件含まれている。一方、kohler whitehaven farmhouse kitchen sink は 10 番目の一致の類似度が最も高く、0.647 である。

0.5 を閾値にすると、最初のクエリでは 0 件、2 番目のクエリでは 103 件の商品が残り、2 件のテストクエリでは何も残らない。また、意味検索が空の結果を返すことはない。どの商品も持っていないコードに対しても、上位 10 件は満たされて返ってくる。クエリに一致するものがないという判断は、別の方法で行う。例えば、字句検索がそのクエリの希少な語を一つも見つけられない場合に、一致なしと判断する。

ベクトルの不一致は、エラーを出さずに失敗する。 クエリのベクトルと商品のベクトルは、同じモデル、同じリビジョン、同じプロンプトから作らなければならない。二つのモデルのベクトルは、異なる空間にある数値である。両者の次元が同じであれば、比較してもエラーは起きない。Sentence Transformers の encode_query は、モデルに事前定義されたクエリ用プロンプトがあれば、それをクエリに付加する(API リファレンス)。実際のクエリを埋め込むコードが、インデックス作成時のコードと同じ処理をしていることを確認する。

インデックスはカタログに自動では追従しない。 faiss の wikiによれば、faiss の HNSW はベクトルの削除をサポートしていない。そのため、販売終了になった商品は、結果から除外するか再構築するまでグラフに残る。Douze ら (2024)は、faiss はデータベースではなく、同時書き込み、シャーディング、トランザクションを持たないと述べている。厳密検索に対する再現率は、一つのカタログと一組の設定に固有のものであるから、再構築のたびに測り直す。

長いテキストは途中で切り捨てられる。 モデルが読むのは最大 512 トークンまでであり、Sentence Transformers の API リファレンスは、それより長い入力は切り詰められると述べている。この店で最も長い商品は 201 トークンであるが、説明文が長いカタログでは、上限を超えた部分をモデルは読まない。recordings/embeddings.py が行っているように、埋め込む前にトークン数を数える。

タイプミスはベクトルを動かす。 Zhuang と Zuccon (2021)は、誤字を含むデータで学習していない BERT 系の検索モデルでは、タイプミスによって検索とランキングの有効性が大きく低下しうることを見いだした。この店のクエリのうち 4 件にタイプミスがあり、それぞれに正しい綴りのプローブがある。修正済みのプローブは元のクエリと同じものを求めているので、そのクエリの判定で採点できる。

spelling-probes.py Python 3.12.15 · bm25s 0.3.16 · numpy 2.5.5 · pandas 3.1.1 · scipy 1.18.1 合格
"""Scores each correctly spelled probe with the judgments of the query it corrects, for semantic retrieval and BM25, as typed and as corrected. The prose's point rests on the semantic nDCG@10 lines for each spelling and on the smallest-p line."""from lib.lexical import LexicalSearchfrom lib.metrics import ndcg_at_kfrom lib.recorded import probesfrom lib.semantic import SemanticSearchfrom lib.store import load_products, load_queries, qrels# Each spelling probe is the correctly spelled text of one store query (of_query_id), embedded by the same# model. It asks for the same thing, so it is scored with that query's judgments. A diagnosis, not a test.probe_queries, probe_vectors = probes()products, queries, judgments = load_products(), load_queries(), qrels()semantic, bm25 = SemanticSearch(), LexicalSearch(products)spelling = probe_queries[probe_queries["kind"] == "spelling"]for row, probe in spelling.iterrows():    query_id, pair = int(probe["of_query_id"]), probe["pair"]    typed, corrected = semantic.search(query_id, k=10).index, semantic.search_vector(probe_vectors[row], k=10).index    print(f"{pair}: {queries.at[query_id, 'query']} | {probe['query']} ({queries.at[query_id, 'split']} query)")    print(f"{pair}, similarity of the two spellings: {semantic.query_vectors[query_id] @ probe_vectors[row]:.3f}")    print(f"{pair}, products in both semantic top 10s:", len(set(typed) & set(corrected)))    print(f"{pair}, semantic ndcg@10 as typed: {ndcg_at_k(list(typed), judgments[query_id], 10):.3f}")    print(f"{pair}, semantic ndcg@10 corrected: {ndcg_at_k(list(corrected), judgments[query_id], 10):.3f}")    for name, text in [("as typed", queries.at[query_id, "query"]), ("corrected", probe["query"])]:        print(f"{pair}, bm25 ndcg@10 {name}: {ndcg_at_k(list(bm25.search(text, k=10).index), judgments[query_id], 10):.3f}")    print()print("spelling probes:", len(spelling))# a sign-flip test on n queries has 2 ** n outcomes, and its two most extreme give the smallest two-sided pprint(f"smallest two-sided permutation p-value possible with {len(spelling)} queries: {2 / 2 ** len(spelling):.3f}")
出力python spelling-probes.py
ligth: ligth bulb | light bulb (train query)
ligth, similarity of the two spellings: 0.710
ligth, products in both semantic top 10s: 5
ligth, semantic ndcg@10 as typed: 1.000
ligth, semantic ndcg@10 corrected: 1.000
ligth, bm25 ndcg@10 as typed: 0.945
ligth, bm25 ndcg@10 corrected: 0.945

foutains: foutains with brick look | fountains with brick look (train query)
foutains, similarity of the two spellings: 0.668
foutains, products in both semantic top 10s: 0
foutains, semantic ndcg@10 as typed: 0.353
foutains, semantic ndcg@10 corrected: 0.645
foutains, bm25 ndcg@10 as typed: 0.356
foutains, bm25 ndcg@10 corrected: 0.507

desins: twin over full bunk beds cool desins | twin over full bunk beds cool designs (train query)
desins, similarity of the two spellings: 0.829
desins, products in both semantic top 10s: 6
desins, semantic ndcg@10 as typed: 0.565
desins, semantic ndcg@10 corrected: 0.616
desins, bm25 ndcg@10 as typed: 0.488
desins, bm25 ndcg@10 corrected: 0.488

ice-an: refrigerator with ice an water in door | refrigerator with ice and water in door (test query)
ice-an, similarity of the two spellings: 0.975
ice-an, products in both semantic top 10s: 9
ice-an, semantic ndcg@10 as typed: 0.807
ice-an, semantic ndcg@10 corrected: 0.829
ice-an, bm25 ndcg@10 as typed: 0.501
ice-an, bm25 ndcg@10 corrected: 0.501

spelling probes: 4
smallest two-sided permutation p-value possible with 4 queries: 0.125

タイプミスが一つあるだけで、クエリの意味検索の上位 10 件全体が変わることがある。学習用クエリ foutains with brick look では、入力どおりの綴りと修正後の綴りで共通する商品は 0 件であり、nDCG@10 は 0.353 から 0.645 に変わる。BM25 も 0.356 から 0.507 に動く。

他のプローブでは、スコアの変化はより小さい。その中に一つだけあるテストクエリ refrigerator with ice an water in door では、0.807 から 0.829 になる。これらからは、何らかの傾向を確立することはできない。プローブが 4 件では、取りうる両側並べ替え検定の p 値の最小値は 0.125 である。

実践ルール: 固定の類似度でフィルタしない。すべてのベクトルにモデル、リビジョン、プロンプトを記録し、クエリを埋め込む場所でそれらを確認する。カタログの変化に合わせてインデックスを再構築して測り直し、埋め込む前にトークン数を数える。ログにタイプミスが見られる場合は、クエリを埋め込む前に、各単語の近傍を確認するスペル修正器で綴りを修正する(第 8 章)。

この章で下した決定

最終変更:版2026.11.8

訓練クエリで測定できる選択については、測定して決めた。

  • モデル: Snowflake/snowflake-arctic-embed-m-v1.5。recordings/embeddings.py が固定したリビジョンで一度だけ記録する。オープンで Apache-2.0 のライセンスの下にあり、小型で、短いベクトル向けに訓練されている。クエリにはモデルカードが求めるプレフィックスを付け、商品には付けない。
  • 類似度: 単位長ベクトルの内積であり、これはコサイン類似度に等しい。ベクトルは一度正規化し、変更を加えるたびに再度正規化する。
  • 検索: lib.vectors.exact_search を通じて numpy で行う厳密検索とする。ストアのベクトルが 36.9 MB を占めるにすぎないためである。これはあらゆるインデックスの正解にもなる。
  • カタログが厳密検索の手に負えなくなった場合: M = 32 と、efSearch を 144 にした faiss の HNSW を用いる。この値は、訓練時のスイープで、すべてのクエリが真の上位10件のうち少なくとも9件を保つ最小の設定である。テストクエリでは、厳密検索に対するこのインデックスの recall@10 は 1.000 である。
  • インデックスの recall は厳密検索を基準に測る。 訓練クエリを用い、パイプラインが使う k で、平均だけでなく最も悪いクエリも読む。
  • 次元数: 768 次元すべてを使う。256 次元に縮めれば、商品1件あたりのサイズは 3,072 バイトから 1,024 バイトに減るが、その代償は訓練クエリでは見極められず、ストアには節約できる余地がほとんどない。
  • プローブクエリは、判定では検証できないものを検証する。 否定、コード、綴りなどがそれである。判定付きのクエリとは別に、上位10件どうしを比較する。同じことを尋ねている修正済みの綴りだけは、元のクエリの判定で採点する。
  • 既知の弱点は他の段階に任せる。 コードは語彙検索に、正確なサイズは属性フィールドに任せ、トークナイザーが数字を保持するようになれば語彙検索に任せる(第5章、第8章)。どちらについても、混ぜ合わせるのではなく、判定に使うフィルターとする(第7章)。否定は属性フィルターと再ランキングに任せる。
  • この章が示したこと: インデックスの良さは、出荷するカタログと設定で厳密検索に対して測定した再現率によって決まる。プローブにより、埋め込みはほとんどのサイズを扱えるが、コードや、ラグとコーヒーテーブルの組にある否定は扱えないことが分かった。また、いくつかの組は何も証明しない。

この版における本章

第2026.11.10版2026年11月19日時点で例が通過この章の16 例件のうち16件が合格

英語版の書籍(版2026.11.10)から翻訳した。

この節を引用する

形式

いずれの形式も、本訳書が従う英語版の書籍を出典として示す。

いずれの形式もこの版を明示している。そのリンクは現行版を開き、節の見出しには本文が最後に変更された日付が示される。アクセス日と取得日は、これを開いた日である。

このページの内容

全11章中の第6章: セマンティック検索

  1. Turn text into embeddings
  2. Compare embeddings with cosine similarity
  3. Find the exact nearest neighbours with NumPy
  4. Trade recall for speed with HNSW
  5. テストクエリでインデックスを検証する
  6. Check queries that contain digits
  7. 商品コードの重みは小さいと見込む
  8. 否定表現に対する埋め込みを検証する
  9. 埋め込みモデルを選ぶ
  10. ベンチマークは候補を絞る材料として読む
  11. Truncate dimensions, then test the loss
  12. 本番環境で起きる問題
  13. この章で下した決定

ページの先頭に戻るすべての章