語意檢索

閱讀約需 85 分鐘 分鐘,13 節 中共有 16 個範例。

版次 2026.11.10 範例於 2026年11 月 19 通過測試 132 個範例中有 132 個通過 變更內容

由 AI 翻譯。經人工以英文驗證。 我們如何製作書籍

譯自英文書,版次 2026.11.10。 閱讀原文

讀完本章,你可以根據儲存好的嵌入向量,為商品目錄建立語意檢索:以餘弦相似度比較向量,用 NumPy 精確找出查詢最近的商品,在商品目錄大到無法使用精確搜尋時改用近似索引,並衡量該索引遺漏了什麼,還能找出嵌入表現不佳的查詢。你也可以用自己的相關性判斷來測試並選擇向量維度,並以同樣的方式比較不同的模型。

對於 48 inch kitchen hood,語意檢索會回傳其他寬度的排油煙機,對於 rug that is not round,則會回傳圓形地毯。然而在測試查詢上,它的平均 nDCG@10 為 0.830。它領先 BM25(得分為 0.725)的幅度,是第 3 章所確立的。本書一直把它當作一個類別來使用,即 lib.semantic.SemanticSearch。第 1 章為這家商店建立了它,該商店的商品目錄從沙發到浴簾無所不包,使用的向量由某個模型事先一次算好。在商店依賴它之前,你需要知道這些向量包含了什麼、對規模大得多的商品目錄進行搜尋要付出多少成本,以及它們會在哪些查詢上出錯。本章親手建立精確搜尋,以它為基準衡量近似索引,並在商店的查詢與相關性判斷上測試兩者。

在測試查詢上,本章規劃了 2 項比較,兩者都在測試索引的一節中進行。每項比較都讓精確搜尋(將查詢與每項商品比較)對上 HNSW(一種近似索引,只將查詢與其中一小部分商品比較):一次使用 faiss 的預設設定,一次使用在訓練查詢上選出的設定。每項比較都須符合本書的規則,此處為 p 值低於 0.025。本章中對測試查詢所做的其他檢視,一律屬於探索性質。

此部分尚未翻譯,因此以英文呈現。

Turn text into embeddings

最近一次變動於第 2026.11.10 版

An embedding places texts with similar meanings near each other; this section opens how. The model that computes an embedding is an encoder, trained so that texts with similar meanings get vectors that point in similar directions, whether or not they share words. Every text gets the same count of numbers, the embedding's dimension, whether it's a three-word query or a product's whole description. One number means nothing on its own; only comparisons between whole vectors do.

Search uses the encoder as a bi-encoder: it encodes each product by itself, once, ahead of time, and each query by itself when the shopper types it. Comparing a query with a product is then arithmetic on two vectors rather than a run of the model, which is what makes it cheap enough to compare one query with a whole catalog. Chapter 1 gives the size of the difference from the Sentence-BERT paper: about 65 hours for BERT to read every pair of 10,000 sentences, against about 5 seconds to embed each sentence once and compare. A model that reads a query and a product together is a cross-encoder, and chapter 1 uses one to rerank a shortlist.

The store's encoder is Snowflake/snowflake-arctic-embed-m-v1.5, an open model with 109 million parameters under the Apache-2.0 licence (model card). Merrick et al. (2024) report that its family's training kept only documents identified as mainly English. We chose it because anyone may use it commercially, it's small, and it was trained so that its vectors can be shortened, which the section on dimensions tests. Examples read a recording instead: vectors that the script recordings/embeddings.py computed once, at a pinned revision, and saved at unit length, so every example sees the same numbers. The model card asks for a prefix on queries only, so the script adds it to each query and embeds products as they are. Examples read the vectors through lib.recorded.embeddings():

inspect-recorded-embeddings.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 通過
"""Loads the recorded vectors and shows the lengths lib.recorded stores and rescales, and how close two shopper queries are. The prose's point about rounding rests on the stored- and rescaled-length lines, and its point about nearby queries on the median and nearest-query lines."""import reimport numpy as npfrom lib.recorded import RECORDED, embeddings, manifestfrom lib.store import load_queries# unit-length rows, one per ID; the queries carry the model's query prefixproduct_ids, product_vectors, query_ids, query_vectors = embeddings()recording = manifest("embeddings")print("embedding model:", recording["model"])print("query prompt:", recording["prompts"]["query"])print(f"products embedded: {len(product_ids):,}")print("dimensions:", product_vectors.shape[1])print("stored as:", recording["dtype"])print("longest product text, tokens:", recording["longest_product_tokens"])print("model's input limit, tokens:", recording["max_seq_length"])# row 0 is product 1; [0, :4] takes its first four numbersprint(f"product {product_ids[0]}, first four numbers:", ", ".join(f"{x:.3f}" for x in product_vectors[0, :4]))stored = np.load(RECORDED / "embeddings" / "products.npy").astype(np.float32)   # as saved, before lib.recordedstored_lengths, lengths = np.linalg.norm(stored, axis=1), np.linalg.norm(product_vectors, axis=1)print(f"stored vector lengths: {stored_lengths.min():.5f} to {stored_lengths.max():.5f}")print(f"lengths after lib.recorded rescales: {lengths.min():.5f} to {lengths.max():.5f}")# How close are two shopper queries? Compare every query with every other one.queries = load_queries().loc[query_ids, "query"].tolist()similarity = query_vectors @ query_vectors.T  # unit length, so the dot product is the cosinenp.fill_diagonal(similarity, np.nan)               # leave out each query's similarity to itselfprint(f"median similarity of two queries: {np.nanmedian(similarity):.3f}")for query in ["whimsical solar lights", "wall design shelf"]:    row = queries.index(query)    nearest = int(np.nanargmax(similarity[row]))    # whole words as typed, with no stemming, so `lights` and `light` differ    shared = set(re.findall(r"\w+", query)) & set(re.findall(r"\w+", queries[nearest]))    print()    print(f"{query}, nearest other query:", queries[nearest])    print(f"{query}, similarity: {similarity[row, nearest]:.3f}")    print(f"{query}, words in common:", len(shared))
輸出python inspect-recorded-embeddings.py
embedding model: Snowflake/snowflake-arctic-embed-m-v1.5
query prompt: Represent this sentence for searching relevant passages:
products embedded: 12,000
dimensions: 768
stored as: float16
longest product text, tokens: 201
model's input limit, tokens: 512
product 1, first four numbers: -0.033, 0.051, 0.111, -0.012
stored vector lengths: 0.99994 to 1.00006
lengths after lib.recorded rescales: 1.00000 to 1.00000
median similarity of two queries: 0.291

whimsical solar lights, nearest other query: outdoor light fixtures
whimsical solar lights, similarity: 0.637
whimsical solar lights, words in common: 0

wall design shelf, nearest other query: kitchen storage shelves
wall design shelf, similarity: 0.679
wall design shelf, words in common: 0

Each of the 12,000 products is a vector of 768 numbers; product 1's begins -0.033, 0.051, 0.111, -0.012, and none of those numbers means anything alone. The recording stores each number as a float16, in 2 bytes, and lib.recorded widens them to 32-bit floats when it loads them. That rounding moves the stored lengths slightly off 1: they run from 0.99994 to 1.00006. So lib.recorded also scales each vector back to length 1, and the lengths it returns run from 1.00000 to 1.00000. The store's encoder reads text as tokens, word pieces from its own vocabulary, not the terms of chapter 5's tokenizer. The longest product text is 201 tokens, inside the model's limit of 512 tokens, so the model read every product in full.

The median similarity between two of the store's queries is 0.291, on a scale where 1 means two vectors point the same way (cosine similarity). Yet whimsical solar lights finds outdoor light fixtures at 0.637, and wall design shelf finds kitchen storage shelves at 0.679, with 0 and 0 words in common: lights and light are different strings, and so are shelves and shelf. The baseline BM25 doesn't stem words, so it sees nothing shared in either pair. Chapter 5's stemming would match lights with light, but not shelves with shelf.

Working rule: Embed products ahead of time and queries at search time, with the same model at the same revision, and follow the model card on prompts: here, a prefix on queries only. Record the model, revision and prompts next to the vectors, as each manifest.json in examples/data/recorded/ does. Check that your longest product fits the model's input limit before you trust its embedding.

此部分尚未翻譯,因此以英文呈現。

Compare embeddings with cosine similarity

最近一次變動於第 2026.11.10 版

To rank products for a query, you need one number per pair that says how alike two vectors are. Cosine similarity is the cosine of the angle between two vectors: 1 when they point the same way, 0 at right angles and −1 when they point in opposite directions. It ignores how long the vectors are. The dot product multiplies two vectors number by number and adds up the results, so it equals the cosine times both lengths. Scaling every vector to length 1, called normalizing, makes the two the same number, as Sentence Transformers states. The dot product is then the cheaper choice, because it skips computing the lengths.

dot(q, d)    = q1 × d1 + q2 × d2 + ... + qn × dn
length(q)    = √(q1² + q2² + ... + qn²)
cosine(q, d) = dot(q, d) / (length(q) × length(d))

To see the formula at work, take the test query clear shower curtain and its most similar product: compute the dot product and the cosine by hand, and check the cosine with scikit-learn 1.9.1. Then break the one assumption that makes them agree. Keep the first 256 of each vector's 768 numbers, a shortcut that the section on dimensions tests, but skip the rescaling, and rank every query's products both ways:

cosine-by-hand.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 · scikit-learn 1.9.1 · scipy 1.18.1 通過
"""Computes the cosine of one test query and its best product by hand and with scikit-learn, then ranks every query's products by dot product and by cosine after the vectors are shortened. The prose's point rests on the line that counts queries whose top 10 changes."""import numpy as npfrom sklearn.metrics.pairwise import cosine_similarityfrom lib.recorded import embeddingsfrom lib.store import load_productsproducts = load_products()product_ids, product_vectors, query_ids, query_vectors = embeddings()query = query_vectors[list(query_ids).index(246)]           # clear shower curtainbest = int(np.argmax(product_vectors @ query))product = product_vectors[best]print("product:", products.at[product_ids[best], "name"])# cosine = dot product / (length of q * length of d)dot = query @ productcosine = dot / (np.linalg.norm(query) * np.linalg.norm(product))print(f"dot product: {dot:.3f}")print(f"cosine by hand: {cosine:.3f}")print(f"cosine from scikit-learn: {cosine_similarity([query], [product])[0, 0]:.3f}")# Keep the first 256 numbers of each vector and don't rescale: the lengths are no longer 1.short_products, short_queries = product_vectors[:, :256], query_vectors[:, :256]lengths = np.linalg.norm(short_products, axis=1)print(f"shortened product lengths: {lengths.min():.3f} to {lengths.max():.3f}")# stable sort: tied scores keep their stored orderby_dot = np.argsort(-(short_queries @ short_products.T), axis=1, kind="stable")[:, :10]# only the products are rescaled: a query's length scales all its scores alike, so it can't change their orderby_cosine = np.argsort(-(short_queries @ (short_products / lengths[:, None]).T), axis=1, kind="stable")[:, :10]# compares the sets: a new order within the same 10 doesn't countchanged = sum(set(dot_top) != set(cosine_top) for dot_top, cosine_top in zip(by_dot, by_cosine))print("queries:", len(query_ids))print("queries whose top 10 changes:", changed)
輸出python cosine-by-hand.py
product: woolley vinyl solid color single shower curtain
dot product: 0.577
cosine by hand: 0.577
cosine from scikit-learn: 0.577
shortened product lengths: 0.706 to 0.778
queries: 80
queries whose top 10 changes: 61

On the store's unit-length vectors, the dot product is 0.577, the cosine by hand is 0.577, and scikit-learn's cosine_similarity gives 0.577, so the three agree.

Shortening breaks that. The shortened vectors have lengths from 0.706 to 0.778, and for 61 of the 80 queries, ranking by dot product gives a different top 10 from ranking by cosine. A query's own length scales all of its scores by the same factor, so it can't change their order. A product's length can: it multiplies that product's score for every query, a boost the shopper never asked for.

Some libraries report a distance instead of a similarity. faiss's L2 indexes return the squared Euclidean distance, and its inner product equals cosine similarity only when the vectors are normalized, according to the faiss wiki. For unit vectors, the squared distance is 2 − 2 × cosine, so it ranks products in the same order as cosine, smallest distance first.

Working rule: Normalize every vector once, when you store it, and rank by dot product. Normalize again after anything that changes a vector, such as shortening it or averaging several. If a model's documentation says it was trained for the dot product of unnormalized vectors, follow the documentation instead, and never mix the two in one index.

此部分尚未翻譯,因此以英文呈現。

Find the exact nearest neighbours with NumPy

最近一次變動於第 2026.11.9 版

Retrieval by embeddings is a nearest-neighbour search: find the k products whose vectors are most similar to the query's. Exact search compares the query with every product and keeps the best k, so it never misses one. With unit-length vectors, comparing one query with every product is one matrix–vector product, and comparing many queries is one matrix product, with a row per query and a column per product. What's left is to find the k highest scores in each row. Sorting every row does more work than needed. NumPy's argpartition moves the k highest scores to one side without ordering either side, and NumPy's documentation leaves that order undefined, so you sort only those k.

Chapter 1's semantic search took its rankings from lib.semantic.SemanticSearch. Build its core by hand for the 30 test queries with NumPy 2.5.5, then check it twice: against lib.semantic, and against faiss's IndexFlatIP, which computes the inner product with every vector. The faiss guidelines name two indexes that guarantee exact results, IndexFlatIP and IndexFlatL2:

exact-search-numpy.py Python 3.12.15 · faiss-cpu 1.16.0 · numpy 2.5.5 · pandas 3.1.1 通過
"""Builds exact search from one matrix product, then checks its top 10 against lib.semantic and faiss's IndexFlatIP. The prose's point rests on the two 'same top 10' lines, which show the hand-built search matches both."""import faissimport numpy as npfrom lib.metrics import evaluatefrom lib.recorded import embeddingsfrom lib.semantic import SemanticSearchfrom lib.store import load_queries, qrelsproduct_ids, product_vectors, query_ids, query_vectors = embeddings()test = load_queries("test").indexrows = np.isin(query_ids, test)def top_k(scores, k):    """Column indices of the k highest scores in each row, best first."""    # argpartition's kth is 0-based: with k - 1, the first k positions hold the k highest scores, in no order    candidates = np.argpartition(-scores, k - 1, axis=1)[:, :k]          # the top k, in no order    # stable sort: tied scores keep their candidate order    order = np.argsort(-np.take_along_axis(scores, candidates, axis=1), axis=1, kind="stable")    return np.take_along_axis(candidates, order, axis=1)scores = query_vectors[rows] @ product_vectors.T      # one row per test query, one column per producttop = top_k(scores, 10)print("similarities computed:", f"{scores.size:,}")print("similarities per query:", f"{scores.shape[1]:,}")semantic = SemanticSearch()# two lists match only if the same products come in the same ordersame = sum(list(product_ids[found]) == list(semantic.search(query_id, 10).index)           for query_id, found in zip(query_ids[rows], top))print("same top 10 as lib.semantic:", same, "of", rows.sum())flat = faiss.IndexFlatIP(product_vectors.shape[1])     # faiss's exact index: inner product with every productflat.add(product_vectors)_, found = flat.search(query_vectors[rows], 10)# the same products in the same order, row by rowprint("same top 10 as faiss IndexFlatIP:", (found == top).all(axis=1).sum(), "of", rows.sum())run = {query_id: list(product_ids[row]) for query_id, row in zip(query_ids[rows], top)}print(f"ndcg@10 of the exact run: {evaluate(run, qrels('test'))['ndcg@10'].mean():.3f}")
輸出python exact-search-numpy.py
similarities computed: 360,000
similarities per query: 12,000
same top 10 as lib.semantic: 30 of 30
same top 10 as faiss IndexFlatIP: 30 of 30
ndcg@10 of the exact run: 0.830

The hand-built search agrees with both checks: it returns the same top 10 as lib.semantic for 30 of 30 test queries, and as faiss for 30 of 30, so its run scores the same mean nDCG@10, 0.830. One matrix product computed every similarity: 360,000 of them, 12,000 per query. The book's shared module, lib.vectors, keeps this top_k for later examples and wraps it in exact_search.

Whether exact search is enough comes down to its cost. Its work grows in step with the catalog: one similarity per product, for every query. Its memory is the vectors themselves:

exact-search-memory.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 通過
"""Prints the bytes one product's float32 vector takes, and the memory the catalog's vectors use, for the store and for a million products. The prose's point rests on the MB line for the store's catalog and the GB line for a million products."""from lib.recorded import embeddings# Exact search keeps every product vector in memory, as 32-bit floats.product_ids, product_vectors, _, _ = embeddings()print(f"products: {len(product_ids):,}")print(f"float32 bytes per product: {product_vectors[0].nbytes:,}")# decimal MB and GB: a megabyte is a million bytes, a gigabyte a billionprint(f"product vectors in memory, MB: {product_vectors.nbytes / 1e6:.1f}")print(f"the same for 1,000,000 products, GB: {1_000_000 * product_vectors[0].nbytes / 1e9:.1f}")
輸出python exact-search-memory.py
products: 12,000
float32 bytes per product: 3,072
product vectors in memory, MB: 36.9
the same for 1,000,000 products, GB: 3.1

Each product takes 3,072 bytes as 32-bit floats, so the store's catalog takes 36.9 MB, and a million products would take 3.1 GB. Douze et al. (2024) advise deciding first whether an index is needed at all, because in some cases a brute-force search is the best option. We judge the store's catalog to be such a case.

Working rule: We recommend exact search until it no longer fits in memory or meets your latency budget, measured with your own catalog on your own hardware. It has no settings to tune and never misses a neighbour, so it is also the ground truth for any approximate index. Store the normalized vectors as one 32-bit matrix, take the top k with argpartition, and sort only those.

以 HNSW 用召回率換取速度

最近一次變動於第 2026.11.10 版

當精確搜尋的成本過高時,近似最近鄰(approximate nearest-neighbour,ANN)索引會略過大部分比較,代價是可能漏掉部分真正的最近鄰。ANN 索引的品質以其相對於精確搜尋的 recall@k 衡量:也就是精確搜尋的前 k 名中,索引同樣回傳的比例。這與第 3 章的 recall@k不同,後者是將排名與判定結果相比較。相對於精確搜尋召回率完美的索引,回傳的結果與精確搜尋完全相同,無論結果好壞。

HNSW 是 hierarchical navigable small world 的縮寫,是一種圖索引。每項商品都是一個節點,連結到若干最近鄰,並分布在多個層級中:最底層包含所有商品,越往上的層級包含的商品越少。搜尋從最頂層開始,移動到與查詢最接近的相連節點;當無法更接近時就下降一層;到了最底層,則先探索一份候選清單,再回傳最佳的 k 個(Malkov and Yashunin, 2018)。在 faiss 中,M 設定每個節點的連結數。連結越多,越能找到更多真正的鄰居,但也會使用更多記憶體(faiss wiki)。efConstruction 是建構時的擴展因子,決定建構時考慮多少候選;efSearch 是搜尋時的擴展因子,決定一次搜尋考慮多少候選。在 faiss 1.16.0 中,兩者的預設值分別為 40 與 16,而且圖的建構是確定性的,因為 add() 使用的唯一建構方式是確定性的(HNSW.h;changelog)。因此這裡的輸出每次都完全相同。計時結果因機器而異,所以範例改為計算工作量:faiss 會計算每次搜尋所算出的距離數(HNSW.h),對這個索引而言就是相似度。這個數字未計入沿連結移動與讀取記憶體的成本,因此只是速度的參考,而非精確的量測。

這家商店並不需要索引,但擁有數百萬項商品的商店就需要;而小到能逐一檢查每個答案的商品目錄,正適合用來學習這項取捨。以 hnsw_index(product_vectors, links=32) 為商店的 12,000 項商品建立 HNSW,這是 lib.vectors 對 M = 32 的稱呼。接著在訓練查詢上,將 efSearch 從 16 掃描到 256:

hnsw-efsearch-sweep.py Python 3.12.15 · faiss-cpu 1.16.0 · numpy 2.5.5 · pandas 3.1.1 通過
"""Builds HNSW with M = 32 and sweeps efSearch on the train queries against exact search, printing recall, the worst query and the work per query. The prose's point rests on the efSearch 16 lines and the two smallest-efSearch lines printed after the table."""import faissimport numpy as npfrom lib.recorded import embeddingsfrom lib.store import load_products, load_queries# EF_SEARCH_GRID runs 16 to 256 in steps of 16; WORST_RECALL is 0.9: at least 9 of the true top 10from lib.vectors import EF_SEARCH_GRID, WORST_RECALL, exact_search, hnsw_index, hnsw_search, recall_against_exactproduct_ids, product_vectors, query_ids, query_vectors = embeddings()names = load_products().loc[product_ids, "name"].str[:36].to_numpy()   # by row number, cut to 36 charactersrows = np.isin(query_ids, load_queries("train").index)                 # tune on train queries onlyqueries = load_queries().loc[query_ids[rows], "query"].to_numpy()index = hnsw_index(product_vectors, links=32)          # M = 32, and faiss's default efConstruction of 40levels = faiss.vector_to_array(index.hnsw.levels)      # how many layers each product reachesfor layer in range(1, levels.max() + 1):    print(f"products in layer {layer}: {(levels >= layer).sum():,}")exact = exact_search(query_vectors[rows], product_vectors, 10)print("\nefSearch  mean recall@10  worst  queries at 0  similarities per query")sweep = {}for ef_search in EF_SEARCH_GRID:    # similarities is faiss's count of the distances each search computed, per query, not a score    found, similarities = hnsw_search(index, query_vectors[rows], 10, ef_search)    # share of exact search's top 10 that the approximate search also found, order ignored    recall = recall_against_exact(found, exact)    sweep[ef_search] = (recall, similarities)    print(f"{ef_search:>8}  {recall.mean():>14.3f}  {recall.min():>5.1f}  {(recall == 0).sum():>12}  {similarities:>22.0f}")recall, similarities = sweep[16]                       # faiss's defaultprint(f"\nefSearch 16, recall@10 against exact: {recall.mean():.3f}")print(f"efSearch 16, similarities computed per query: {similarities:.0f}")print(f"efSearch 16, train queries with recall@10 of 0: {(recall == 0).sum()}")print("efSearch 16, those queries:", ", ".join(queries[recall == 0]))print("smallest efSearch with no query at 0:", min(ef for ef, (r, _) in sweep.items() if r.min() > 0))chosen = min(ef for ef, (r, _) in sweep.items() if r.min() >= WORST_RECALL)print(f"smallest efSearch with every query at {WORST_RECALL} or more:", chosen)print(f"chosen efSearch, recall@10 against exact: {sweep[chosen][0].mean():.3f}")print(f"chosen efSearch, similarities computed per query: {sweep[chosen][1]:.0f}\n")# the top 3 at the default efSearch, for the query that failsfound, _ = hnsw_search(index, query_vectors[rows], 3, 16)row = list(queries).index("foutains with brick look")for rank, (approximate, nearest) in enumerate(zip(found[row], exact[row]), start=1):    print(f"{rank}  hnsw: {names[approximate]:<36}  exact: {names[nearest]}")
輸出python hnsw-efsearch-sweep.py
products in layer 1: 12,000
products in layer 2: 369
products in layer 3: 10

efSearch  mean recall@10  worst  queries at 0  similarities per query
      16           0.934    0.0             2                     282
      32           0.936    0.0             2                     346
      48           0.940    0.0             2                     404
      64           0.940    0.0             2                     463
      80           0.956    0.0             1                     530
      96           0.978    0.3             0                     611
     112           0.980    0.3             0                     682
     128           0.994    0.8             0                     767
     144           0.998    0.9             0                     835
     160           0.998    0.9             0                     924
     176           0.998    0.9             0                    1020
     192           0.998    0.9             0                    1115
     208           0.998    0.9             0                    1191
     224           1.000    1.0             0                    1271
     240           1.000    1.0             0                    1349
     256           1.000    1.0             0                    1441

efSearch 16, recall@10 against exact: 0.934
efSearch 16, similarities computed per query: 282
efSearch 16, train queries with recall@10 of 0: 2
efSearch 16, those queries: foutains with brick look, sheffield home bath set
smallest efSearch with no query at 0: 96
smallest efSearch with every query at 0.9 or more: 144
chosen efSearch, recall@10 against exact: 0.998
chosen efSearch, similarities computed per query: 835

1  hnsw: dayse outdoor square cushion with fi  exact: tribeca 2 '' x 10 '' porcelain brick
2  hnsw: 3 piece wall decor set                exact: barbados 5 '' x 10 '' ceramic brick
3  hnsw: rustic autumn ivy branch              exact: kimary 0.75 '' x 1.5 '' glass brick

圖縮小得很快:最底層包含全部 12,000 項商品,下一層有 369 項,最頂層有 10 項。

平均值掩蓋了一種失敗。在預設的 efSearch 為 16 時,每次查詢計算 282 次相似度,而非 12,000 次,相對於精確搜尋的平均 recall@10 為 0.934。然而有 2 個訓練查詢完全沒有找到真正前 10 名中的任何一項。

這些查詢是 foutains with brick look, sheffield home bath set。以其中之一 foutains with brick look 為例:HNSW 回傳一個靠墊、一組牆面裝飾和一根常春藤枝條,而精確搜尋的前 3 名是磚紋磁磚。

在掃描中,使所有訓練查詢都不為 0 的最小 efSearch 是 96,但掃描表顯示其最差的查詢仍然漏掉大部分真正的前 10 名。因此改用更嚴格的規則:選擇使每個訓練查詢都至少找到其真正前 10 名中 9 項的最小 efSearch。結果是 144,平均召回率為 0.998,每次查詢計算 835 次相似度。

**實用原則:**只有在精確搜尋太慢或太大時,才使用 ANN 索引。以訓練查詢,依其相對於精確搜尋的召回率,在管線所用的 k 上進行調校,並依最差的查詢而非平均值來選擇。我們建議採用使每個查詢都至少保有其真正前 10 名中 9 項的最小 efSearch,並搭配 M = 32。本章只調校 efSearch;M 與 efConstruction 會改變圖本身,在大型目錄上同樣值得進行這樣的掃描。Aumüller et al. (2018)發現差異極大的 ANN 方法能達到相當的品質與效能取捨,所以請以自己的資料來衡量,而不是憑名聲。

在測試查詢上檢查索引

最近一次變動於第 2026.11.8 版

在訓練查詢上調校過的索引,仍須在它沒見過的查詢上檢查,包括它相對於精確搜尋的召回率,以及它對評判分數的影響。本章規劃了 2 項比較,每項都使用成對排列檢定,並附上 t 檢定:將 faiss 預設 efSearch 為 16 的 HNSW 索引,以及在訓練查詢上選定設定的 HNSW 索引,分別與精確搜尋比較。搜尋時也要求 k = 100,因為混合檢索會融合各檢索器的前 100 名結果:

hnsw-test-recall.py Python 3.12.15 · faiss-cpu 1.16.0 · numpy 2.5.5 · pandas 3.1.1 · scipy 1.18.1 通過
"""Compares HNSW at faiss's default efSearch, and at the setting chosen on train queries, with exact search on the test queries, using paired tests at the planned threshold. The prose's point rests on the default setting's permutation and interval lines, and on the chosen setting's recall line."""import numpy as npfrom lib.metrics import evaluatefrom lib.recorded import embeddingsfrom lib.significance import compare, verdictfrom lib.store import load_queries, qrelsfrom lib.vectors import as_run, choose_ef_search, exact_search, hnsw_index, hnsw_search, recall_against_exact# Planned before scoring: HNSW at faiss's default efSearch, and at the setting chosen on train, each against exact.PLANNED = 2threshold = 0.05 / PLANNED  # the book's rule splits 0.05 across the planned comparisonsproduct_ids, product_vectors, query_ids, query_vectors = embeddings()rows = np.isin(query_ids, load_queries("test").index)index = hnsw_index(product_vectors, links=32)# the smallest efSearch at which every train query keeps WORST_RECALL of its true top 10chosen = choose_ef_search(index, query_vectors[np.isin(query_ids, load_queries("train").index)], product_vectors)# k = 100 because hybrid retrieval fuses each retriever's top 100exact = exact_search(query_vectors[rows], product_vectors, 100)exact_ndcg = evaluate(as_run(exact, query_ids[rows], product_ids), qrels("test"))["ndcg@10"]print("planned test comparisons:", PLANNED)print(f"p-value threshold: {threshold:.3f}")print("efSearch chosen on train queries:", chosen)print(f"exact ndcg@10: {exact_ndcg.mean():.3f}")for name, ef_search in [("default", 16), ("chosen", chosen)]:    found, _ = hnsw_search(index, query_vectors[rows], 100, ef_search)    # the first 10 of the 100    recall = recall_against_exact(found[:, :10], exact[:, :10])    ndcg = evaluate(as_run(found, query_ids[rows], product_ids), qrels("test"))["ndcg@10"]    result = compare(ndcg, exact_ndcg, confidence=1 - threshold)  # a 97.5% interval, at the test's level    print(f"\n{name}, recall@10 against exact: {recall.mean():.3f}")    print(f"{name}, queries with recall@10 of 0: {(recall == 0).sum()}")    print(f"{name}, those queries:", ", ".join(load_queries().loc[query_ids[rows][recall == 0], "query"]) or "none")    print(f"{name}, recall@100 against exact: {recall_against_exact(found, exact).mean():.3f}")    print(f"{name}, hnsw ndcg@10: {ndcg.mean():.3f}")    # compare rounds each difference to 3 decimals: wins and losses are the nonzero ones    print(f"{name}, queries whose ndcg@10 changes: {result['wins'] + result['losses']}")    print(f"{name}, permutation p: {result['permutation p']:.3f}")    print(f"{name}, t-test p: {result['t-test p']:.3f}")    print(f"{name}, {1 - threshold:.1%} interval: {result['ci low']:.4f} to {result['ci high']:.4f}")    for check, passed in verdict(result, threshold).items():      # established, t-test agrees, interval agrees        print(f"{name}, {check}: {'yes' if passed else 'no'}")
輸出python hnsw-test-recall.py
planned test comparisons: 2
p-value threshold: 0.025
efSearch chosen on train queries: 144
exact ndcg@10: 0.830

default, recall@10 against exact: 0.930
default, queries with recall@10 of 0: 2
default, those queries: liberty hardware francisco, 48" sliding single track , barn door for laundry
default, recall@100 against exact: 0.849
default, hnsw ndcg@10: 0.791
default, queries whose ndcg@10 changes: 3
default, permutation p: 0.256
default, t-test p: 0.154
default, 97.5% interval: -0.1544 to -0.0006
default, established: no
default, t-test agrees: yes
default, interval agrees: no

chosen, recall@10 against exact: 1.000
chosen, queries with recall@10 of 0: 0
chosen, those queries: none
chosen, recall@100 against exact: 0.994
chosen, hnsw ndcg@10: 0.830
chosen, queries whose ndcg@10 changes: 0
chosen, permutation p: 1.000
chosen, t-test p: 1.000
chosen, 97.5% interval: 0.0000 to 0.0000
chosen, established: no
chosen, t-test agrees: yes
chosen, interval agrees: yes

在訓練查詢上選定的設定表現穩定。它相對於精確搜尋的 recall@10 為 1.000,平均 nDCG@10 為 0.830,與精確搜尋的 0.830 相同,且有 0 個查詢的分數改變。

在 faiss 的預設設定下,有 2 個測試查詢完全沒有取得其真實前 10 名中的任何一個(liberty hardware francisco, 48" sliding single track , barn door for laundry),平均 nDCG@10 降至 0.791,但分數改變的查詢只有 3 個,這項下降並未獲得確立:排列檢定給出 p = 0.256,t 檢定的結果一致,p = 0.154。97.5% 區間為 -0.1544 to -0.0006,不包含 0,因此與檢定的結果不一致。由於承載差異的查詢極少,這裡的發現是相對於精確搜尋所損失的召回率,而不是經測量確認的品質損失。

要求更多結果,並不能彌補過小的 efSearch:在預設設定下,recall@100 為 0.849,而選定設定下為 0.994。

除了向量之外,圖結構也會耗用記憶體:

hnsw-index-memory.py Python 3.12.15 · faiss-cpu 1.16.0 · numpy 2.5.5 · pandas 3.1.1 通過
"""Counts the links per product in the HNSW graph, and the bytes per product of the exact and the HNSW index. The prose's point rests on the two bytes-per-product lines, set against faiss's estimate."""import faissfrom lib.recorded import embeddingsfrom lib.vectors import hnsw_indexproduct_ids, product_vectors, _, _ = embeddings()dimensions = product_vectors.shape[1]flat = faiss.IndexFlatIP(dimensions)                    # exact search: the vectors onlyflat.add(product_vectors)index = hnsw_index(product_vectors, links=32)           # the vectors, plus the graph's links# layer 0 is the bottom layer: 2 × M links there, and M on each upper layerprint("links per product, upper layers:", index.hnsw.nb_neighbors(1))print("links per product, bottom layer:", index.hnsw.nb_neighbors(0))# 8M with M = 32: 4 bytes for each of the 2M bottom-layer linksprint(f"faiss's estimate, 4d + 8M bytes per product: {4 * dimensions + 8 * 32:,}")# the saved index's size, spread over the productsfor name, built in [("exact index", flat), ("hnsw index", index)]:    print(f"bytes per product, {name}: {faiss.serialize_index(built).nbytes / len(product_ids):,.0f}")
輸出python hnsw-index-memory.py
links per product, upper layers: 32
links per product, bottom layer: 64
faiss's estimate, 4d + 8M bytes per product: 3,328
bytes per product, exact index: 3,072
bytes per product, hnsw index: 3,344

每個產品在上層有 32 條連結,在底層有 64 條,是 M 的兩倍。faiss 估計每個向量需要 4d + 8M 位元組(faiss wiki):d 個數字各 4 位元組,加上底層 2M 條連結各 4 位元組,此處為 3,328。儲存後的索引與此相當接近,每個產品為 3,344 位元組,而精確索引為 3,072。

實用原則: 在保留的查詢上,以相對於精確搜尋的召回率檢查調校後的索引,並先檢視那些完全失去前 k 名結果的查詢,再看平均分數。若 nDCG@10 的下降僅依賴少數查詢,在成對檢定確立之前,都應視為未經證實。編列記憶體預算時,除了向量之外,也要計入連結。

此部分尚未翻譯,因此以英文呈現。

Check queries that contain digits

最近一次變動於第 2026.11.10 版

An index can only return what the embeddings put near a query, so the next question is which queries the embeddings get wrong. An encoder squeezes a whole query into one vector, so a short token that changes the meaning, such as a size, a model number or a negation, may move that vector very little. Deng et al. (2026) measured this for numbers, in a study of 13 embedding models on synthetic financial text. Their accuracy at telling texts apart by a number was 0.54, only slightly above the 0.5 of random guessing, and four-digit numbers were at the level of guessing. Products aren't financial reports, so check the store's queries.

Few of the store's queries hold a digit, so compare BM25 and semantic retrieval on each that does, from both splits. Nothing is tuned on these queries, and a handful can't establish anything:

digits-in-queries.py Python 3.12.15 · bm25s 0.3.16 · numpy 2.5.5 · pandas 3.1.1 · scipy 1.18.1 通過
"""Scores BM25 and semantic retrieval on each query with a digit, in both splits, and counts the products whose text writes 7qt or 7 qt. The prose's point rests on the three lines that count which ranker wins and the two product counts."""import pandas as pdfrom lib.lexical import LexicalSearchfrom lib.metrics import evaluatefrom lib.semantic import SemanticSearchfrom lib.store import load_products, load_queries, product_text, qrelsproducts, queries = load_products(), load_queries()with_digit = queries[queries["query"].str.contains(r"\d")]       # both splits: a diagnosis, nothing is tunedjudgments = {query_id: grades for query_id, grades in qrels().items() if query_id in with_digit.index}# each ranker returns its top 100; nDCG@10 scores the first 10 of themscores = pd.DataFrame({    "split": with_digit["split"],    "bm25": evaluate(LexicalSearch(products).run(with_digit, k=100), judgments)["ndcg@10"],    "semantic": evaluate(SemanticSearch().run(with_digit, k=100), judgments)["ndcg@10"],    "query": with_digit["query"],}).round(3)print("queries with a digit:", len(scores))print(scores.to_string())print("semantic better:", (scores["semantic"] > scores["bm25"]).sum())print("bm25 better:", (scores["bm25"] > scores["semantic"]).sum())# a gap under 0.03 counts as a near tie: the book's choice, not a measured limitprint("gaps under 0.03:", ((scores["semantic"] - scores["bm25"]).abs() < 0.03).sum())text = product_text(products).str.lower()print("products whose text has the word 7qt:", text.str.contains(r"\b7qt\b").sum())# '7 qt', but not '17 qt' or '3.7 qt'print("products whose text has 7 qt:", text.str.contains(r"(?<![\d.])7 qt\b").sum())
輸出python digits-in-queries.py
queries with a digit: 7
          split   bm25  semantic                                             query
query_id
45        train  0.288     0.316                            70s inspired furniture
62        train  0.554     0.930                                   7qt slow cooker
101       train  0.569     0.770        48 in entry table with side by side drawer
203       train  0.606     0.740                         ge top loading washer 4.5
205       train  0.935     0.930                          ines shag rug w003096809
285        test  0.592     0.555  48" sliding single track , barn door for laundry
482        test  1.000     0.921                              48 inch kitchen hood
semantic better: 4
bm25 better: 3
gaps under 0.03: 2
products whose text has the word 7qt: 0
products whose text has 7 qt: 8

Neither ranker owns the 7 queries with a digit. Semantic retrieval scores higher on 4 of them and BM25 on 3, and 2 of the gaps are under 0.03. BM25's clearest loss is 7qt slow cooker: the word 7qt appears in 0 products, and 8 write 7 qt.

On the test query 48 inch kitchen hood, BM25 scores higher, so here is semantic retrieval's top 10:

hood-query-digits.py Python 3.12.15 · bm25s 0.3.16 · numpy 2.5.5 · pandas 3.1.1 · scipy 1.18.1 通過
"""Lists semantic retrieval's top 10 for 48 inch kitchen hood with each hood's width, read from the attributes column the embeddings never saw. The prose's point rests on the two width-48 lines, one for each ranker."""from lib.lexical import LexicalSearchfrom lib.semantic import SemanticSearchfrom lib.store import load_products, load_queries, qrelsLABELS = {2: "exact", 1: "partial", 0: "irrelevant"}products = load_products()query_id = 482  # 48 inch kitchen hood, a test queryquery = load_queries().at[query_id, "query"]judgments = qrels()[query_id]# The width sits in the attributes column, which the embeddings never saw: product_text leaves it out.width = products["attributes"].str.extract(r"\bwidth: ([\d.]+)", expand=False).astype(float)semantic = SemanticSearch().search(query_id, k=10).indexbm25 = LexicalSearch(products).search(query, k=10).indexprint("query:", query)print("     judgment   width  name")for rank, product_id in enumerate(semantic, start=1):    label = LABELS.get(judgments.get(product_id), "unjudged")    print(f"{rank:>2}   {label:<10} {width[product_id]:>5g}  {products.at[product_id, 'name'][:50]}")print()# exact: a 48-inch hood is 48 wide, where probe-pair-overlap allows half an inch for hoods listed at 35.75for name, top in [("semantic", semantic), ("bm25", bm25)]:    print(f"{name}, width 48 in the top 10:", (width[top] == 48).sum())    print(f"{name}, exact in the top 10:", sum(judgments.get(product_id) == 2 for product_id in top))
輸出python hood-query-digits.py
query: 48 inch kitchen hood
     judgment   width  name
 1   exact         48  48 '' 1200 cfm ducted wall mount range hood in bru
 2   exact         48  48 '' 400 cfm ducted wall mount range hood in brus
 3   exact         48  48 '' 400 cfm convertible wall mount range hood
 4   partial    35.75  36 '' ws-48 series 500 cfm ducted under cabinet ra
 5   exact         48  48 '' hand crafted designer wood island hood serie
 6   exact         48  48 '' island range hood
 7   exact         48  48 inch wall mount chimney range hood with 600 cfm
 8   exact         48  48 '' ducted island range hood
 9   exact         48  48 '' professional 900cfm ducted island range hood
10   partial     29.8  30 '' 480 cfm convertible under cabinet range hood

semantic, width 48 in the top 10: 8
semantic, exact in the top 10: 8
bm25, width 48 in the top 10: 10
bm25, exact in the top 10: 10

For this query, the embedding matched the digits as text, not the width they stand for. Of semantic retrieval's top 10, 8 hoods have a width of 48. The others are a 36-inch hood from a series named ws-48 and a 30-inch hood rated at 480 cfm, cubic feet per minute, the airflow of its fan. Both contain the digits 48, neither is 48 wide, and the judges marked both partial. BM25's top 10 holds 10 hoods of width 48.

Each hood's width comes from the attributes column, which holds curated fields from WANDS's product features, such as width, shape and material. product_text, the text the model embedded, leaves that column out, so the embeddings never saw it, and a size the shopper types can be checked against it exactly.

One query can't show a pattern. To see how the model handles sizes in general, use probe queries: pairs of queries written for the book that differ only in a size, a code or a negation, which recordings/probes.py embeds with the same model. One of them repeats the test query 48 inch kitchen hood word for word. Probes have no judgments, so the example counts their top 10s and never scores them. The right size comes from the width in attributes for the hoods, and from the name for the rest:

probe-pair-overlap.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 通過
"""Counts, for each probe pair, the products the two top 10s share and the top-10 products of the right size, without scoring anything. The prose's point rests on the right-size lines for the hood, slow-cooker and bed probes, and on the seat-count and loveseat lines for the sofa probe."""from lib.recorded import probesfrom lib.semantic import SemanticSearchfrom lib.store import load_products# Probe queries written for the book, embedded by the same model. They have no judgments,# so this example counts what each top 10 holds; it doesn't score anything.probe_queries, probe_vectors = probes()products = load_products()names = products["name"]width = products["attributes"].str.extract(r"\bwidth: ([\d.]+)", expand=False).astype(float)semantic = SemanticSearch()def states(size, unit):    """Names that state this size, such as "7 qt" or "7-quart", but not "3.7 qt"."""    return names.str.contains(rf"(?<![\d.]){size}[ -]?(?:{unit})")RIGHT_SIZE = {                       # the size each probe asks for: the width attribute, or the name    # within half an inch of 36 or 48, so a 35.75-inch hood counts as 36    "36 inch kitchen hood": (width - 36).abs() <= 0.5, "48 inch kitchen hood": (width - 48).abs() <= 0.5,    "4 qt slow cooker": states(4, "qt|quart"), "7 qt slow cooker": states(7, "qt|quart"),    # the whole word, which also matches 'twin over full' bunk beds    "twin bed": names.str.contains(r"\btwin\b"), "king bed": names.str.contains(r"\bking\b"),    "2 seat sofa": names.str.contains(r"\bloveseat\b"),        # a two-seat sofa is a loveseat}DIGIT_ONLY = {"2 seat sofa": states(2, " in 1| piece"), "4 seat sofa": states(4, " in 1| piece")}for pair in ["hood", "cooker", "bed", "seats"]:    first, second = probe_queries.index[probe_queries["pair"] == pair]  # each pair is two probes, in file order    tops = {row: semantic.search_vector(probe_vectors[row], k=10).index for row in (first, second)}    print(f"{pair}, similarity of the two queries: {probe_vectors[first] @ probe_vectors[second]:.3f}")    print(f"{pair}, products in both top 10s:", len(set(tops[first]) & set(tops[second])))    for row, top in tops.items():        query = probe_queries.at[row, "query"]        if query in RIGHT_SIZE:            print(f"{query}, right size in the top 10:", RIGHT_SIZE[query][top].sum())        if query in DIGIT_ONLY:            print(f"{query}, same digit but a piece count or 2 in 1:", DIGIT_ONLY[query][top].sum())    print()# the two counts the sofa probes ask forprint("catalog names that give a seat count:", (states(2, "seat") | states(4, "seat")).sum())print("catalog products named loveseat:", names.str.contains(r"\bloveseat\b").sum())
輸出python probe-pair-overlap.py
hood, similarity of the two queries: 0.795
hood, products in both top 10s: 0
36 inch kitchen hood, right size in the top 10: 8
48 inch kitchen hood, right size in the top 10: 8

cooker, similarity of the two queries: 0.841
cooker, products in both top 10s: 1
4 qt slow cooker, right size in the top 10: 6
7 qt slow cooker, right size in the top 10: 7

bed, similarity of the two queries: 0.697
bed, products in both top 10s: 0
twin bed, right size in the top 10: 10
king bed, right size in the top 10: 10

seats, similarity of the two queries: 0.759
seats, products in both top 10s: 1
2 seat sofa, right size in the top 10: 0
2 seat sofa, same digit but a piece count or 2 in 1: 4
4 seat sofa, same digit but a piece count or 2 in 1: 4

catalog names that give a seat count: 0
catalog products named loveseat: 14

For these probes, the model mostly finds the right size. The two hood probes share 0 products, yet each top 10 holds hoods of the right width, 8 for 36 inches and 8 for 48; the slow-cooker probes find 6 and 7 of the right capacity, and the bed probes 10 and 10 of the right size. Where the catalog has no number to match, the digit takes over. The number of product names that give a seat count is 0, so the top 10 for 2 seat sofa holds 4 products that share only the digit, two-piece sets and "2 in 1" kids' sofas, and 0 of the catalog's 14 loveseats.

預期產品代碼的權重很低

最近一次變動於第 2026.11.8 版

產品代碼應該能精確對應到單一產品。對編碼器而言,它只是一串罕見的字母與數字。探測查詢中有兩組用來測試代碼:一組以沒有任何產品擁有的代碼為基礎,取自訓練查詢 ines shag rug w003096809;另一組則以恰好只有一項產品的文字中出現的螢幕解析度為基礎:

code-probes.py Python 3.12.15 · bm25s 0.3.16 · numpy 2.5.5 · pandas 3.1.1 · scipy 1.18.1 通過
"""Compares the top 10s of two probe queries that differ only in a code no product holds, and shows where one product ranks for two resolutions, semantically and with BM25. The prose's point rests on the shared-top-10 line for the code and on the rank lines for the resolutions."""from lib.lexical import LexicalSearchfrom lib.recorded import probesfrom lib.semantic import SemanticSearchfrom lib.store import load_products, product_text# Probe queries written for the book, embedded by the same model; no judgments, so nothing is scored.probe_queries, probe_vectors = probes()products = load_products()text = product_text(products).str.lower()semantic, bm25 = SemanticSearch(), LexicalSearch(products)# A code no product holds: does changing it change the results?first, second = probe_queries.index[probe_queries["pair"] == "code"]tops = [set(semantic.search_vector(probe_vectors[row], k=10).index) for row in (first, second)]print("products whose text contains w003096809:", text.str.contains("w003096809").sum())print(f"code, similarity of the two queries: {probe_vectors[first] @ probe_vectors[second]:.3f}")print("code, products in both top 10s:", len(tops[0] & tops[1]))# A code one product holds: where does that product rank, with its code and with another?holder = text.index[text.str.contains("1920x1080p")]print()print("products whose text contains 1920x1080p:", len(holder))print("that product:", products.at[holder[0], "name"])print("products in the Projectors class:", (products["product_class"] == "Projectors").sum())for row in probe_queries.index[probe_queries["pair"] == "resolution"]:    query = probe_queries.at[row, "query"]    ranked = semantic.search_vector(probe_vectors[row], k=len(products))  # ranks every product, so the rank is exact    lexical = list(bm25.search(query, k=100).index)  # BM25 ranks are read from its top 100 only    print(f"{query}, its semantic rank: {list(ranked.index).index(holder[0]) + 1}")  # ranks count from 1    print(f"{query}, its similarity: {ranked[holder[0]]:.3f}")    # the best product other than this one    print(f"{query}, similarity of the next product: {ranked.drop(holder[0]).iloc[0]:.3f}")    print(f"{query}, its bm25 rank:", lexical.index(holder[0]) + 1 if holder[0] in lexical else "not in the top 100")
輸出python code-probes.py
products whose text contains w003096809: 0
code, similarity of the two queries: 0.967
code, products in both top 10s: 10

products whose text contains 1920x1080p: 1
that product: 5000 portable transmissive projector
products in the Projectors class: 4
1920x1080p projector, its semantic rank: 1
1920x1080p projector, its similarity: 0.562
1920x1080p projector, similarity of the next product: 0.496
1920x1080p projector, its bm25 rank: 1
2560x1440p projector, its semantic rank: 1
2560x1440p projector, its similarity: 0.509
2560x1440p projector, similarity of the next product: 0.508
2560x1440p projector, its bm25 rank: not in the top 100

更改未知代碼的最後幾位數字後,兩個查詢的相似度為 0.967,前 10 名結果共有 10 項相同的產品。無法對應到任何產品的代碼不會改變任何結果,而 BM25 同樣沒有可以比對的內容。

只有一項產品擁有的代碼,會決定 BM25 的排名,卻只會輕微影響嵌入的排名。解析度 1920x1080p 只出現在5000 portable transmissive projector的文字中。對於 1920x1080p projector,BM25 將它排在第 1 名;對於 2560x1440p projector,則是第 not in the top 100 名。Sciavolino 等人(2021)發現,在涉及罕見實體的問題上,密集(以嵌入為基礎)檢索器的表現遠不如稀疏方法;Karpukhin 等人(2020)則指出 BM25 對高度具選擇性的關鍵字與詞組反應靈敏。

嵌入在查詢含有自身代碼時將它排在第 1 名,含有另一個代碼時排在第 1 名,原因是商店中只有 4 款投影機。代碼的影響只體現在與下一項產品之間的差距:有代碼時為 0.562 對 0.496,沒有代碼時為 0.509 對 0.508。

**實用原則:**列出紀錄中含有尺寸、數量或代碼的查詢,並在信任任何一種排序器之前,先用這兩種排序器分別比較。當代碼很重要時,以詞彙比對來對應它。當精確的尺寸很重要時,則將它與產品欄位比對,例如 attributes 中的寬度;或者在斷詞器保留數字、且單位寫法一致之後,改以詞彙比對(第 5 章、第 8 章)。查詢的其餘部分交給語意檢索處理。第 7 章會測試如何結合這兩者。

測試嵌入對否定的處理

最近一次變動於第 2026.11.8 版

否定只用一個短詞,就能反轉購物者想要的東西:一張不是圓形的地毯。整個查詢的嵌入向量,可能仍然與少了那個詞的同一查詢非常接近。NevIR 是一個基準測試,其文件對僅以否定作區別,結果發現沒有任何雙編碼器(bi-encoder)架構的成對準確率超過 12%,而隨機排序的成對準確率為 25%(Weller et al., 2024)。在 2025 年的一項重現研究中,van den Elsen et al. 發現,一個 70 億參數的雙編碼器 GritLM-7B 達到 39.0%,其他雙編碼器進步甚微,只有交叉編碼器(cross-encoder)與列表式(listwise)LLM 重新排序器能對兩項否定任務都有相當不錯的表現。

商店查詢中含有「no」、「not」、「without」或「non」的比例為 0,因此相關性判斷無法用來測試否定。我們改用 3 組探測查詢,每組各包含一個查詢及其否定形式,並計算各自前 10 筆結果的內容。

目錄中的商品足以滿足每一個否定探測:其中有 42 件商品名稱含 armless、62 張咖啡桌的內容從未提到玻璃,以及 90 張區域地毯的名稱不含 round。

在執行之前,請先猜猜看,語意檢索針對 rug that is not round 所回傳的前 10 筆結果中,有多少件的名稱含 round。

negation-probes.py Python 3.12.15 · bm25s 0.3.16 · numpy 2.5.5 · pandas 3.1.1 · scipy 1.18.1 通過
"""Counts what each negation probe's top 10 holds for semantic retrieval and for BM25, without scoring it. The prose's point rests on the semantic top 10 lines for the rug, coffee-table and sofa probes."""import bm25sfrom lib.lexical import LexicalSearchfrom lib.recorded import probesfrom lib.semantic import SemanticSearchfrom lib.store import load_products, load_queries, product_text# Probe queries written for the book, with no judgments: count what the top 10 holds, never score it.probe_queries, probe_vectors = probes()products = load_products()names, text = products["name"].str.lower(), product_text(products).str.lower()semantic, bm25 = SemanticSearch(), LexicalSearch(products)# armless and round are matched in product names, glass in the full textFEATURES = {"arms": ("armless", names), "glass": ("glass", text), "round": ("round", names)}negated = load_queries()["query"].str.contains(r"\b(?:no|not|without|non)\b")print("store queries with no, not, without or non:", negated.sum())# the tokenizer settings LexicalSearch uses, so these are the terms BM25 searchestokens = bm25s.tokenize(["rug that is not round"], stopwords="en", return_ids=False, show_progress=False)print("bm25 terms of rug that is not round:", " ".join(tokens[0]))print("products named armless:", names.str.contains(r"\barmless\b").sum())coffee_tables = products["product_class"] == "Coffee & Cocktail Tables"area_rugs = products["product_class"] == "Area Rugs"print("coffee tables that never mention glass:", (coffee_tables & ~text.str.contains(r"\bglass\b")).sum())print("area rugs not named round:", (area_rugs & ~names.str.contains(r"\bround\b")).sum())negation = probe_queries[probe_queries["kind"] == "negation"]print("negation probe pairs:", negation["pair"].nunique())for row, probe in negation.iterrows():    word, field = FEATURES[probe["pair"]]    has_feature = field.str.contains(rf"\b{word}\b")    semantic_top = semantic.search_vector(probe_vectors[row], k=10).index    bm25_top = bm25.search(probe["query"], k=10).index    print()    print(f"{probe['query']}, semantic top 10 with {word}:", has_feature[semantic_top].sum())    print(f"{probe['query']}, bm25 top 10 with {word}:", has_feature[bm25_top].sum())
輸出python negation-probes.py
store queries with no, not, without or non: 0
bm25 terms of rug that is not round: rug round
products named armless: 42
coffee tables that never mention glass: 62
area rugs not named round: 90
negation probe pairs: 3

sofa with arms, semantic top 10 with armless: 1
sofa with arms, bm25 top 10 with armless: 0

sofa without arms, semantic top 10 with armless: 8
sofa without arms, bm25 top 10 with armless: 0

coffee table with glass top, semantic top 10 with glass: 10
coffee table with glass top, bm25 top 10 with glass: 9

coffee table not glass, semantic top 10 with glass: 10
coffee table not glass, bm25 top 10 with glass: 10

round rug, semantic top 10 with round: 10
round rug, bm25 top 10 with round: 10

rug that is not round, semantic top 10 with round: 10
rug that is not round, bm25 top 10 with round: 10

如果你猜的數量很少,那表示你是以購物者的方式理解「not」,而嵌入並非如此。在地毯與咖啡桌的探測中,嵌入保留了屬性,卻遺失了「not」。語意檢索針對 rug that is not round 的前 10 筆結果中,有 10 件商品名稱含 round;針對 round rug 的前 10 筆結果中則有 10 件。對於 coffee table not glass,前 10 筆中有 10 筆提到玻璃;coffee table with glass top 的對應數字則是 10。

sofa without arms 的情況正好相反,其前 10 筆中有 8 件名稱含 armless;sofa with arms 的對應數字則是 1。目錄中對這種否定只有一個詞,即 armless,而模型把「without arms」放在它的附近。

BM25 在這些探測上也沒有比較好。它的分詞器會把「not」、「that」和「is」當作停用詞移除,因此它把 rug that is not round 當成 rug round 來搜尋。它的前 10 筆結果中有 10 張圓形地毯,以及 0 張無扶手沙發。

這些探測只有少數幾組,並不構成衡量結果,而且沙發這一組的情況正好相反。但地毯與咖啡桌這兩組與 NevIR 的結果一致:查詢向量仍然靠近查詢所提到的屬性。

實用原則: 不要依賴第一階段的檢索器(無論是詞彙式或語意式)來處理否定。當購物者排除的屬性是目錄中有儲存的屬性,例如形狀、材質或扶手時,請偵測該排除條件,並依該屬性進行篩選。否則就進行重新排序,因為在 van den Elsen et al. 的重現研究中,交叉編碼器與 LLM 重新排序器處理否定的表現好得多。第 9 章使用交叉編碼器與 LLM對商店的語意檢索進行重新排序,而其中沒有任何一個重新排序器在商店已判斷的查詢上勝過語意檢索,這些查詢無法測試否定。請保留類似這樣的探測作為檢查,每當更換模型或加入重新排序器時就重新執行。

選擇嵌入模型

模型決定了向量能夠掌握哪些資訊,而日後若更換模型,就得重新嵌入整個商品目錄。公開的基準測試可以縮小選擇範圍,最終則由商店自己的判斷來決定。

將基準測試當作候選清單來讀

最近一次變動於第 2026.11.8 版

MTEB(Massive Text Embedding Benchmark,大規模文字嵌入基準)以多項任務為模型評分。其作者發現,沒有任何單一方法能在所有任務上都居於領先(Muennighoff et al., 2023)。作者也發現,其平均分數偏向資料集數量較多的任務,特別是檢索、分類與分群,因此應閱讀與自己任務相對應的分數。

公開分數也可能高估模型在未見過的資料上的表現。檢索基準 RTEB 的製作者指出,模型在公開基準上回報的分數,與其在全新、未見過的資料上的實際表現之間存在落差(Liu et al., 2025)。RTEB 的 48 個資料集中有 28 個不公開。Verma et al. (2026) 發現,RTEB 對模型的排名與 MTEB 及 BEIR(Thakur et al., 2021,一個較早期、涵蓋多個領域的檢索基準)的排名,相關係數僅為 ρ = 0.38。此處 ρ 是等級相關係數,1 表示順序完全相同。模型大小也不等於品質。Enevoldsen et al. (2025) 報告指出,在 MTEB 的多語言後繼基準 MMTEB 中,公開可取得的模型裡表現最好的是 multilingual-e5-large-instruct,其參數量僅有 5.6 億。

商店考慮了四個開放模型,而它們的模型卡都沒有回報在 WANDS 上的結果。這四個模型包括商店所用的編碼器,以及 Qwen/Qwen3-Embedding-0.6B(模型卡)。另外兩個是 BAAI/bge-base-en-v1.5(模型卡)與 ibm-granite/granite-embedding-english-r2(模型卡)。商品搜尋基準,例如 ESCI,會將查詢與商品配對標記為完全符合、替代品、互補品或不相關,並包含約 13 萬筆查詢與 260 萬筆判斷(Reddy et al., 2022)。這類基準更接近商店的任務,但它們仍然不是商店自己的商品目錄。因此,商店的候選清單依據的是排行榜無法進行的檢查:允許商業使用的授權、支援英文、不需特殊硬體即可執行的大小,以及有文件記載的較短向量支援。商店所選模型的模型卡回報其在 MTEB 檢索任務上的 nDCG@10 為 55.14,這是廠商針對一般文字而非商品所做的量測(模型卡)。若要在候選模型之間做選擇,應在相同的商品文字與查詢上記錄每個模型的輸出,以訓練查詢為每個模型評分,並使用本書的成對檢定加以比較,下一小節就以同一模型的多種大小做了這樣的比較。

截斷維度,然後檢驗損失

最近一次變動於第 2026.11.8 版

每個維度以 32 位元浮點數儲存時,每項商品需要 4 位元組,而且每次相似度計算都會用到全部維度,因此較短的向量在儲存與搜尋上都比較便宜。Matryoshka(俄羅斯套娃)訓練會讓模型把最有用的資訊放在前面的維度,使向量的開頭本身就能當作較小的嵌入向量來使用。Kusupati et al. (2022) 指出,在 ImageNet-1K 分類上,嵌入向量最多可縮小至 14 分之 1,而準確率維持在相同水準。縮短向量能節省儲存與搜尋的成本,編碼則不能:如 Sentence Transformers 的文件所述,Matryoshka 模型執行起來並不會比較快。只有以這種方式訓練的模型,才是為了承受截斷而設計的;而且截斷後的向量長度不再是 1,所以要像模型卡自己的範例一樣,重新正規化。

商店的編碼器有一份模型卡,記載了可截斷至 256 維的做法。lib.recorded.embeddings(dims=...) 會保留每個向量的前 dims 個數值,並再次正規化。在訓練查詢上為每種長度評分:

truncate-embeddings.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 通過
"""Scores semantic retrieval on the train queries at 768, 256 and 128 dimensions, with each product's float32 size. The prose's point rests on the nDCG@10 line for each length, set against 768 dimensions."""from lib.metrics import evaluatefrom lib.semantic import SemanticSearchfrom lib.store import load_queries, qrelstrain, judgments = load_queries("train"), qrels("train")       # choose on train queries# dims keeps the first dims numbers of every vector and rescales each to length 1 (lib.recorded).for dims in [768, 256, 128]:    semantic = SemanticSearch(dims=dims)    scores = evaluate(semantic.run(train, k=100), judgments)    # semantic.products holds the product vectors, not the product table    print(f"{dims} dimensions, float32 bytes per product: {semantic.products[0].nbytes:,}")    print(f"{dims} dimensions, ndcg@10: {scores['ndcg@10'].mean():.3f}")    print(f"{dims} dimensions, judged@10: {scores['judged@10'].mean():.3f}")    print()
輸出python truncate-embeddings.py
768 dimensions, float32 bytes per product: 3,072
768 dimensions, ndcg@10: 0.772
768 dimensions, judged@10: 0.964

256 dimensions, float32 bytes per product: 1,024
256 dimensions, ndcg@10: 0.762
256 dimensions, judged@10: 0.952

128 dimensions, float32 bytes per product: 512
128 dimensions, ndcg@10: 0.680
128 dimensions, judged@10: 0.874

截斷至 256 維,會使每項商品從 3,072 位元組降到 1,024,並使平均 nDCG@10 從 0.772 變為 0.762。

在 128 維時,平均值為 0.680,judged@10 為 0.874,而 768 維時為 0.964,因此這項下降有一部分可能來自無人判定過的商品。

這個變化是真實的,還是 50 個查詢之間的雜訊?用 lib.significance.compare 來檢驗,它會執行第 3 章的排列檢定與成對 t 檢定,也會回報自助法區間。這些比較使用訓練查詢,因此它們是在調整向量大小,並不是關於測試查詢的證據:

truncation-paired-test.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 · scipy 1.18.1 通過
"""Tests the change in nDCG@10 from shortening to 256 and 128 dimensions against the full vectors, on the train queries, with a permutation test, a paired t-test and a bootstrap interval. The prose's point rests on the permutation, interval and effect-size lines for each length."""from lib.metrics import evaluatefrom lib.semantic import SemanticSearchfrom lib.significance import comparefrom lib.store import load_queries, qrels# Train queries: these comparisons tune the vector size. They are not evidence about the test queries.train, judgments = load_queries("train"), qrels("train")def ndcg(dims):    return evaluate(SemanticSearch(dims=dims).run(train, k=100), judgments)["ndcg@10"]def p_value(p):    return "below 0.001" if p < 0.001 else f"{p:.3f}"full = ndcg(768)for dims in [256, 128]:    # compare's default interval is 95%    result = compare(ndcg(dims), full)                    # paired by train query: shortened minus full    print(f"{dims} vs 768, mean difference in ndcg@10: {result['mean difference']:.3f}")    print(f"{dims} vs 768, queries worse: {result['losses']}")    print(f"{dims} vs 768, permutation test p: {p_value(result['permutation p'])}")    print(f"{dims} vs 768, paired t-test p: {p_value(result['t-test p'])}")    print(f"{dims} vs 768, 95% bootstrap interval: {result['ci low']:.4f} to {result['ci high']:.4f}")    print(f"{dims} vs 768, interval excludes 0: {'yes' if result['ci high'] < 0 or result['ci low'] > 0 else 'no'}")    print(f"{dims} vs 768, effect size: {result['effect size']:.3f}")    print()
輸出python truncation-paired-test.py
256 vs 768, mean difference in ndcg@10: -0.010
256 vs 768, queries worse: 24
256 vs 768, permutation test p: 0.076
256 vs 768, paired t-test p: 0.080
256 vs 768, 95% bootstrap interval: -0.0211 to -0.0002
256 vs 768, interval excludes 0: yes
256 vs 768, effect size: -0.253

128 vs 768, mean difference in ndcg@10: -0.092
128 vs 768, queries worse: 33
128 vs 768, permutation test p: below 0.001
128 vs 768, paired t-test p: below 0.001
128 vs 768, 95% bootstrap interval: -0.1569 to -0.0565
128 vs 768, interval excludes 0: yes
128 vs 768, effect size: -0.547

在 128 維時,損失很明確。平均差異為 -0.092,有 33 個訓練查詢變差。排列檢定得到 p below 0.001,而 95% 區間(範圍為 -0.1569 to -0.0565)不包含 0。

在 256 維時,證據處於臨界,因為檢定與區間的結論並不一致。平均差異為 -0.010;排列檢定得到 p = 0.076,t 檢定得到 p = 0.080,然而區間(範圍為 -0.0211 to -0.0002)只是勉強不包含 0。效果量在 256 維為 -0.253,在 128 維為 -0.547,這表示 256 維的損失即使存在,也是兩者中較小的。

對這家商店而言,決定取決於規模,而不是檢定結果。完整向量只佔 36.9 MB,可節省的空間不多;因此它保留全部 768 個維度,測試分數也不會改變。擁有數億項商品的目錄,則要權衡每項商品從 3,072 降到 1,024 位元組的節省,與這麼多訓練查詢無法確定的損失,並且會在決定前先判定更多查詢。

實用原則: 以授權、語言、大小與領域來篩選候選模型,並把基準測試的排名當作參考。要在模型之間,或在向量大小之間做決定,應在自己的訓練查詢上進行成對檢定,並且除了 p 值之外也要看區間。只對經過相應訓練的模型截斷向量,截斷後要正規化,而且只在記憶體或搜尋時間有需要時才這麼做。

正式環境中會出什麼問題

最近一次變動於第 2026.11.8 版

相似度不等於信心。 當最佳相似度偏低時,很容易想乾脆什麼都不顯示。但不同查詢之間的相似度並不在同一個尺度上:

similarity-scale.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 通過
"""Finds the test query with the lowest best similarity and the one with the highest 10th similarity, and counts the products above a 0.5 cutoff. The prose's point rests on the two similarity lines and the cutoff counts."""import numpy as npfrom lib.recorded import embeddingsfrom lib.semantic import SemanticSearchfrom lib.store import load_queries, qrelsproduct_ids, product_vectors, query_ids, query_vectors = embeddings()test = load_queries("test")rows = np.isin(query_ids, test.index)similarity = np.sort(query_vectors[rows] @ product_vectors.T, axis=1)[:, ::-1]   # best first, per queryqueries = test.loc[query_ids[rows], "query"].to_numpy()weakest = similarity[:, 0].argmin()                    # the query whose best match is least similarstrongest = similarity[:, 9].argmax()                  # the query whose 10th match is most similarprint("first query, lowest best similarity:", queries[weakest])print(f"its best similarity: {similarity[weakest, 0]:.3f}")judgments = qrels()[query_ids[rows][weakest]]top = SemanticSearch().search(query_ids[rows][weakest], k=10).indexprint("exact products in its top 10:", sum(judgments.get(product_id) == 2 for product_id in top))print("second query, highest 10th similarity:", queries[strongest])print(f"its 10th similarity: {similarity[strongest, 9]:.3f}")cutoff = 0.5  # an illustrative cutoff; nothing about the model sets itprint()print(f"products above {cutoff} for the first query:", (similarity[weakest] > cutoff).sum())print(f"products above {cutoff} for the second query:", (similarity[strongest] > cutoff).sum())print(f"test queries with no product above {cutoff}:", (similarity[:, 0] <= cutoff).sum())
輸出python similarity-scale.py
first query, lowest best similarity: liberty hardware francisco
its best similarity: 0.436
exact products in its top 10: 5
second query, highest 10th similarity: kohler whitehaven farmhouse kitchen sink
its 10th similarity: 0.647

products above 0.5 for the first query: 0
products above 0.5 for the second query: 103
test queries with no product above 0.5: 2

測試查詢中的兩個極端說明了這一點。liberty hardware francisco的最佳匹配最弱,只有0.436,但它的前 10 名中有5個精確商品;而kohler whitehaven farmhouse kitchen sink的第 10 名匹配相似度最高,達到0.647。

若以 0.5 為門檻,第一個查詢會保留0個商品,第二個查詢會保留103個商品,而有2個測試查詢則一個商品也不會留下。語意檢索也從不會回傳空結果:即使是沒有任何商品使用的代碼,仍然會得到完整的前 10 名。要判定某個查詢沒有匹配項,應該改用其他方式,例如詞彙檢索找不到該查詢中的任何罕見詞彙時。

向量不匹配時會無聲地失敗。 查詢向量與商品向量必須來自相同的模型、修訂版本與提示詞。來自兩個不同模型的向量,是位於不同空間中的數字;當兩者維度相同時,比較它們不會產生任何錯誤。Sentence Transformers 的encode_query會在模型有預先定義的查詢提示詞時自動加上該提示詞(API 參考文件)。請確認負責嵌入即時查詢的程式碼,所做的事與建立索引的程式碼一致。

索引不會自行跟著商品目錄更新。 根據 faiss wiki,faiss 的 HNSW 不支援移除向量,因此已停產的商品會一直留在圖中,除非你在結果中把它過濾掉,或重新建立索引。Douze 等人(2024)指出,faiss 不是資料庫:它沒有並行寫入、分片或交易功能。相對於精確搜尋所測得的召回率,只適用於某一份商品目錄與某一組設定,因此每次重建之後都要再測量一次。

過長的文字會被截斷。 模型最多讀取512個 token,而 Sentence Transformers 的 API 參考文件指出,較長的輸入會被截斷。這家商店最長的商品文字有201個 token,但在描述很長的商品目錄中,超出上限的部分模型根本讀不到。嵌入之前請先計算 token 數,就像recordings/embeddings.py所做的那樣。

錯字會移動向量。 Zhuang 與 Zuccon(2021)發現,對於未以拼字錯誤訓練過的 BERT 類檢索器,錯字可能造成檢索與排序效果顯著下降。這家商店的查詢中有4個含有錯字,且每一個都有拼寫正確的探測查詢。校正後的探測查詢與原查詢要求的是同一件事,因此可以用原查詢的相關性判斷來為它評分:

spelling-probes.py Python 3.12.15 · bm25s 0.3.16 · numpy 2.5.5 · pandas 3.1.1 · scipy 1.18.1 通過
"""Scores each correctly spelled probe with the judgments of the query it corrects, for semantic retrieval and BM25, as typed and as corrected. The prose's point rests on the semantic nDCG@10 lines for each spelling and on the smallest-p line."""from lib.lexical import LexicalSearchfrom lib.metrics import ndcg_at_kfrom lib.recorded import probesfrom lib.semantic import SemanticSearchfrom lib.store import load_products, load_queries, qrels# Each spelling probe is the correctly spelled text of one store query (of_query_id), embedded by the same# model. It asks for the same thing, so it is scored with that query's judgments. A diagnosis, not a test.probe_queries, probe_vectors = probes()products, queries, judgments = load_products(), load_queries(), qrels()semantic, bm25 = SemanticSearch(), LexicalSearch(products)spelling = probe_queries[probe_queries["kind"] == "spelling"]for row, probe in spelling.iterrows():    query_id, pair = int(probe["of_query_id"]), probe["pair"]    typed, corrected = semantic.search(query_id, k=10).index, semantic.search_vector(probe_vectors[row], k=10).index    print(f"{pair}: {queries.at[query_id, 'query']} | {probe['query']} ({queries.at[query_id, 'split']} query)")    print(f"{pair}, similarity of the two spellings: {semantic.query_vectors[query_id] @ probe_vectors[row]:.3f}")    print(f"{pair}, products in both semantic top 10s:", len(set(typed) & set(corrected)))    print(f"{pair}, semantic ndcg@10 as typed: {ndcg_at_k(list(typed), judgments[query_id], 10):.3f}")    print(f"{pair}, semantic ndcg@10 corrected: {ndcg_at_k(list(corrected), judgments[query_id], 10):.3f}")    for name, text in [("as typed", queries.at[query_id, "query"]), ("corrected", probe["query"])]:        print(f"{pair}, bm25 ndcg@10 {name}: {ndcg_at_k(list(bm25.search(text, k=10).index), judgments[query_id], 10):.3f}")    print()print("spelling probes:", len(spelling))# a sign-flip test on n queries has 2 ** n outcomes, and its two most extreme give the smallest two-sided pprint(f"smallest two-sided permutation p-value possible with {len(spelling)} queries: {2 / 2 ** len(spelling):.3f}")
輸出python spelling-probes.py
ligth: ligth bulb | light bulb (train query)
ligth, similarity of the two spellings: 0.710
ligth, products in both semantic top 10s: 5
ligth, semantic ndcg@10 as typed: 1.000
ligth, semantic ndcg@10 corrected: 1.000
ligth, bm25 ndcg@10 as typed: 0.945
ligth, bm25 ndcg@10 corrected: 0.945

foutains: foutains with brick look | fountains with brick look (train query)
foutains, similarity of the two spellings: 0.668
foutains, products in both semantic top 10s: 0
foutains, semantic ndcg@10 as typed: 0.353
foutains, semantic ndcg@10 corrected: 0.645
foutains, bm25 ndcg@10 as typed: 0.356
foutains, bm25 ndcg@10 corrected: 0.507

desins: twin over full bunk beds cool desins | twin over full bunk beds cool designs (train query)
desins, similarity of the two spellings: 0.829
desins, products in both semantic top 10s: 6
desins, semantic ndcg@10 as typed: 0.565
desins, semantic ndcg@10 corrected: 0.616
desins, bm25 ndcg@10 as typed: 0.488
desins, bm25 ndcg@10 corrected: 0.488

ice-an: refrigerator with ice an water in door | refrigerator with ice and water in door (test query)
ice-an, similarity of the two spellings: 0.975
ice-an, products in both semantic top 10s: 9
ice-an, semantic ndcg@10 as typed: 0.807
ice-an, semantic ndcg@10 corrected: 0.829
ice-an, bm25 ndcg@10 as typed: 0.501
ice-an, bm25 ndcg@10 corrected: 0.501

spelling probes: 4
smallest two-sided permutation p-value possible with 4 queries: 0.125

一個錯字就可能改變某個查詢整個語意前 10 名。以訓練查詢foutains with brick look為例,輸入原樣的拼寫與校正後的拼寫共有0個相同商品,nDCG@10 由0.353變為0.645;BM25 也有變動,由0.356變為0.507。

至於其他探測查詢,分數的變動較小:其中唯一的測試查詢refrigerator with ice an water in door,由0.807變為0.829。這些結果都無法確立某種規律:在只有4個探測查詢的情況下,雙尾排列檢定所能得到的最小 p 值是0.125。

實用原則: 不要依固定的相似度來過濾。為每個向量記錄模型、修訂版本與提示詞,並在嵌入查詢的地方檢查它們。隨著商品目錄的變動重建索引並重新測量,嵌入之前先計算 token 數;若日誌顯示有錯字,則在嵌入查詢之前先校正拼字,並使用會檢查每個詞的相鄰詞的校正器(第 8 章)。

本章做出的決定

最近一次變動於第 2026.11.8 版

凡是訓練查詢能夠衡量的選擇,本章都以訓練查詢衡量。

  • 模型:Snowflake/snowflake-arctic-embed-m-v1.5,由 recordings/embeddings.py 在固定的修訂版本下記錄一次。它是開放模型,採用 Apache-2.0 授權,體積小,並且針對較短的向量訓練。查詢會加上模型卡要求的前綴,商品則不加。
  • **相似度:**單位長度向量的內積,也就是餘弦相似度。向量會先正規化一次,之後每次有任何更動,也會再正規化。
  • **搜尋:**透過 lib.vectors.exact_search 以 NumPy 做精確搜尋,因為商店的向量只佔 36.9 MB。精確搜尋同時也是任何索引的基準真值。
  • **若商品目錄大到無法使用精確搜尋:**採用 faiss HNSW,設定 M = 32,並將 efSearch 設為 144。這是訓練掃描中最小的設定,能讓每個查詢的真實前 10 名中至少保留 9 筆。在測試查詢上,它相對於精確搜尋的 recall@10 為 1.000。
  • **索引的召回率要以精確搜尋為基準來衡量,**使用訓練查詢,採用管線實際使用的 k,並且除了平均值之外,也要檢視表現最差的查詢。
  • **維度:**全數保留 768 維。縮短為 256 維,可將每項商品從 3,072 位元組減少到 1,024 位元組,但代價無法由訓練查詢確切判定,而且商店可節省的空間不多。
  • **探測查詢測試的是判定無法涵蓋的部分,**例如否定、代碼與拼字。它們以各自的前 10 名結果比較,與有判定的查詢分開處理;只有修正後的拼字,由於問的是同一件事,才會連同其原查詢的判定一併計分。
  • **已知的弱點交由其他階段處理:**代碼交給詞彙檢索,精確尺寸交給屬性欄位;待分詞器保留數字之後,也可交給詞彙檢索(第 5 章、第 8 章)。這兩者都應採用可供測試的過濾條件,而非混合加權(第 7 章)。否定則交給屬性過濾與重排序。
  • **本章所呈現的:**索引的好壞,取決於它在你實際上線的商品目錄與設定下,相對於精確搜尋所量得的召回率。探測發現,嵌入能處理大多數尺寸,但無法處理代碼,也無法處理地毯與咖啡桌配對中的否定,另有少數配對什麼也證明不了。

本章,於本版次中

版次 2026.11.10範例於 2026年11 月 19 通過測試本章 16 個範例 個範例中有 16 個通過

譯自英文書,版次 2026.11.10。

引用本節

格式

各種格式皆引用英文原書,本譯本即依循該原書。

每種格式都會標示本版。其連結會開啟目前版次,節標題會說明該節內文最後一次變動的時間。存取日期與擷取日期即為你開啟此頁的當日。

本頁內容

第 6 章,共 11 章: 語意檢索

  1. Turn text into embeddings
  2. Compare embeddings with cosine similarity
  3. Find the exact nearest neighbours with NumPy
  4. 以 HNSW 用召回率換取速度
  5. 在測試查詢上檢查索引
  6. Check queries that contain digits
  7. 預期產品代碼的權重很低
  8. 測試嵌入對否定的處理
  9. 選擇嵌入模型
  10. 將基準測試當作候選清單來讀
  11. 截斷維度,然後檢驗損失
  12. 正式環境中會出什麼問題
  13. 本章做出的決定

返回頂端所有章節