语义检索

阅读约需 84分钟 分钟,13节中含16个示例。

版次 2026.11.10 示例于 2026年11月19日 通过 132个示例中有132个通过 变化内容

AI 翻译。由人工以英文核验。 我们如何制作图书

译自英文版图书,版次 2026.11.10。 阅读原文

读完本章,你将能够基于已存储的嵌入向量为商品目录构建语义检索:用余弦相似度比较向量,用 numpy 精确找出与查询最接近的商品;当目录规模超出精确搜索的承受范围时,改用近似索引,并度量该索引遗漏了什么;还能找出嵌入向量出错的那些查询。你也可以用自己的相关性判断来测试并选定向量维度,并以同样的方式比较不同的模型。

对于 48 inch kitchen hood,语义检索会返回其他宽度的油烟机;对于 rug that is not round,则会返回圆形地毯。然而在测试查询上,它的平均 nDCG@10 为 0.830。它相对 BM25 的优势(BM25 的得分为 0.725)已由第 3 章确立。本书一直把它当作一个类来使用,即 lib.semantic.SemanticSearch。第 1 章为这家商店构建了它,商店的目录从沙发一直到浴帘,所用的向量由一个模型事先一次性算好。在商店依赖它之前,你需要弄清:这些向量中保存了什么;对于大得多的目录,搜索这些向量要付出什么代价;以及哪些查询它会出错。本章手工构建精确搜索,以它为基准度量一种近似索引,并在商店的查询和相关性判断上对两者进行测试。

在测试查询上,本章计划进行 2 次比较,均见于关于测试索引的一节。每次比较都让精确搜索(将查询与每一件商品比较)对阵 HNSW(一种近似索引,只将查询与其中一小部分商品比较):一次采用 faiss 的默认设置,一次采用在训练查询上选定的设置。每次比较都要符合本书的规则,这里即 p 值低于 0.025。本章对测试查询的其他任何查看,都属于探索性的。

这部分尚未翻译,因此以英文显示。

Turn text into embeddings

最近一次修改见于版次 2026.11.10

An embedding places texts with similar meanings near each other; this section opens how. The model that computes an embedding is an encoder, trained so that texts with similar meanings get vectors that point in similar directions, whether or not they share words. Every text gets the same count of numbers, the embedding's dimension, whether it's a three-word query or a product's whole description. One number means nothing on its own; only comparisons between whole vectors do.

Search uses the encoder as a bi-encoder: it encodes each product by itself, once, ahead of time, and each query by itself when the shopper types it. Comparing a query with a product is then arithmetic on two vectors rather than a run of the model, which is what makes it cheap enough to compare one query with a whole catalog. Chapter 1 gives the size of the difference from the Sentence-BERT paper: about 65 hours for BERT to read every pair of 10,000 sentences, against about 5 seconds to embed each sentence once and compare. A model that reads a query and a product together is a cross-encoder, and chapter 1 uses one to rerank a shortlist.

The store's encoder is Snowflake/snowflake-arctic-embed-m-v1.5, an open model with 109 million parameters under the Apache-2.0 licence (model card). Merrick et al. (2024) report that its family's training kept only documents identified as mainly English. We chose it because anyone may use it commercially, it's small, and it was trained so that its vectors can be shortened, which the section on dimensions tests. Examples read a recording instead: vectors that the script recordings/embeddings.py computed once, at a pinned revision, and saved at unit length, so every example sees the same numbers. The model card asks for a prefix on queries only, so the script adds it to each query and embeds products as they are. Examples read the vectors through lib.recorded.embeddings():

inspect-recorded-embeddings.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 通过
"""Loads the recorded vectors and shows the lengths lib.recorded stores and rescales, and how close two shopper queries are. The prose's point about rounding rests on the stored- and rescaled-length lines, and its point about nearby queries on the median and nearest-query lines."""import reimport numpy as npfrom lib.recorded import RECORDED, embeddings, manifestfrom lib.store import load_queries# unit-length rows, one per ID; the queries carry the model's query prefixproduct_ids, product_vectors, query_ids, query_vectors = embeddings()recording = manifest("embeddings")print("embedding model:", recording["model"])print("query prompt:", recording["prompts"]["query"])print(f"products embedded: {len(product_ids):,}")print("dimensions:", product_vectors.shape[1])print("stored as:", recording["dtype"])print("longest product text, tokens:", recording["longest_product_tokens"])print("model's input limit, tokens:", recording["max_seq_length"])# row 0 is product 1; [0, :4] takes its first four numbersprint(f"product {product_ids[0]}, first four numbers:", ", ".join(f"{x:.3f}" for x in product_vectors[0, :4]))stored = np.load(RECORDED / "embeddings" / "products.npy").astype(np.float32)   # as saved, before lib.recordedstored_lengths, lengths = np.linalg.norm(stored, axis=1), np.linalg.norm(product_vectors, axis=1)print(f"stored vector lengths: {stored_lengths.min():.5f} to {stored_lengths.max():.5f}")print(f"lengths after lib.recorded rescales: {lengths.min():.5f} to {lengths.max():.5f}")# How close are two shopper queries? Compare every query with every other one.queries = load_queries().loc[query_ids, "query"].tolist()similarity = query_vectors @ query_vectors.T  # unit length, so the dot product is the cosinenp.fill_diagonal(similarity, np.nan)               # leave out each query's similarity to itselfprint(f"median similarity of two queries: {np.nanmedian(similarity):.3f}")for query in ["whimsical solar lights", "wall design shelf"]:    row = queries.index(query)    nearest = int(np.nanargmax(similarity[row]))    # whole words as typed, with no stemming, so `lights` and `light` differ    shared = set(re.findall(r"\w+", query)) & set(re.findall(r"\w+", queries[nearest]))    print()    print(f"{query}, nearest other query:", queries[nearest])    print(f"{query}, similarity: {similarity[row, nearest]:.3f}")    print(f"{query}, words in common:", len(shared))
输出python inspect-recorded-embeddings.py
embedding model: Snowflake/snowflake-arctic-embed-m-v1.5
query prompt: Represent this sentence for searching relevant passages:
products embedded: 12,000
dimensions: 768
stored as: float16
longest product text, tokens: 201
model's input limit, tokens: 512
product 1, first four numbers: -0.033, 0.051, 0.111, -0.012
stored vector lengths: 0.99994 to 1.00006
lengths after lib.recorded rescales: 1.00000 to 1.00000
median similarity of two queries: 0.291

whimsical solar lights, nearest other query: outdoor light fixtures
whimsical solar lights, similarity: 0.637
whimsical solar lights, words in common: 0

wall design shelf, nearest other query: kitchen storage shelves
wall design shelf, similarity: 0.679
wall design shelf, words in common: 0

Each of the 12,000 products is a vector of 768 numbers; product 1's begins -0.033, 0.051, 0.111, -0.012, and none of those numbers means anything alone. The recording stores each number as a float16, in 2 bytes, and lib.recorded widens them to 32-bit floats when it loads them. That rounding moves the stored lengths slightly off 1: they run from 0.99994 to 1.00006. So lib.recorded also scales each vector back to length 1, and the lengths it returns run from 1.00000 to 1.00000. The store's encoder reads text as tokens, word pieces from its own vocabulary, not the terms of chapter 5's tokenizer. The longest product text is 201 tokens, inside the model's limit of 512 tokens, so the model read every product in full.

The median similarity between two of the store's queries is 0.291, on a scale where 1 means two vectors point the same way (cosine similarity). Yet whimsical solar lights finds outdoor light fixtures at 0.637, and wall design shelf finds kitchen storage shelves at 0.679, with 0 and 0 words in common: lights and light are different strings, and so are shelves and shelf. The baseline BM25 doesn't stem words, so it sees nothing shared in either pair. Chapter 5's stemming would match lights with light, but not shelves with shelf.

Working rule: Embed products ahead of time and queries at search time, with the same model at the same revision, and follow the model card on prompts: here, a prefix on queries only. Record the model, revision and prompts next to the vectors, as each manifest.json in examples/data/recorded/ does. Check that your longest product fits the model's input limit before you trust its embedding.

这部分尚未翻译,因此以英文显示。

Compare embeddings with cosine similarity

最近一次修改见于版次 2026.11.10

To rank products for a query, you need one number per pair that says how alike two vectors are. Cosine similarity is the cosine of the angle between two vectors: 1 when they point the same way, 0 at right angles and −1 when they point in opposite directions. It ignores how long the vectors are. The dot product multiplies two vectors number by number and adds up the results, so it equals the cosine times both lengths. Scaling every vector to length 1, called normalizing, makes the two the same number, as Sentence Transformers states. The dot product is then the cheaper choice, because it skips computing the lengths.

dot(q, d)    = q1 × d1 + q2 × d2 + ... + qn × dn
length(q)    = √(q1² + q2² + ... + qn²)
cosine(q, d) = dot(q, d) / (length(q) × length(d))

To see the formula at work, take the test query clear shower curtain and its most similar product: compute the dot product and the cosine by hand, and check the cosine with scikit-learn 1.9.1. Then break the one assumption that makes them agree. Keep the first 256 of each vector's 768 numbers, a shortcut that the section on dimensions tests, but skip the rescaling, and rank every query's products both ways:

cosine-by-hand.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 · scikit-learn 1.9.1 · scipy 1.18.1 通过
"""Computes the cosine of one test query and its best product by hand and with scikit-learn, then ranks every query's products by dot product and by cosine after the vectors are shortened. The prose's point rests on the line that counts queries whose top 10 changes."""import numpy as npfrom sklearn.metrics.pairwise import cosine_similarityfrom lib.recorded import embeddingsfrom lib.store import load_productsproducts = load_products()product_ids, product_vectors, query_ids, query_vectors = embeddings()query = query_vectors[list(query_ids).index(246)]           # clear shower curtainbest = int(np.argmax(product_vectors @ query))product = product_vectors[best]print("product:", products.at[product_ids[best], "name"])# cosine = dot product / (length of q * length of d)dot = query @ productcosine = dot / (np.linalg.norm(query) * np.linalg.norm(product))print(f"dot product: {dot:.3f}")print(f"cosine by hand: {cosine:.3f}")print(f"cosine from scikit-learn: {cosine_similarity([query], [product])[0, 0]:.3f}")# Keep the first 256 numbers of each vector and don't rescale: the lengths are no longer 1.short_products, short_queries = product_vectors[:, :256], query_vectors[:, :256]lengths = np.linalg.norm(short_products, axis=1)print(f"shortened product lengths: {lengths.min():.3f} to {lengths.max():.3f}")# stable sort: tied scores keep their stored orderby_dot = np.argsort(-(short_queries @ short_products.T), axis=1, kind="stable")[:, :10]# only the products are rescaled: a query's length scales all its scores alike, so it can't change their orderby_cosine = np.argsort(-(short_queries @ (short_products / lengths[:, None]).T), axis=1, kind="stable")[:, :10]# compares the sets: a new order within the same 10 doesn't countchanged = sum(set(dot_top) != set(cosine_top) for dot_top, cosine_top in zip(by_dot, by_cosine))print("queries:", len(query_ids))print("queries whose top 10 changes:", changed)
输出python cosine-by-hand.py
product: woolley vinyl solid color single shower curtain
dot product: 0.577
cosine by hand: 0.577
cosine from scikit-learn: 0.577
shortened product lengths: 0.706 to 0.778
queries: 80
queries whose top 10 changes: 61

On the store's unit-length vectors, the dot product is 0.577, the cosine by hand is 0.577, and scikit-learn's cosine_similarity gives 0.577, so the three agree.

Shortening breaks that. The shortened vectors have lengths from 0.706 to 0.778, and for 61 of the 80 queries, ranking by dot product gives a different top 10 from ranking by cosine. A query's own length scales all of its scores by the same factor, so it can't change their order. A product's length can: it multiplies that product's score for every query, a boost the shopper never asked for.

Some libraries report a distance instead of a similarity. faiss's L2 indexes return the squared Euclidean distance, and its inner product equals cosine similarity only when the vectors are normalized, according to the faiss wiki. For unit vectors, the squared distance is 2 − 2 × cosine, so it ranks products in the same order as cosine, smallest distance first.

Working rule: Normalize every vector once, when you store it, and rank by dot product. Normalize again after anything that changes a vector, such as shortening it or averaging several. If a model's documentation says it was trained for the dot product of unnormalized vectors, follow the documentation instead, and never mix the two in one index.

这部分尚未翻译,因此以英文显示。

Find the exact nearest neighbours with NumPy

最近一次修改见于版次 2026.11.9

Retrieval by embeddings is a nearest-neighbour search: find the k products whose vectors are most similar to the query's. Exact search compares the query with every product and keeps the best k, so it never misses one. With unit-length vectors, comparing one query with every product is one matrix–vector product, and comparing many queries is one matrix product, with a row per query and a column per product. What's left is to find the k highest scores in each row. Sorting every row does more work than needed. NumPy's argpartition moves the k highest scores to one side without ordering either side, and NumPy's documentation leaves that order undefined, so you sort only those k.

Chapter 1's semantic search took its rankings from lib.semantic.SemanticSearch. Build its core by hand for the 30 test queries with NumPy 2.5.5, then check it twice: against lib.semantic, and against faiss's IndexFlatIP, which computes the inner product with every vector. The faiss guidelines name two indexes that guarantee exact results, IndexFlatIP and IndexFlatL2:

exact-search-numpy.py Python 3.12.15 · faiss-cpu 1.16.0 · numpy 2.5.5 · pandas 3.1.1 通过
"""Builds exact search from one matrix product, then checks its top 10 against lib.semantic and faiss's IndexFlatIP. The prose's point rests on the two 'same top 10' lines, which show the hand-built search matches both."""import faissimport numpy as npfrom lib.metrics import evaluatefrom lib.recorded import embeddingsfrom lib.semantic import SemanticSearchfrom lib.store import load_queries, qrelsproduct_ids, product_vectors, query_ids, query_vectors = embeddings()test = load_queries("test").indexrows = np.isin(query_ids, test)def top_k(scores, k):    """Column indices of the k highest scores in each row, best first."""    # argpartition's kth is 0-based: with k - 1, the first k positions hold the k highest scores, in no order    candidates = np.argpartition(-scores, k - 1, axis=1)[:, :k]          # the top k, in no order    # stable sort: tied scores keep their candidate order    order = np.argsort(-np.take_along_axis(scores, candidates, axis=1), axis=1, kind="stable")    return np.take_along_axis(candidates, order, axis=1)scores = query_vectors[rows] @ product_vectors.T      # one row per test query, one column per producttop = top_k(scores, 10)print("similarities computed:", f"{scores.size:,}")print("similarities per query:", f"{scores.shape[1]:,}")semantic = SemanticSearch()# two lists match only if the same products come in the same ordersame = sum(list(product_ids[found]) == list(semantic.search(query_id, 10).index)           for query_id, found in zip(query_ids[rows], top))print("same top 10 as lib.semantic:", same, "of", rows.sum())flat = faiss.IndexFlatIP(product_vectors.shape[1])     # faiss's exact index: inner product with every productflat.add(product_vectors)_, found = flat.search(query_vectors[rows], 10)# the same products in the same order, row by rowprint("same top 10 as faiss IndexFlatIP:", (found == top).all(axis=1).sum(), "of", rows.sum())run = {query_id: list(product_ids[row]) for query_id, row in zip(query_ids[rows], top)}print(f"ndcg@10 of the exact run: {evaluate(run, qrels('test'))['ndcg@10'].mean():.3f}")
输出python exact-search-numpy.py
similarities computed: 360,000
similarities per query: 12,000
same top 10 as lib.semantic: 30 of 30
same top 10 as faiss IndexFlatIP: 30 of 30
ndcg@10 of the exact run: 0.830

The hand-built search agrees with both checks: it returns the same top 10 as lib.semantic for 30 of 30 test queries, and as faiss for 30 of 30, so its run scores the same mean nDCG@10, 0.830. One matrix product computed every similarity: 360,000 of them, 12,000 per query. The book's shared module, lib.vectors, keeps this top_k for later examples and wraps it in exact_search.

Whether exact search is enough comes down to its cost. Its work grows in step with the catalog: one similarity per product, for every query. Its memory is the vectors themselves:

exact-search-memory.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 通过
"""Prints the bytes one product's float32 vector takes, and the memory the catalog's vectors use, for the store and for a million products. The prose's point rests on the MB line for the store's catalog and the GB line for a million products."""from lib.recorded import embeddings# Exact search keeps every product vector in memory, as 32-bit floats.product_ids, product_vectors, _, _ = embeddings()print(f"products: {len(product_ids):,}")print(f"float32 bytes per product: {product_vectors[0].nbytes:,}")# decimal MB and GB: a megabyte is a million bytes, a gigabyte a billionprint(f"product vectors in memory, MB: {product_vectors.nbytes / 1e6:.1f}")print(f"the same for 1,000,000 products, GB: {1_000_000 * product_vectors[0].nbytes / 1e9:.1f}")
输出python exact-search-memory.py
products: 12,000
float32 bytes per product: 3,072
product vectors in memory, MB: 36.9
the same for 1,000,000 products, GB: 3.1

Each product takes 3,072 bytes as 32-bit floats, so the store's catalog takes 36.9 MB, and a million products would take 3.1 GB. Douze et al. (2024) advise deciding first whether an index is needed at all, because in some cases a brute-force search is the best option. We judge the store's catalog to be such a case.

Working rule: We recommend exact search until it no longer fits in memory or meets your latency budget, measured with your own catalog on your own hardware. It has no settings to tune and never misses a neighbour, so it is also the ground truth for any approximate index. Store the normalized vectors as one 32-bit matrix, take the top k with argpartition, and sort only those.

用 HNSW 以召回换速度

最近一次修改见于版次 2026.11.10

当精确搜索代价过高时,近似最近邻(ANN)索引会跳过其中大部分比较,作为回报,它可能漏掉一部分真正的最近邻。ANN 索引的质量用它相对于精确搜索的 recall@k 来衡量:即精确搜索的前 k 个结果中,它也返回了的比例。这不同于第 3 章的 recall@k,后者是把排序结果与相关性判断相比较。一个相对于精确搜索召回率完美的索引,返回的正是精确搜索所返回的结果,无论结果好坏。

HNSW 是 hierarchical navigable small world(分层可导航小世界)的缩写,是一种图索引。每个商品是一个节点,与它的若干最近邻相连,节点分布在多层结构中:最底层包含所有商品,往上每一层包含的商品逐层减少。搜索从顶层开始,移动到与查询最近的相连节点;当无法再靠近时下降一层;到达底层后,先探索一个候选列表,再返回最好的 k 个结果(Malkov and Yashunin, 2018)。在 faiss 中,M 设定每个节点的连接数。连接越多,能找到的真正近邻越多,但占用的内存也越多(faiss wiki)。efConstruction 是构建时的扩展因子,决定构建时考虑多少候选;efSearch 是搜索时的扩展因子,决定一次搜索考虑多少候选。在 faiss 1.16.0 中,它们的默认值分别为 40 和 16,并且图的构建是确定性的,因为 add() 使用的唯一构建方式就是确定性的(HNSW.h;changelog)。因此这里的输出可以精确复现。计时结果因机器而异,所以示例改为统计工作量:faiss 会统计每次搜索中计算的距离数(HNSW.h),对这个索引而言,这些距离就是相似度。该计数不包括沿连接跳转和读取内存的开销,所以它是速度的参考,而不是度量。

这个商店并不需要索引,但拥有数百万商品的商店就需要;而一个小到可以逐一核对每个答案的商品目录,正是学习这种权衡的合适场所。用 hnsw_index(product_vectors, links=32) 在商店的 12,000 个商品上构建 HNSW,其中 lib.vectors 用该名称表示 M = 32。然后在训练查询上,让 efSearch 从 16 扫描到 256:

hnsw-efsearch-sweep.py Python 3.12.15 · faiss-cpu 1.16.0 · numpy 2.5.5 · pandas 3.1.1 通过
"""Builds HNSW with M = 32 and sweeps efSearch on the train queries against exact search, printing recall, the worst query and the work per query. The prose's point rests on the efSearch 16 lines and the two smallest-efSearch lines printed after the table."""import faissimport numpy as npfrom lib.recorded import embeddingsfrom lib.store import load_products, load_queries# EF_SEARCH_GRID runs 16 to 256 in steps of 16; WORST_RECALL is 0.9: at least 9 of the true top 10from lib.vectors import EF_SEARCH_GRID, WORST_RECALL, exact_search, hnsw_index, hnsw_search, recall_against_exactproduct_ids, product_vectors, query_ids, query_vectors = embeddings()names = load_products().loc[product_ids, "name"].str[:36].to_numpy()   # by row number, cut to 36 charactersrows = np.isin(query_ids, load_queries("train").index)                 # tune on train queries onlyqueries = load_queries().loc[query_ids[rows], "query"].to_numpy()index = hnsw_index(product_vectors, links=32)          # M = 32, and faiss's default efConstruction of 40levels = faiss.vector_to_array(index.hnsw.levels)      # how many layers each product reachesfor layer in range(1, levels.max() + 1):    print(f"products in layer {layer}: {(levels >= layer).sum():,}")exact = exact_search(query_vectors[rows], product_vectors, 10)print("\nefSearch  mean recall@10  worst  queries at 0  similarities per query")sweep = {}for ef_search in EF_SEARCH_GRID:    # similarities is faiss's count of the distances each search computed, per query, not a score    found, similarities = hnsw_search(index, query_vectors[rows], 10, ef_search)    # share of exact search's top 10 that the approximate search also found, order ignored    recall = recall_against_exact(found, exact)    sweep[ef_search] = (recall, similarities)    print(f"{ef_search:>8}  {recall.mean():>14.3f}  {recall.min():>5.1f}  {(recall == 0).sum():>12}  {similarities:>22.0f}")recall, similarities = sweep[16]                       # faiss's defaultprint(f"\nefSearch 16, recall@10 against exact: {recall.mean():.3f}")print(f"efSearch 16, similarities computed per query: {similarities:.0f}")print(f"efSearch 16, train queries with recall@10 of 0: {(recall == 0).sum()}")print("efSearch 16, those queries:", ", ".join(queries[recall == 0]))print("smallest efSearch with no query at 0:", min(ef for ef, (r, _) in sweep.items() if r.min() > 0))chosen = min(ef for ef, (r, _) in sweep.items() if r.min() >= WORST_RECALL)print(f"smallest efSearch with every query at {WORST_RECALL} or more:", chosen)print(f"chosen efSearch, recall@10 against exact: {sweep[chosen][0].mean():.3f}")print(f"chosen efSearch, similarities computed per query: {sweep[chosen][1]:.0f}\n")# the top 3 at the default efSearch, for the query that failsfound, _ = hnsw_search(index, query_vectors[rows], 3, 16)row = list(queries).index("foutains with brick look")for rank, (approximate, nearest) in enumerate(zip(found[row], exact[row]), start=1):    print(f"{rank}  hnsw: {names[approximate]:<36}  exact: {names[nearest]}")
输出python hnsw-efsearch-sweep.py
products in layer 1: 12,000
products in layer 2: 369
products in layer 3: 10

efSearch  mean recall@10  worst  queries at 0  similarities per query
      16           0.934    0.0             2                     282
      32           0.936    0.0             2                     346
      48           0.940    0.0             2                     404
      64           0.940    0.0             2                     463
      80           0.956    0.0             1                     530
      96           0.978    0.3             0                     611
     112           0.980    0.3             0                     682
     128           0.994    0.8             0                     767
     144           0.998    0.9             0                     835
     160           0.998    0.9             0                     924
     176           0.998    0.9             0                    1020
     192           0.998    0.9             0                    1115
     208           0.998    0.9             0                    1191
     224           1.000    1.0             0                    1271
     240           1.000    1.0             0                    1349
     256           1.000    1.0             0                    1441

efSearch 16, recall@10 against exact: 0.934
efSearch 16, similarities computed per query: 282
efSearch 16, train queries with recall@10 of 0: 2
efSearch 16, those queries: foutains with brick look, sheffield home bath set
smallest efSearch with no query at 0: 96
smallest efSearch with every query at 0.9 or more: 144
chosen efSearch, recall@10 against exact: 0.998
chosen efSearch, similarities computed per query: 835

1  hnsw: dayse outdoor square cushion with fi  exact: tribeca 2 '' x 10 '' porcelain brick
2  hnsw: 3 piece wall decor set                exact: barbados 5 '' x 10 '' ceramic brick
3  hnsw: rustic autumn ivy branch              exact: kimary 0.75 '' x 1.5 '' glass brick

图的规模缩减得很快:最底层包含全部 12,000 个商品,下一层有 369 个,顶层有 10 个。

平均值掩盖了一种失败。在默认的 efSearch 为 16 时,每次查询计算 282 次相似度,而不是 12,000 次,相对于精确搜索的平均 recall@10 为 0.934。然而有 2 个训练查询完全没有找到各自真正的前 10 个结果中的任何一个。

这些查询是 foutains with brick look, sheffield home bath set。以其中之一 foutains with brick look 为例:HNSW 返回了一个靠垫、一套墙面装饰和一根常春藤枝条,而精确搜索的前 3 个结果是砖纹瓷砖。

在扫描中,使得没有任何训练查询为 0 的最小 efSearch 是 96,但扫描表显示,它最差的查询仍然漏掉了真正前 10 个结果中的大部分。因此应采用更严格的规则:选择使每个训练查询都至少找到其真正前 10 个结果中 9 个的最小 efSearch。它是 144,平均召回率为 0.998,每次查询计算 835 次相似度。

工作准则: 只有当精确搜索太慢或太占空间时,才使用 ANN 索引。在训练查询上,按它相对于精确搜索的召回率进行调优,使用流水线所用的 k,并按最差的查询而不是平均值来选择。我们建议选择使每个查询都至少保留其真正前 10 个结果中 9 个的最小 efSearch,并配合 M = 32。本章只调优 efSearch;M 和 efConstruction 会改变图本身,在大型商品目录上同样值得做这样的扫描。Aumüller et al. (2018) 发现,差异很大的各种 ANN 方法能达到相当的质量与性能权衡,所以应在自己的数据上测量,而不是凭名气判断。

在测试查询上检验索引

最近一次修改见于版次 2026.11.8

在训练查询上调好的索引,仍须在它没有见过的查询上检验:既看它相对精确检索的召回率,也看它对人工判定得分的影响。这是本章计划的 2 项比较,每一项都用配对置换检验,并附上t 检验:将 faiss 默认 efSearch 为 16 的 HNSW 索引,以及在训练查询上选定的设置,分别与精确检索对比。检索同时要求 k = 100,因为混合检索会融合每个检索器的前 100 个结果:

hnsw-test-recall.py Python 3.12.15 · faiss-cpu 1.16.0 · numpy 2.5.5 · pandas 3.1.1 · scipy 1.18.1 通过
"""Compares HNSW at faiss's default efSearch, and at the setting chosen on train queries, with exact search on the test queries, using paired tests at the planned threshold. The prose's point rests on the default setting's permutation and interval lines, and on the chosen setting's recall line."""import numpy as npfrom lib.metrics import evaluatefrom lib.recorded import embeddingsfrom lib.significance import compare, verdictfrom lib.store import load_queries, qrelsfrom lib.vectors import as_run, choose_ef_search, exact_search, hnsw_index, hnsw_search, recall_against_exact# Planned before scoring: HNSW at faiss's default efSearch, and at the setting chosen on train, each against exact.PLANNED = 2threshold = 0.05 / PLANNED  # the book's rule splits 0.05 across the planned comparisonsproduct_ids, product_vectors, query_ids, query_vectors = embeddings()rows = np.isin(query_ids, load_queries("test").index)index = hnsw_index(product_vectors, links=32)# the smallest efSearch at which every train query keeps WORST_RECALL of its true top 10chosen = choose_ef_search(index, query_vectors[np.isin(query_ids, load_queries("train").index)], product_vectors)# k = 100 because hybrid retrieval fuses each retriever's top 100exact = exact_search(query_vectors[rows], product_vectors, 100)exact_ndcg = evaluate(as_run(exact, query_ids[rows], product_ids), qrels("test"))["ndcg@10"]print("planned test comparisons:", PLANNED)print(f"p-value threshold: {threshold:.3f}")print("efSearch chosen on train queries:", chosen)print(f"exact ndcg@10: {exact_ndcg.mean():.3f}")for name, ef_search in [("default", 16), ("chosen", chosen)]:    found, _ = hnsw_search(index, query_vectors[rows], 100, ef_search)    # the first 10 of the 100    recall = recall_against_exact(found[:, :10], exact[:, :10])    ndcg = evaluate(as_run(found, query_ids[rows], product_ids), qrels("test"))["ndcg@10"]    result = compare(ndcg, exact_ndcg, confidence=1 - threshold)  # a 97.5% interval, at the test's level    print(f"\n{name}, recall@10 against exact: {recall.mean():.3f}")    print(f"{name}, queries with recall@10 of 0: {(recall == 0).sum()}")    print(f"{name}, those queries:", ", ".join(load_queries().loc[query_ids[rows][recall == 0], "query"]) or "none")    print(f"{name}, recall@100 against exact: {recall_against_exact(found, exact).mean():.3f}")    print(f"{name}, hnsw ndcg@10: {ndcg.mean():.3f}")    # compare rounds each difference to 3 decimals: wins and losses are the nonzero ones    print(f"{name}, queries whose ndcg@10 changes: {result['wins'] + result['losses']}")    print(f"{name}, permutation p: {result['permutation p']:.3f}")    print(f"{name}, t-test p: {result['t-test p']:.3f}")    print(f"{name}, {1 - threshold:.1%} interval: {result['ci low']:.4f} to {result['ci high']:.4f}")    for check, passed in verdict(result, threshold).items():      # established, t-test agrees, interval agrees        print(f"{name}, {check}: {'yes' if passed else 'no'}")
输出python hnsw-test-recall.py
planned test comparisons: 2
p-value threshold: 0.025
efSearch chosen on train queries: 144
exact ndcg@10: 0.830

default, recall@10 against exact: 0.930
default, queries with recall@10 of 0: 2
default, those queries: liberty hardware francisco, 48" sliding single track , barn door for laundry
default, recall@100 against exact: 0.849
default, hnsw ndcg@10: 0.791
default, queries whose ndcg@10 changes: 3
default, permutation p: 0.256
default, t-test p: 0.154
default, 97.5% interval: -0.1544 to -0.0006
default, established: no
default, t-test agrees: yes
default, interval agrees: no

chosen, recall@10 against exact: 1.000
chosen, queries with recall@10 of 0: 0
chosen, those queries: none
chosen, recall@100 against exact: 0.994
chosen, hnsw ndcg@10: 0.830
chosen, queries whose ndcg@10 changes: 0
chosen, permutation p: 1.000
chosen, t-test p: 1.000
chosen, 97.5% interval: 0.0000 to 0.0000
chosen, established: no
chosen, t-test agrees: yes
chosen, interval agrees: yes

在训练查询上选定的设置经受住了检验。它相对精确检索的 recall@10 为 1.000,平均 nDCG@10 为 0.830,与精确检索的 0.830 相同,有 0 个查询的得分发生变化。

在 faiss 的默认设置下,有 2 个测试查询一个真实的前 10 结果也没有取到(liberty hardware francisco, 48" sliding single track , barn door for laundry),平均 nDCG@10 降至 0.791,但得分发生变化的查询只有 3 个,这一下降并未得到确证:置换检验给出 p = 0.256,t 检验与之一致,p = 0.154。97.5% 区间为 -0.1544 to -0.0006,不包含 0,因此与两项检验的结论不一致。由于承载这一差异的查询太少,这里的发现是相对精确检索损失了召回,而不是测得了质量的损失。

增加返回结果的数量并不能弥补过小的 efSearch:默认设置下 recall@100 为 0.849,选定设置下则为 0.994。

除向量之外,图结构也要占用内存:

hnsw-index-memory.py Python 3.12.15 · faiss-cpu 1.16.0 · numpy 2.5.5 · pandas 3.1.1 通过
"""Counts the links per product in the HNSW graph, and the bytes per product of the exact and the HNSW index. The prose's point rests on the two bytes-per-product lines, set against faiss's estimate."""import faissfrom lib.recorded import embeddingsfrom lib.vectors import hnsw_indexproduct_ids, product_vectors, _, _ = embeddings()dimensions = product_vectors.shape[1]flat = faiss.IndexFlatIP(dimensions)                    # exact search: the vectors onlyflat.add(product_vectors)index = hnsw_index(product_vectors, links=32)           # the vectors, plus the graph's links# layer 0 is the bottom layer: 2 × M links there, and M on each upper layerprint("links per product, upper layers:", index.hnsw.nb_neighbors(1))print("links per product, bottom layer:", index.hnsw.nb_neighbors(0))# 8M with M = 32: 4 bytes for each of the 2M bottom-layer linksprint(f"faiss's estimate, 4d + 8M bytes per product: {4 * dimensions + 8 * 32:,}")# the saved index's size, spread over the productsfor name, built in [("exact index", flat), ("hnsw index", index)]:    print(f"bytes per product, {name}: {faiss.serialize_index(built).nbytes / len(product_ids):,.0f}")
输出python hnsw-index-memory.py
links per product, upper layers: 32
links per product, bottom layer: 64
faiss's estimate, 4d + 8M bytes per product: 3,328
bytes per product, exact index: 3,072
bytes per product, hnsw index: 3,344

每个商品在上层有 32 条链接,在底层有 64 条,是 M 的两倍。faiss 估计每个向量占 4d + 8M 字节(faiss wiki):d 个数各占 4 字节,底层的 2M 条链接各占 4 字节,此处为 3,328。保存下来的索引与之接近,每个商品 3,344 字节,而精确索引为 3,072 字节。

工作准则: 用相对精确检索的召回率在留出查询上检验调好的索引,并且先查看那些丢掉整个前 k 结果的查询,再看平均得分。如果 nDCG@10 的下降只依赖少数几个查询,在配对检验确证之前,应视为未经证实。内存预算既要算向量,也要算链接。

这部分尚未翻译,因此以英文显示。

Check queries that contain digits

最近一次修改见于版次 2026.11.10

An index can only return what the embeddings put near a query, so the next question is which queries the embeddings get wrong. An encoder squeezes a whole query into one vector, so a short token that changes the meaning, such as a size, a model number or a negation, may move that vector very little. Deng et al. (2026) measured this for numbers, in a study of 13 embedding models on synthetic financial text. Their accuracy at telling texts apart by a number was 0.54, only slightly above the 0.5 of random guessing, and four-digit numbers were at the level of guessing. Products aren't financial reports, so check the store's queries.

Few of the store's queries hold a digit, so compare BM25 and semantic retrieval on each that does, from both splits. Nothing is tuned on these queries, and a handful can't establish anything:

digits-in-queries.py Python 3.12.15 · bm25s 0.3.16 · numpy 2.5.5 · pandas 3.1.1 · scipy 1.18.1 通过
"""Scores BM25 and semantic retrieval on each query with a digit, in both splits, and counts the products whose text writes 7qt or 7 qt. The prose's point rests on the three lines that count which ranker wins and the two product counts."""import pandas as pdfrom lib.lexical import LexicalSearchfrom lib.metrics import evaluatefrom lib.semantic import SemanticSearchfrom lib.store import load_products, load_queries, product_text, qrelsproducts, queries = load_products(), load_queries()with_digit = queries[queries["query"].str.contains(r"\d")]       # both splits: a diagnosis, nothing is tunedjudgments = {query_id: grades for query_id, grades in qrels().items() if query_id in with_digit.index}# each ranker returns its top 100; nDCG@10 scores the first 10 of themscores = pd.DataFrame({    "split": with_digit["split"],    "bm25": evaluate(LexicalSearch(products).run(with_digit, k=100), judgments)["ndcg@10"],    "semantic": evaluate(SemanticSearch().run(with_digit, k=100), judgments)["ndcg@10"],    "query": with_digit["query"],}).round(3)print("queries with a digit:", len(scores))print(scores.to_string())print("semantic better:", (scores["semantic"] > scores["bm25"]).sum())print("bm25 better:", (scores["bm25"] > scores["semantic"]).sum())# a gap under 0.03 counts as a near tie: the book's choice, not a measured limitprint("gaps under 0.03:", ((scores["semantic"] - scores["bm25"]).abs() < 0.03).sum())text = product_text(products).str.lower()print("products whose text has the word 7qt:", text.str.contains(r"\b7qt\b").sum())# '7 qt', but not '17 qt' or '3.7 qt'print("products whose text has 7 qt:", text.str.contains(r"(?<![\d.])7 qt\b").sum())
输出python digits-in-queries.py
queries with a digit: 7
          split   bm25  semantic                                             query
query_id
45        train  0.288     0.316                            70s inspired furniture
62        train  0.554     0.930                                   7qt slow cooker
101       train  0.569     0.770        48 in entry table with side by side drawer
203       train  0.606     0.740                         ge top loading washer 4.5
205       train  0.935     0.930                          ines shag rug w003096809
285        test  0.592     0.555  48" sliding single track , barn door for laundry
482        test  1.000     0.921                              48 inch kitchen hood
semantic better: 4
bm25 better: 3
gaps under 0.03: 2
products whose text has the word 7qt: 0
products whose text has 7 qt: 8

Neither ranker owns the 7 queries with a digit. Semantic retrieval scores higher on 4 of them and BM25 on 3, and 2 of the gaps are under 0.03. BM25's clearest loss is 7qt slow cooker: the word 7qt appears in 0 products, and 8 write 7 qt.

On the test query 48 inch kitchen hood, BM25 scores higher, so here is semantic retrieval's top 10:

hood-query-digits.py Python 3.12.15 · bm25s 0.3.16 · numpy 2.5.5 · pandas 3.1.1 · scipy 1.18.1 通过
"""Lists semantic retrieval's top 10 for 48 inch kitchen hood with each hood's width, read from the attributes column the embeddings never saw. The prose's point rests on the two width-48 lines, one for each ranker."""from lib.lexical import LexicalSearchfrom lib.semantic import SemanticSearchfrom lib.store import load_products, load_queries, qrelsLABELS = {2: "exact", 1: "partial", 0: "irrelevant"}products = load_products()query_id = 482  # 48 inch kitchen hood, a test queryquery = load_queries().at[query_id, "query"]judgments = qrels()[query_id]# The width sits in the attributes column, which the embeddings never saw: product_text leaves it out.width = products["attributes"].str.extract(r"\bwidth: ([\d.]+)", expand=False).astype(float)semantic = SemanticSearch().search(query_id, k=10).indexbm25 = LexicalSearch(products).search(query, k=10).indexprint("query:", query)print("     judgment   width  name")for rank, product_id in enumerate(semantic, start=1):    label = LABELS.get(judgments.get(product_id), "unjudged")    print(f"{rank:>2}   {label:<10} {width[product_id]:>5g}  {products.at[product_id, 'name'][:50]}")print()# exact: a 48-inch hood is 48 wide, where probe-pair-overlap allows half an inch for hoods listed at 35.75for name, top in [("semantic", semantic), ("bm25", bm25)]:    print(f"{name}, width 48 in the top 10:", (width[top] == 48).sum())    print(f"{name}, exact in the top 10:", sum(judgments.get(product_id) == 2 for product_id in top))
输出python hood-query-digits.py
query: 48 inch kitchen hood
     judgment   width  name
 1   exact         48  48 '' 1200 cfm ducted wall mount range hood in bru
 2   exact         48  48 '' 400 cfm ducted wall mount range hood in brus
 3   exact         48  48 '' 400 cfm convertible wall mount range hood
 4   partial    35.75  36 '' ws-48 series 500 cfm ducted under cabinet ra
 5   exact         48  48 '' hand crafted designer wood island hood serie
 6   exact         48  48 '' island range hood
 7   exact         48  48 inch wall mount chimney range hood with 600 cfm
 8   exact         48  48 '' ducted island range hood
 9   exact         48  48 '' professional 900cfm ducted island range hood
10   partial     29.8  30 '' 480 cfm convertible under cabinet range hood

semantic, width 48 in the top 10: 8
semantic, exact in the top 10: 8
bm25, width 48 in the top 10: 10
bm25, exact in the top 10: 10

For this query, the embedding matched the digits as text, not the width they stand for. Of semantic retrieval's top 10, 8 hoods have a width of 48. The others are a 36-inch hood from a series named ws-48 and a 30-inch hood rated at 480 cfm, cubic feet per minute, the airflow of its fan. Both contain the digits 48, neither is 48 wide, and the judges marked both partial. BM25's top 10 holds 10 hoods of width 48.

Each hood's width comes from the attributes column, which holds curated fields from WANDS's product features, such as width, shape and material. product_text, the text the model embedded, leaves that column out, so the embeddings never saw it, and a size the shopper types can be checked against it exactly.

One query can't show a pattern. To see how the model handles sizes in general, use probe queries: pairs of queries written for the book that differ only in a size, a code or a negation, which recordings/probes.py embeds with the same model. One of them repeats the test query 48 inch kitchen hood word for word. Probes have no judgments, so the example counts their top 10s and never scores them. The right size comes from the width in attributes for the hoods, and from the name for the rest:

probe-pair-overlap.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 通过
"""Counts, for each probe pair, the products the two top 10s share and the top-10 products of the right size, without scoring anything. The prose's point rests on the right-size lines for the hood, slow-cooker and bed probes, and on the seat-count and loveseat lines for the sofa probe."""from lib.recorded import probesfrom lib.semantic import SemanticSearchfrom lib.store import load_products# Probe queries written for the book, embedded by the same model. They have no judgments,# so this example counts what each top 10 holds; it doesn't score anything.probe_queries, probe_vectors = probes()products = load_products()names = products["name"]width = products["attributes"].str.extract(r"\bwidth: ([\d.]+)", expand=False).astype(float)semantic = SemanticSearch()def states(size, unit):    """Names that state this size, such as "7 qt" or "7-quart", but not "3.7 qt"."""    return names.str.contains(rf"(?<![\d.]){size}[ -]?(?:{unit})")RIGHT_SIZE = {                       # the size each probe asks for: the width attribute, or the name    # within half an inch of 36 or 48, so a 35.75-inch hood counts as 36    "36 inch kitchen hood": (width - 36).abs() <= 0.5, "48 inch kitchen hood": (width - 48).abs() <= 0.5,    "4 qt slow cooker": states(4, "qt|quart"), "7 qt slow cooker": states(7, "qt|quart"),    # the whole word, which also matches 'twin over full' bunk beds    "twin bed": names.str.contains(r"\btwin\b"), "king bed": names.str.contains(r"\bking\b"),    "2 seat sofa": names.str.contains(r"\bloveseat\b"),        # a two-seat sofa is a loveseat}DIGIT_ONLY = {"2 seat sofa": states(2, " in 1| piece"), "4 seat sofa": states(4, " in 1| piece")}for pair in ["hood", "cooker", "bed", "seats"]:    first, second = probe_queries.index[probe_queries["pair"] == pair]  # each pair is two probes, in file order    tops = {row: semantic.search_vector(probe_vectors[row], k=10).index for row in (first, second)}    print(f"{pair}, similarity of the two queries: {probe_vectors[first] @ probe_vectors[second]:.3f}")    print(f"{pair}, products in both top 10s:", len(set(tops[first]) & set(tops[second])))    for row, top in tops.items():        query = probe_queries.at[row, "query"]        if query in RIGHT_SIZE:            print(f"{query}, right size in the top 10:", RIGHT_SIZE[query][top].sum())        if query in DIGIT_ONLY:            print(f"{query}, same digit but a piece count or 2 in 1:", DIGIT_ONLY[query][top].sum())    print()# the two counts the sofa probes ask forprint("catalog names that give a seat count:", (states(2, "seat") | states(4, "seat")).sum())print("catalog products named loveseat:", names.str.contains(r"\bloveseat\b").sum())
输出python probe-pair-overlap.py
hood, similarity of the two queries: 0.795
hood, products in both top 10s: 0
36 inch kitchen hood, right size in the top 10: 8
48 inch kitchen hood, right size in the top 10: 8

cooker, similarity of the two queries: 0.841
cooker, products in both top 10s: 1
4 qt slow cooker, right size in the top 10: 6
7 qt slow cooker, right size in the top 10: 7

bed, similarity of the two queries: 0.697
bed, products in both top 10s: 0
twin bed, right size in the top 10: 10
king bed, right size in the top 10: 10

seats, similarity of the two queries: 0.759
seats, products in both top 10s: 1
2 seat sofa, right size in the top 10: 0
2 seat sofa, same digit but a piece count or 2 in 1: 4
4 seat sofa, same digit but a piece count or 2 in 1: 4

catalog names that give a seat count: 0
catalog products named loveseat: 14

For these probes, the model mostly finds the right size. The two hood probes share 0 products, yet each top 10 holds hoods of the right width, 8 for 36 inches and 8 for 48; the slow-cooker probes find 6 and 7 of the right capacity, and the bed probes 10 and 10 of the right size. Where the catalog has no number to match, the digit takes over. The number of product names that give a seat count is 0, so the top 10 for 2 seat sofa holds 4 products that share only the digit, two-piece sets and "2 in 1" kids' sofas, and 0 of the catalog's 14 loveseats.

预期编码对产品编号几乎不赋予权重

最近一次修改见于版次 2026.11.8

产品编号应当与某一个产品精确匹配。但对编码器来说,它只是一串罕见的字母和数字。探测查询中有两对用于测试编号:一对基于没有任何产品持有的编号,取自训练查询 ines shag rug w003096809;另一对基于恰好只有一个产品的文本中出现的屏幕分辨率:

code-probes.py Python 3.12.15 · bm25s 0.3.16 · numpy 2.5.5 · pandas 3.1.1 · scipy 1.18.1 通过
"""Compares the top 10s of two probe queries that differ only in a code no product holds, and shows where one product ranks for two resolutions, semantically and with BM25. The prose's point rests on the shared-top-10 line for the code and on the rank lines for the resolutions."""from lib.lexical import LexicalSearchfrom lib.recorded import probesfrom lib.semantic import SemanticSearchfrom lib.store import load_products, product_text# Probe queries written for the book, embedded by the same model; no judgments, so nothing is scored.probe_queries, probe_vectors = probes()products = load_products()text = product_text(products).str.lower()semantic, bm25 = SemanticSearch(), LexicalSearch(products)# A code no product holds: does changing it change the results?first, second = probe_queries.index[probe_queries["pair"] == "code"]tops = [set(semantic.search_vector(probe_vectors[row], k=10).index) for row in (first, second)]print("products whose text contains w003096809:", text.str.contains("w003096809").sum())print(f"code, similarity of the two queries: {probe_vectors[first] @ probe_vectors[second]:.3f}")print("code, products in both top 10s:", len(tops[0] & tops[1]))# A code one product holds: where does that product rank, with its code and with another?holder = text.index[text.str.contains("1920x1080p")]print()print("products whose text contains 1920x1080p:", len(holder))print("that product:", products.at[holder[0], "name"])print("products in the Projectors class:", (products["product_class"] == "Projectors").sum())for row in probe_queries.index[probe_queries["pair"] == "resolution"]:    query = probe_queries.at[row, "query"]    ranked = semantic.search_vector(probe_vectors[row], k=len(products))  # ranks every product, so the rank is exact    lexical = list(bm25.search(query, k=100).index)  # BM25 ranks are read from its top 100 only    print(f"{query}, its semantic rank: {list(ranked.index).index(holder[0]) + 1}")  # ranks count from 1    print(f"{query}, its similarity: {ranked[holder[0]]:.3f}")    # the best product other than this one    print(f"{query}, similarity of the next product: {ranked.drop(holder[0]).iloc[0]:.3f}")    print(f"{query}, its bm25 rank:", lexical.index(holder[0]) + 1 if holder[0] in lexical else "not in the top 100")
输出python code-probes.py
products whose text contains w003096809: 0
code, similarity of the two queries: 0.967
code, products in both top 10s: 10

products whose text contains 1920x1080p: 1
that product: 5000 portable transmissive projector
products in the Projectors class: 4
1920x1080p projector, its semantic rank: 1
1920x1080p projector, its similarity: 0.562
1920x1080p projector, similarity of the next product: 0.496
1920x1080p projector, its bm25 rank: 1
2560x1440p projector, its semantic rank: 1
2560x1440p projector, its similarity: 0.509
2560x1440p projector, similarity of the next product: 0.508
2560x1440p projector, its bm25 rank: not in the top 100

改动未知编号的最后几位数字后,两个查询的相似度为 0.967,它们的前 10 名结果共有 10 个相同产品。与任何产品都不匹配的编号不会带来任何变化,BM25 也同样无从匹配。

只有一个产品持有的编号,会决定 BM25 的排序,却只能轻微推动嵌入的排序。分辨率 1920x1080p 只出现在5000 portable transmissive projector的文本中。对于 1920x1080p projector,BM25 将它排在 1;对于 2560x1440p projector,它排在 not in the top 100。Sciavolino 等人(2021)发现,在涉及罕见实体的问题上,稠密(基于嵌入的)检索器的表现远逊于稀疏方法;而 Karpukhin 等人(2020)则指出,BM25 对高度选择性的关键词和短语反应灵敏。

嵌入对该产品的排序,在使用其自身编号时为 1,使用另一个编号时为 1,原因是商店里只有 4 款投影仪。编号只体现在它与下一个产品的差距上:带有编号时为 0.562 对 0.496,不带编号时为 0.509 对 0.508。

工作准则: 列出日志中包含尺寸、数量或编号的查询,在信任任何一种排序器之前,先用两种排序器分别对这些查询做比较。编号重要时,用词汇匹配来匹配它。精确尺寸重要时,将它与产品字段匹配,例如 attributes 中的宽度;或者在分词器保留数字、且单位写法统一之后,采用词汇匹配(第 5 章、第 8 章)。查询的其余部分交给语义检索。第 7 章测试了将两者结合的做法。

用否定句测试嵌入

最近一次修改见于版次 2026.11.8

否定只用一个短词就能反转顾客的需求:一块不是圆形的地毯。整条查询的嵌入向量,可能仍与去掉这个词的同一查询非常接近。NevIR 是一个由仅相差一处否定的文档对构成的基准,它发现没有任何双编码器架构的配对准确率超过 12%,而随机排序的得分为 25%(Weller et al., 2024)。在 2025 年的一项复现研究中,van den Elsen et al. 发现,有一个 70 亿参数的双编码器 GritLM-7B 达到了 39.0%,其他双编码器几乎没有提升,只有交叉编码器和列表式 LLM 重排器能较好地完成这两项否定任务。

商店的查询中有 0 含有 "no"、"not"、"without" 或 "non",因此这些相关性判断无法用来测试否定。我们改用 3 对探测查询,每对由一条查询及其否定形式组成,并统计各自前 10 个结果中包含什么。

目录本可以满足每一条带否定的探测查询:其中有 42 个产品名称含 armless,有 62 张咖啡桌的描述从未提到玻璃,还有 90 块地毯的名称不含 round。

运行之前,先猜一猜:对于 rug that is not round,语义检索的前 10 个结果中有多少个名称含 round。

negation-probes.py Python 3.12.15 · bm25s 0.3.16 · numpy 2.5.5 · pandas 3.1.1 · scipy 1.18.1 通过
"""Counts what each negation probe's top 10 holds for semantic retrieval and for BM25, without scoring it. The prose's point rests on the semantic top 10 lines for the rug, coffee-table and sofa probes."""import bm25sfrom lib.lexical import LexicalSearchfrom lib.recorded import probesfrom lib.semantic import SemanticSearchfrom lib.store import load_products, load_queries, product_text# Probe queries written for the book, with no judgments: count what the top 10 holds, never score it.probe_queries, probe_vectors = probes()products = load_products()names, text = products["name"].str.lower(), product_text(products).str.lower()semantic, bm25 = SemanticSearch(), LexicalSearch(products)# armless and round are matched in product names, glass in the full textFEATURES = {"arms": ("armless", names), "glass": ("glass", text), "round": ("round", names)}negated = load_queries()["query"].str.contains(r"\b(?:no|not|without|non)\b")print("store queries with no, not, without or non:", negated.sum())# the tokenizer settings LexicalSearch uses, so these are the terms BM25 searchestokens = bm25s.tokenize(["rug that is not round"], stopwords="en", return_ids=False, show_progress=False)print("bm25 terms of rug that is not round:", " ".join(tokens[0]))print("products named armless:", names.str.contains(r"\barmless\b").sum())coffee_tables = products["product_class"] == "Coffee & Cocktail Tables"area_rugs = products["product_class"] == "Area Rugs"print("coffee tables that never mention glass:", (coffee_tables & ~text.str.contains(r"\bglass\b")).sum())print("area rugs not named round:", (area_rugs & ~names.str.contains(r"\bround\b")).sum())negation = probe_queries[probe_queries["kind"] == "negation"]print("negation probe pairs:", negation["pair"].nunique())for row, probe in negation.iterrows():    word, field = FEATURES[probe["pair"]]    has_feature = field.str.contains(rf"\b{word}\b")    semantic_top = semantic.search_vector(probe_vectors[row], k=10).index    bm25_top = bm25.search(probe["query"], k=10).index    print()    print(f"{probe['query']}, semantic top 10 with {word}:", has_feature[semantic_top].sum())    print(f"{probe['query']}, bm25 top 10 with {word}:", has_feature[bm25_top].sum())
输出python negation-probes.py
store queries with no, not, without or non: 0
bm25 terms of rug that is not round: rug round
products named armless: 42
coffee tables that never mention glass: 62
area rugs not named round: 90
negation probe pairs: 3

sofa with arms, semantic top 10 with armless: 1
sofa with arms, bm25 top 10 with armless: 0

sofa without arms, semantic top 10 with armless: 8
sofa without arms, bm25 top 10 with armless: 0

coffee table with glass top, semantic top 10 with glass: 10
coffee table with glass top, bm25 top 10 with glass: 9

coffee table not glass, semantic top 10 with glass: 10
coffee table not glass, bm25 top 10 with glass: 10

round rug, semantic top 10 with round: 10
round rug, bm25 top 10 with round: 10

rug that is not round, semantic top 10 with round: 10
rug that is not round, bm25 top 10 with round: 10

如果你猜的是很少,说明你像顾客一样理解了 "not",而嵌入没有。在地毯和咖啡桌的探测中,它保留了属性,丢掉了 "not"。对于 rug that is not round,语义检索的前 10 个结果中有 10 个产品名称含 round;对于 round rug,这一数字是 10。对于 coffee table not glass,前 10 个结果中有 10 个提到了玻璃,而 coffee table with glass top 为 10。

sofa without arms 的情况则相反,它的前 10 个结果中有 8 个名称含 armless,而 sofa with arms 为 1。目录中表示这种否定的只有一个词 armless,模型把 "without arms" 放在了它的附近。

BM25 在这些探测上也没有更好。它的分词器会把 "not"、"that" 和 "is" 当作停用词丢弃,因此它把 rug that is not round 当作 rug round 来检索。它的前 10 个结果中有 10 块圆形地毯和 0 张无扶手沙发。

这些探测只有寥寥几对,不能算作测量,而且沙发这一对的结果正好相反。但地毯和咖啡桌这两对与 NevIR 的发现一致:查询向量仍然靠近查询中提到的属性。

工作准则: 不要依赖第一阶段的检索器(无论是词法检索还是语义检索)来处理否定。当顾客排除的属性在你的目录中有存储时,例如形状、材质或扶手,就检测这种排除并按该属性过滤。否则就做重排,因为在 van den Elsen et al. 的复现中,交叉编码器和 LLM 重排器处理否定的效果要好得多。第 9 章用交叉编码器和 LLM 对商店的语义检索结果进行重排,而在商店已有相关性判断的查询上,它的所有重排器都没有胜过语义检索,这些查询无法测试否定。请保留这类探测,作为检查手段,每当更换模型或加入重排器时就重新运行。

选择嵌入模型

模型决定了向量能够“知道”什么,而日后更换模型,就意味着要把整个商品目录重新嵌入一遍。公开基准测试可以缩小候选范围,最终则要由商店自己的判断来决定。

将基准测试视为候选清单

最近一次修改见于版次 2026.11.8

MTEB(Massive Text Embedding Benchmark,大规模文本嵌入基准)在众多任务上为模型打分。其作者发现,没有任何单一方法能在所有任务上占优(Muennighoff et al., 2023)。作者还发现,其平均分偏向数据集较多的任务,尤其是检索、分类和聚类,因此应当查看与自己的任务相对应的分数。

公开分数也可能高估模型在未见过的数据上的表现。检索基准 RTEB 的制作者指出,模型在公开基准上报告的分数与其在全新、未见数据上的实际表现之间存在差距(Liu et al., 2025)。RTEB 的 48 个数据集中有 28 个不公开。Verma et al. (2026) 发现,RTEB 对模型的排名与 MTEB 和 BEIR(Thakur et al., 2021,一个早期的跨多个领域的检索基准)的排名之间,相关系数仅为 ρ = 0.38。这里的 ρ 是秩相关系数,1 表示排序完全一致。规模也不等于质量。Enevoldsen et al. (2025) 报告称,在 MTEB 的多语言后继基准 MMTEB 中,公开可用的最佳模型是 multilingual-e5-large-instruct,它只有 5.6 亿个参数。

商店考虑过四个开源模型,它们的模型卡都没有报告在 WANDS 上的结果。其中包括商店所用的编码器和 Qwen/Qwen3-Embedding-0.6B(模型卡),另外两个是 BAAI/bge-base-en-v1.5(模型卡)和 ibm-granite/granite-embedding-english-r2(模型卡)。ESCI 等商品搜索基准更接近商店的任务,它把查询与商品的配对标注为完全匹配、替代、互补或不相关,包含约 130,000 个查询和 260 万条判断(Reddy et al., 2022),但它们仍然不是商店自己的商品目录。因此,商店的候选清单依据的是排行榜无法完成的检查:允许商用的许可证、英语支持、无需特殊硬件即可运行的规模,以及对较短向量的文档化支持。商店所选模型的模型卡报告其在 MTEB 检索任务上的 nDCG@10 为 55.14,这是厂商在通用文本而非商品上的测量结果(模型卡)。要在候选模型之间做出选择,应在相同的商品文本和查询上记录每个模型的结果,在训练查询上为各模型打分,并用本书的配对检验进行比较,下一小节对同一模型的多种规模所做的正是这件事。

截断维度,然后检验损失

最近一次修改见于版次 2026.11.8

每个维度以 32 位浮点数存储时,每件商品占 4 字节,而每次相似度计算都要用到全部维度,因此更短的向量在存储和搜索上都更便宜。Matryoshka(套娃)训练让模型把最有用的信息放在前面的维度中,于是向量的开头部分本身就可以作为一个更小的嵌入使用。Kusupati 等人(2022)报告称,在 ImageNet-1K 分类任务中,嵌入最多可缩小到原来的 1/14,而准确率保持在同一水平。缩短向量节省的是存储和搜索开销,而不是编码开销:正如 Sentence Transformers 的文档所指出的,Matryoshka 模型运行起来并不会更快。只有以这种方式训练的模型才能经受住截断,而且缩短后的向量不再是单位长度,因此应当重新归一化,模型卡自带的示例也是这样做的。

商店的编码器有一份模型卡,其中记录了可缩短到 256 维。lib.recorded.embeddings(dims=...)保留每个向量的前 dims 个数,并再次归一化。在训练查询上为每种长度打分:

truncate-embeddings.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 通过
"""Scores semantic retrieval on the train queries at 768, 256 and 128 dimensions, with each product's float32 size. The prose's point rests on the nDCG@10 line for each length, set against 768 dimensions."""from lib.metrics import evaluatefrom lib.semantic import SemanticSearchfrom lib.store import load_queries, qrelstrain, judgments = load_queries("train"), qrels("train")       # choose on train queries# dims keeps the first dims numbers of every vector and rescales each to length 1 (lib.recorded).for dims in [768, 256, 128]:    semantic = SemanticSearch(dims=dims)    scores = evaluate(semantic.run(train, k=100), judgments)    # semantic.products holds the product vectors, not the product table    print(f"{dims} dimensions, float32 bytes per product: {semantic.products[0].nbytes:,}")    print(f"{dims} dimensions, ndcg@10: {scores['ndcg@10'].mean():.3f}")    print(f"{dims} dimensions, judged@10: {scores['judged@10'].mean():.3f}")    print()
输出python truncate-embeddings.py
768 dimensions, float32 bytes per product: 3,072
768 dimensions, ndcg@10: 0.772
768 dimensions, judged@10: 0.964

256 dimensions, float32 bytes per product: 1,024
256 dimensions, ndcg@10: 0.762
256 dimensions, judged@10: 0.952

128 dimensions, float32 bytes per product: 512
128 dimensions, ndcg@10: 0.680
128 dimensions, judged@10: 0.874

缩短到 256 维后,每件商品从 3,072 字节降到 1,024 字节,平均 nDCG@10 从 0.772 变为 0.762。

在 128 维时,平均值为 0.680,judged@10为 0.874,而 768 维时为 0.964,因此这一下降有一部分可能来自无人评判过的商品。

这一变化是真实的,还是 50 个查询之间的噪声?用 lib.significance.compare来检验,它会运行第 3 章的置换检验及其配对 t 检验,同时还会报告一个自助法区间。这些比较使用的是训练查询,因此它们是在调整向量大小,不能作为测试查询上的证据:

truncation-paired-test.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 · scipy 1.18.1 通过
"""Tests the change in nDCG@10 from shortening to 256 and 128 dimensions against the full vectors, on the train queries, with a permutation test, a paired t-test and a bootstrap interval. The prose's point rests on the permutation, interval and effect-size lines for each length."""from lib.metrics import evaluatefrom lib.semantic import SemanticSearchfrom lib.significance import comparefrom lib.store import load_queries, qrels# Train queries: these comparisons tune the vector size. They are not evidence about the test queries.train, judgments = load_queries("train"), qrels("train")def ndcg(dims):    return evaluate(SemanticSearch(dims=dims).run(train, k=100), judgments)["ndcg@10"]def p_value(p):    return "below 0.001" if p < 0.001 else f"{p:.3f}"full = ndcg(768)for dims in [256, 128]:    # compare's default interval is 95%    result = compare(ndcg(dims), full)                    # paired by train query: shortened minus full    print(f"{dims} vs 768, mean difference in ndcg@10: {result['mean difference']:.3f}")    print(f"{dims} vs 768, queries worse: {result['losses']}")    print(f"{dims} vs 768, permutation test p: {p_value(result['permutation p'])}")    print(f"{dims} vs 768, paired t-test p: {p_value(result['t-test p'])}")    print(f"{dims} vs 768, 95% bootstrap interval: {result['ci low']:.4f} to {result['ci high']:.4f}")    print(f"{dims} vs 768, interval excludes 0: {'yes' if result['ci high'] < 0 or result['ci low'] > 0 else 'no'}")    print(f"{dims} vs 768, effect size: {result['effect size']:.3f}")    print()
输出python truncation-paired-test.py
256 vs 768, mean difference in ndcg@10: -0.010
256 vs 768, queries worse: 24
256 vs 768, permutation test p: 0.076
256 vs 768, paired t-test p: 0.080
256 vs 768, 95% bootstrap interval: -0.0211 to -0.0002
256 vs 768, interval excludes 0: yes
256 vs 768, effect size: -0.253

128 vs 768, mean difference in ndcg@10: -0.092
128 vs 768, queries worse: 33
128 vs 768, permutation test p: below 0.001
128 vs 768, paired t-test p: below 0.001
128 vs 768, 95% bootstrap interval: -0.1569 to -0.0565
128 vs 768, interval excludes 0: yes
128 vs 768, effect size: -0.547

在 128 维时,损失是明确的。平均差为 -0.092,有 33 个训练查询变差。置换检验给出 p below 0.001,来自 -0.1569 to -0.0565 的 95% 区间不包含 0。

在 256 维时,证据处于临界状态,因为检验与区间结论不一致。平均差为 -0.010;置换检验给出 p = 0.076,t 检验给出 p = 0.080,然而来自 -0.0211 to -0.0002 的区间却以极小的幅度不包含 0。效应量在 256 维时为 -0.253,在 128 维时为 -0.547,这说明 256 维时即使有损失,也是两者中较小的一个。

对商店而言,决定取决于其规模,而不取决于检验。其完整向量占用 36.9 MB,可节省的空间很少;因此它保留全部 768 个维度,测试得分也不会改变。若是拥有数亿件商品的目录,则会在每件商品从 3,072 字节降到 1,024 字节所带来的节省,与这么多训练查询也无法确定的损失之间权衡,并且会在决定之前评判更多查询。

工作准则: 按许可证、语言、规模和领域筛选候选模型,把基准排名只当作提示。在你自己的训练查询上用配对检验在模型之间或向量大小之间做决定,并且既看 p 值也看区间。只对经过相应训练的模型缩短向量,缩短后要归一化,并且只在内存或搜索时间有要求时才这样做。

生产环境中会出什么问题

最近一次修改见于版次 2026.11.8

相似度不等于置信度。 当最佳相似度较低时,什么都不展示,这种做法很诱人。但不同查询的相似度并不在同一个尺度上:

similarity-scale.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 通过
"""Finds the test query with the lowest best similarity and the one with the highest 10th similarity, and counts the products above a 0.5 cutoff. The prose's point rests on the two similarity lines and the cutoff counts."""import numpy as npfrom lib.recorded import embeddingsfrom lib.semantic import SemanticSearchfrom lib.store import load_queries, qrelsproduct_ids, product_vectors, query_ids, query_vectors = embeddings()test = load_queries("test")rows = np.isin(query_ids, test.index)similarity = np.sort(query_vectors[rows] @ product_vectors.T, axis=1)[:, ::-1]   # best first, per queryqueries = test.loc[query_ids[rows], "query"].to_numpy()weakest = similarity[:, 0].argmin()                    # the query whose best match is least similarstrongest = similarity[:, 9].argmax()                  # the query whose 10th match is most similarprint("first query, lowest best similarity:", queries[weakest])print(f"its best similarity: {similarity[weakest, 0]:.3f}")judgments = qrels()[query_ids[rows][weakest]]top = SemanticSearch().search(query_ids[rows][weakest], k=10).indexprint("exact products in its top 10:", sum(judgments.get(product_id) == 2 for product_id in top))print("second query, highest 10th similarity:", queries[strongest])print(f"its 10th similarity: {similarity[strongest, 9]:.3f}")cutoff = 0.5  # an illustrative cutoff; nothing about the model sets itprint()print(f"products above {cutoff} for the first query:", (similarity[weakest] > cutoff).sum())print(f"products above {cutoff} for the second query:", (similarity[strongest] > cutoff).sum())print(f"test queries with no product above {cutoff}:", (similarity[:, 0] <= cutoff).sum())
输出python similarity-scale.py
first query, lowest best similarity: liberty hardware francisco
its best similarity: 0.436
exact products in its top 10: 5
second query, highest 10th similarity: kohler whitehaven farmhouse kitchen sink
its 10th similarity: 0.647

products above 0.5 for the first query: 0
products above 0.5 for the second query: 103
test queries with no product above 0.5: 2

测试查询中的两个极端说明了这一点。liberty hardware francisco的最佳匹配最弱,仅为 0.436,但它的前 10 个结果中却有 5 个精确匹配的商品;而kohler whitehaven farmhouse kitchen sink的第 10 个匹配最为相似,为 0.647。

若以 0.5 为截断值,第一个查询会保留 0 个商品,第二个查询会保留 103 个,而有 2 个测试查询则一个也不会保留。语义检索也从不会返回空结果:即使是一个没有任何商品使用的代码,也仍然会得到完整的前 10 个结果。判断某个查询没有匹配项,应当另寻办法,例如当词法检索找不到该查询中的任何罕见词时。

向量不匹配会悄无声息地出错。 查询向量和商品向量必须来自同一个模型、同一个修订版本,并使用相同的提示词。来自两个模型的向量是不同空间中的数字,当两者维度相同时,对它们做比较不会引发任何错误。Sentence Transformers 的 encode_query 在存在预定义的查询提示时会自动加上该提示(API 参考)。请确保对实时查询做嵌入的代码与索引代码的做法一致。

索引不会自行跟随商品目录变化。 根据 faiss 维基,faiss 的 HNSW 不支持删除向量,因此已停售的商品会一直留在图中,直到你在结果中将其过滤掉或重建索引。Douze 等人(2024)指出,faiss 不是数据库:它不支持并发写入、分片或事务。相对于精确搜索的召回率只对应某一个商品目录和某一组设置,因此每次重建之后都要重新测量。

过长的文本会被截断。 模型最多读取 512 个词元,而 Sentence Transformers 的 API 参考说明,更长的输入会被截断。该店最长的商品文本为 201 个词元,但在描述很长的商品目录中,超出上限的部分模型根本读不到。请像 recordings/embeddings.py 那样,在做嵌入之前先统计词元数。

拼写错误会使向量发生偏移。 Zhuang 和 Zuccon(2021)发现,对于未经拼写错误训练的基于 BERT 的检索器,拼写错误可能导致检索和排序效果显著下降。该店的查询中有 4 个含有拼写错误,每个都有一个拼写正确的探测查询。修正后的探测查询所要查找的内容与原查询相同,因此可以用原查询的相关性判断为它打分:

spelling-probes.py Python 3.12.15 · bm25s 0.3.16 · numpy 2.5.5 · pandas 3.1.1 · scipy 1.18.1 通过
"""Scores each correctly spelled probe with the judgments of the query it corrects, for semantic retrieval and BM25, as typed and as corrected. The prose's point rests on the semantic nDCG@10 lines for each spelling and on the smallest-p line."""from lib.lexical import LexicalSearchfrom lib.metrics import ndcg_at_kfrom lib.recorded import probesfrom lib.semantic import SemanticSearchfrom lib.store import load_products, load_queries, qrels# Each spelling probe is the correctly spelled text of one store query (of_query_id), embedded by the same# model. It asks for the same thing, so it is scored with that query's judgments. A diagnosis, not a test.probe_queries, probe_vectors = probes()products, queries, judgments = load_products(), load_queries(), qrels()semantic, bm25 = SemanticSearch(), LexicalSearch(products)spelling = probe_queries[probe_queries["kind"] == "spelling"]for row, probe in spelling.iterrows():    query_id, pair = int(probe["of_query_id"]), probe["pair"]    typed, corrected = semantic.search(query_id, k=10).index, semantic.search_vector(probe_vectors[row], k=10).index    print(f"{pair}: {queries.at[query_id, 'query']} | {probe['query']} ({queries.at[query_id, 'split']} query)")    print(f"{pair}, similarity of the two spellings: {semantic.query_vectors[query_id] @ probe_vectors[row]:.3f}")    print(f"{pair}, products in both semantic top 10s:", len(set(typed) & set(corrected)))    print(f"{pair}, semantic ndcg@10 as typed: {ndcg_at_k(list(typed), judgments[query_id], 10):.3f}")    print(f"{pair}, semantic ndcg@10 corrected: {ndcg_at_k(list(corrected), judgments[query_id], 10):.3f}")    for name, text in [("as typed", queries.at[query_id, "query"]), ("corrected", probe["query"])]:        print(f"{pair}, bm25 ndcg@10 {name}: {ndcg_at_k(list(bm25.search(text, k=10).index), judgments[query_id], 10):.3f}")    print()print("spelling probes:", len(spelling))# a sign-flip test on n queries has 2 ** n outcomes, and its two most extreme give the smallest two-sided pprint(f"smallest two-sided permutation p-value possible with {len(spelling)} queries: {2 / 2 ** len(spelling):.3f}")
输出python spelling-probes.py
ligth: ligth bulb | light bulb (train query)
ligth, similarity of the two spellings: 0.710
ligth, products in both semantic top 10s: 5
ligth, semantic ndcg@10 as typed: 1.000
ligth, semantic ndcg@10 corrected: 1.000
ligth, bm25 ndcg@10 as typed: 0.945
ligth, bm25 ndcg@10 corrected: 0.945

foutains: foutains with brick look | fountains with brick look (train query)
foutains, similarity of the two spellings: 0.668
foutains, products in both semantic top 10s: 0
foutains, semantic ndcg@10 as typed: 0.353
foutains, semantic ndcg@10 corrected: 0.645
foutains, bm25 ndcg@10 as typed: 0.356
foutains, bm25 ndcg@10 corrected: 0.507

desins: twin over full bunk beds cool desins | twin over full bunk beds cool designs (train query)
desins, similarity of the two spellings: 0.829
desins, products in both semantic top 10s: 6
desins, semantic ndcg@10 as typed: 0.565
desins, semantic ndcg@10 corrected: 0.616
desins, bm25 ndcg@10 as typed: 0.488
desins, bm25 ndcg@10 corrected: 0.488

ice-an: refrigerator with ice an water in door | refrigerator with ice and water in door (test query)
ice-an, similarity of the two spellings: 0.975
ice-an, products in both semantic top 10s: 9
ice-an, semantic ndcg@10 as typed: 0.807
ice-an, semantic ndcg@10 corrected: 0.829
ice-an, bm25 ndcg@10 as typed: 0.501
ice-an, bm25 ndcg@10 corrected: 0.501

spelling probes: 4
smallest two-sided permutation p-value possible with 4 queries: 0.125

一个拼写错误就可能改变某个查询的整个语义前 10 结果。以训练查询 foutains with brick look 为例,按原样拼写和修正后的拼写共有 0 个商品,nDCG@10 从 0.353 变为 0.645;BM25 也有变化,从 0.356 变为 0.507。

对于其他探测查询,分数的变化较小:其中唯一的测试查询 refrigerator with ice an water in door 从 0.807 变为 0.829。这些都不足以确立某种规律:在只有 4 个探测查询的情况下,可能得到的最小双侧置换检验 p 值为 0.125。

工作准则: 不要依据固定的相似度做过滤。为每个向量记录模型、修订版本和提示,并在嵌入查询的地方进行核对。随着商品目录的变化,重建索引并重新测量,在嵌入之前统计词元数,如果日志显示存在拼写错误,就在嵌入查询之前先纠正拼写,并使用能检查每个词的相邻词的纠错器(第 8 章)。

本章做出的决定

最近一次修改见于版次 2026.11.8

凡是训练查询能够衡量的选择,本章都用它们做了衡量。

  • 模型:Snowflake/snowflake-arctic-embed-m-v1.5,由 recordings/embeddings.py 在固定的修订版本上记录一次。它开源,采用 Apache-2.0 许可,体积小,并且针对较短的向量做过训练。查询要加上其模型卡所要求的前缀,商品则不加。
  • **相似度:**单位长度向量的点积,即余弦相似度。向量只归一化一次,此后每次改动之后再归一化一次。
  • **搜索:**通过 lib.vectors.exact_search 使用 NumPy 做精确搜索,因为商店的向量只占 36.9 MB。精确搜索也是任何索引的基准真值。
  • **如果商品目录大到精确搜索无法胜任:**使用 faiss HNSW,取 M = 32,并把 efSearch 设为 144。这是训练扫描中最小的设置,在该设置下每个查询的真实前 10 名中至少保留 9 个。在测试查询上,它相对精确搜索的 recall@10 为 1.000。
  • 索引的召回率要对照精确搜索来衡量,使用训练查询,取流水线所用的 k,并且既看平均值,也看最差的查询。
  • **维度:**全部 768 维。若缩短到 256 维,每个商品将从 3,072 字节降到 1,024 字节,而这一代价训练查询无法确定,商店本来也没有多少可节省的空间。
  • 探测查询检验的是判定无法检验的内容,例如否定、编码和拼写。它们按各自的前 10 名进行比较,与有判定的查询分开;只有改正后的拼写,因为问的是同一件事,才会连同其原查询的判定一起评分。
  • **已知的薄弱环节交给其他阶段:**编码交给词法检索,精确尺寸交给属性字段,或者在分词器保留数字之后交给词法检索(第 5 章、第 8 章)。对这两者,应采用待测试的过滤器,而不是混合加权(第 7 章)。否定交给属性过滤器和重排序。
  • **本章所展示的:**索引的好坏,只取决于它在你所发布的商品目录和设置上,相对精确搜索所测得的召回率。探测发现,嵌入能够处理大多数尺寸,但处理不了编码,也处理不了地毯与咖啡桌这两组配对中的否定,另有少数几组配对什么也证明不了。

本章,在本版次中

版次 2026.11.10示例于 2026年11月19日 通过本章 16个示例 个示例中有 16 个通过

译自英文版图书,版次 2026.11.10。

引用本节

格式

每种格式均引用英文原书,本译本以其为准。

每种格式都会注明该版次。其链接会打开当前版次,节标题处会说明该节文本最后一次修改的时间。访问日期和检索日期即为您打开此页的当天。

本页内容

第 6 章,共 11 章: 语义检索

  1. Turn text into embeddings
  2. Compare embeddings with cosine similarity
  3. Find the exact nearest neighbours with NumPy
  4. 用 HNSW 以召回换速度
  5. 在测试查询上检验索引
  6. Check queries that contain digits
  7. 预期编码对产品编号几乎不赋予权重
  8. 用否定句测试嵌入
  9. 选择嵌入模型
  10. 将基准测试视为候选清单
  11. 截断维度,然后检验损失
  12. 生产环境中会出什么问题
  13. 本章做出的决定

返回顶部全部章节