相關性的意義

閱讀約需 79 分鐘 分鐘,11 節 中共有 10 個範例。

版次 2026.11.10 範例於 2026年11 月 19 通過測試 132 個範例中有 132 個通過 變更內容

由 AI 翻譯。經人工以英文驗證。 我們如何製作書籍

譯自英文書,版次 2026.11.10。 閱讀原文

讀完本章後,你能以批判的眼光閱讀一組相關性判斷:說明每個等級對商品提出了什麼主張,找出這些判斷的來源與缺漏之處,並決定該在多大程度上信任由它們算出的分數。

到目前為止,每個分數都把商店的判斷當成事實。第 1 章以 13,077 筆判斷為依據替兩個排序器評分,這些判斷是人們對「在商店輸入的查詢」與「商店販售的商品」所配成的組合給予的等級。在測試查詢上,BM25 的平均 nDCG@10 為 0.725。語意檢索的分數則為 0.830。這些數字都建立在判斷default 默默做出的選擇之上:判斷記錄的是誰對相關性的看法、差一點就命中值多少分,以及無人判斷過的商品會如何處理。

本章仔細閱讀商店的判斷。本章不更動任何排序器,因此同時閱讀訓練與測試查詢:閱讀判斷並不是調校,而本書中的每項修正仍然只依據訓練查詢來選定。本章在測試查詢上執行的少數檢定屬於探索性質,因此只用來描述判斷,並不能確立任何差異(本書的規則)。第 3 章會把 BM25 與語意檢索之間的差距,當作事先規劃的比較來檢定。

此部分尚未翻譯,因此以英文呈現。

Judge relevance against the shopper's need

最近一次變動於第 2026.11.8 版

A search system returns products that match a query in the way the system defines a match: shared words for lexical retrieval, nearby vectors for semantic retrieval. Information science calls that system relevance. It separates that from topical relevance, whether a product is about the query's subject, and situational relevance, whether it helps with the task at hand (Saracevic, 2007). Some argue that system relevance is always perfect, because the system retrieved what the query asked for. Saracevic's answer is "Not so."

A judgment is a verdict, and this chapter asks what each grade claims. The store's judgments come from WANDS, a public product-search dataset, through a sample of its queries and products (Load the store's data). WANDS's judges label each pair exact, partial or irrelevant, which the book grades 2, 1 and 0. Exact means the product is the kind of thing the query asks for and has every attribute the query names. Partial means it is the right kind of thing but misses an attribute. Irrelevant means it isn't the kind of thing asked for (Read intent as a product and its modifiers). The judge grades the product against the information need behind the query's words, the thing the shopper wanted, not against the words. Manning et al. define a document as relevant when it addresses the stated information need, not merely because it contains the query's words (Manning et al., 2008, chapter 8). Evaluation then makes a bargain. It fixes a set of documents, a set of need statements and their judgments, and abstracts away the real users (Voorhees, 2007). The judgments stand in for shoppers here, and every score in this book inherits that bargain.

If matching words were the same as relevance, a product whose text holds every word of the query would be exact. You can test that on every judged pair in the store: for each pair, ask whether the product's name, class, category and description contain every word of the query, with the sample's descriptions cut to about 400 characters.

Of the judged pairs whose text holds every query word, what share did the judges call exact? Hold a number in mind.

word-match-versus-judgment.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 通過
"""Checks whether a full word match makes a judged pair exact, and how many exact pairs have a text that misses aquery word, with and without attributes. The prose's point rests on the `of them exact` line and the`of them, still missing one with attributes added` line."""import refrom lib.store import load_labels, load_products, load_queries, product_textdef words(text):    """Lowercase word tokens: a rough stand-in for what lexical search matches."""    return set(re.findall(r"\w+", text.lower()))products, queries, labels = load_products(), load_queries(), load_labels()text_words = {product_id: words(text) for product_id, text in product_text(products).items()}attribute_words = {product_id: words(text) for product_id, text in products["attributes"].items()}query_words = {query_id: words(query) for query_id, query in queries["query"].items()}pairs = list(zip(labels["query_id"], labels["product_id"]))# on sets, <= asks whether every query word is in the product's textlabels["in text"] = [query_words[query_id] <= text_words[product_id] for query_id, product_id in pairs]# | is set union: the product's text words with its attribute words addedlabels["in text or attributes"] = [query_words[query_id] <= text_words[product_id] | attribute_words[product_id]                                   for query_id, product_id in pairs]matched = labels[labels["in text"]]print(f"exact share of all judged pairs: {(labels['label'] == 'exact').mean():.3f}")print(f"judged pairs whose text has every query word: {len(matched):,}")for label in ["exact", "partial", "irrelevant"]:    print(f"of them {label}: {(matched['label'] == label).mean():.3f}")exact = labels[labels["label"] == "exact"]print(f"exact pairs: {len(exact):,}")print(f"exact pairs whose text misses a query word: {(~exact['in text']).sum():,}")print(f"of them, still missing one with attributes added: {(~exact['in text or attributes']).sum():,}")curtains = list(exact.loc[exact["query_id"] == 246, "product_id"])                 # query 246: clear shower curtainunsaid = [product_id for product_id in curtains if "clear" not in text_words[product_id]]print("exact products for clear shower curtain:", len(curtains))print("of them without the word clear in their text:", len(unsaid))print("of those, with clear in their attributes:", sum("clear" in attribute_words[product_id] for product_id in unsaid))
輸出python word-match-versus-judgment.py
exact share of all judged pairs: 0.249
judged pairs whose text has every query word: 1,177
of them exact: 0.777
of them partial: 0.211
of them irrelevant: 0.012
exact pairs: 3,252
exact pairs whose text misses a query word: 2,337
of them, still missing one with attributes added: 2,251
exact products for clear shower curtain: 12
of them without the word clear in their text: 8
of those, with clear in their attributes: 4

A full word match is a strong signal, but it doesn't settle the question. Of the 1,177 judged pairs whose text holds every query word, 0.777 are exact, 0.211 partial and 0.012 irrelevant, against an exact share of 0.249 over all judged pairs. The words can be present without the product being what the shopper meant, so the judges decide, not the words. A share near 1 would mean matching words is relevance, and the judges say it isn't.

The other direction is more striking. Most exact pairs, 2,337 of the 3,252, have a product text that misses at least one query word. Adding the attributes still leaves 2,251 of those missing a word, so attributes explain few misses overall.

Some misses are word forms and spelling, such as chairs against chair (chapter 1) or the query ligth bulb. Chapter 5 and chapter 8 take those up. Others are attributes the text doesn't state. WANDS records attributes such as color, material and size apart from the text (Chen et al., 2022, Table 1), and the store keeps a few of them in an attributes column, for example color: clear; material: vinyl (data card). The text the store's search reads, lib.store.product_text, leaves that column out.

For some queries, though, the attributes hold the answer. For clear shower curtain, 8 of the 12 exact products never say clear in their text, and 4 of those carry clear in their attributes, where the store's text search never looks. Chapter 5 tests adding attributes to lexical search. The rest of those curtains say clear nowhere, in their text or their attributes.

The judges had more to go on. WANDS's annotation guidelines show them each product's name, image and web page (guidelines). The photo or the page may be what told them, but that is our guess, because the data doesn't record which evidence a judge used.

Working rule: Treat a judgment as a person's verdict about a need, not a property of the product. Before you blame a ranker for missing an exact product, check whether any field the ranker reads says what the judges saw, and look at the fields it doesn't read, such as attributes. When your judges see more than your ranker does, give the ranker the same evidence where you can, or accept that some exact products will rank low for reasons no change to the ranking can fix.

預期評審者會在部分配對上意見不一

最近一次變動於第 2026.11.8 版

兩個人可能得出不同的判定。WANDS 的每個標籤已經是最多三位標註者的多數決(Chen et al., 2022, section 5),而標註者起初僅達到中等程度的一致性,經過訓練後才逐漸趨於一致。Cohen's kappa 會扣除偶然一致的部分,它從前兩個月的 0.467 上升到第三個月的 0.664,再到第四個月的 0.826(Table 2)。第 4 章會計算 kappa。

WANDS 的檔案中,店家的部分配對出現不只一次,每一列都是一次多數決,其中少數配對的投票結果並不相同。店家的樣本每個配對只保留一列,評分方式是完全符合為 2、部分符合為 1、不相關為 0;若各列不一致,則保留較低的評分(載入店家的資料)。以下是各列不一致的配對,以及樣本所保留的評分:

conflicting-judgments.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 通過
"""Lists the pairs WANDS labelled two ways, with the label the sample kept, and counts the conflicts by kind. Theprose's point rests on the `conflicts between neighbouring grades` line."""import pandas as pdfrom lib.store import DATA, load_labels, load_products, load_queriesGRADES = {"exact": 2, "partial": 1, "irrelevant": 0}products, queries, labels = load_products(), load_queries(), load_labels()conflicts = pd.read_csv(DATA / "label_conflicts.csv")       # pairs WANDS lists more than once, with different labels# the sample's kept label for each pair, which is the lower grade wherever WANDS's rows differkept = labels.set_index(["query_id", "product_id"])["label"]for row in conflicts.itertuples():    print(f"{queries.at[row.query_id, 'query']:<32} {products.at[row.product_id, 'name'][:40]:<41} "          f"{row.labels:<19} kept: {kept[(row.query_id, row.product_id)]}")# each conflicting pair's two labels, from the column where "|" separates thempairs = conflicts["labels"].str.split("|")# a spread of 1 means neighbouring grades: exact and partial, or partial and irrelevantspread = pairs.apply(lambda both: max(GRADES[label] for label in both) - min(GRADES[label] for label in both))print()print("pairs with conflicting labels:", len(conflicts))# frozensets ignore which of the two labels comes firstprint("exact against partial:", (pairs.apply(frozenset) == frozenset(["exact", "partial"])).sum())print("partial against irrelevant:", (pairs.apply(frozenset) == frozenset(["partial", "irrelevant"])).sum())print(f"conflicts between neighbouring grades: {(spread == 1).sum()} of {len(conflicts)}")
輸出python conflicting-judgments.py
whimsical solar lights           solar powered led pathway light pack      exact|partial       kept: partial
indoor multi tiered plant stand  taranto multi-tiered plant stand          exact|partial       kept: partial
clear shower curtain             shower curtain hooks                      irrelevant|partial  kept: irrelevant
breakfast bar table              counter height dining table               exact|partial       kept: partial
wood floor protector mat         hard floor straight rectangular chair ma  exact|partial       kept: partial
malachi sled                     sled coffee table                         irrelevant|partial  kept: irrelevant
kohler purist brushed bronze     purist widespread bathroom faucet with d  exact|partial       kept: partial

pairs with conflicting labels: 7
exact against partial: 5
partial against irrelevant: 2
conflicts between neighbouring grades: 7 of 7

在已評判的配對中,有 7 存在衝突:其中 5 是完全符合對部分符合,2 是部分符合對不相關。所有衝突都發生在相鄰的評分之間,即 7 of 7,因此弱點在於兩個評分之間的界線,而不是完全符合與不相關之間的差距。breakfast bar table 這一列就是其中一例:檯面高度的餐桌是否完全符合該查詢,是關於修飾語的合理疑問,並不表示某一票誤解了任務。

**實用原則:**每個配對只保留一個評分,並寫下決定該評分的規則。我們建議採用較低的評分,如同樣本的做法:這樣的錯誤是偏向不利於排序該商品的排序器,而不是偏袒它。當評審者的意見相差兩個評分,也就是完全符合對不相關時,應與他們一起重讀評判準則,因為他們對定義並沒有共識。

此部分尚未翻譯,因此以英文呈現。

最近一次變動於第 2026.11.10 版

Every judgment starts from a reading of what the query asks for. A shopper's intent is what they want to do. A widely cited taxonomy of web search sorts intent into navigational (reach a particular site), informational (find information) and transactional (do something on the web), and its author makes no assumption that intent can be inferred from the query with any certainty (Broder, 2002). Product search reshapes those kinds. Tsagkias et al. describe navigational product queries as ones that name a serial number or a product, and informational ones as usually mapping to a leaf in the inventory taxonomy or asking about specific attributes. They add that "there is no single taxonomy for eCommerce customer intent", because it depends on the store's business and inventory (Tsagkias et al., 2020, section 2).

WANDS writes intent into its labels. A query names a target entity, the kind of product, and the query's modifiers are the attributes the shopper asked for. Exact satisfies both. Partial matches only the target entity and fails the modifiers (Chen et al., 2022, section 3.2). The guidelines' worked example is a query for a kids' chair: a product that is a chair but not for kids is partial (guidelines). Even with that rule, a broad query can be hard to judge. The authors' own example is a shopper who wants an induction cooktop and types cooktop (section 1).

The store's queries cover the range. ines shag rug w003096809 and kohler whitehaven farmhouse kitchen sink name a product line or a model, which makes them navigational. outdoor light fixtures names a category. clear shower curtain and 48 inch kitchen hood add a modifier to an entity. The store's queries also carry query_class, WANDS's predicted class for each query (Table 1). It is a guess at the entity, says nothing about modifiers, and isn't a judgment. Check how often it agrees with the products the judges called exact:

predicted-class-check.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 通過
"""For each query, measures how many of its exact products carry the predicted class, and lists the queries where nonedoes. The prose's point rests on the `queries where most exact products have the query_class` line and the`queries where no exact product has the query_class` line."""import pandas as pdfrom lib.store import load_labels, load_products, load_queriesproducts, queries, labels = load_products(), load_queries(), load_labels()exact = labels[labels["label"] == "exact"].join(products["product_class"], on="product_id")rows = []for query_id, group in exact.groupby("query_id"):    predicted = queries.at[query_id, "query_class"]    classes = group["product_class"].str.split("|")                 # a product can have several classes    # the commonest product_class value, which can join several classes, comes first in value_counts(); a missing class is an empty string, which prints as (none)    rows.append({"query": queries.at[query_id, "query"], "query_class": predicted,                 "share of exact products in it": classes.apply(lambda names: predicted in names).mean(),                 "commonest class of exact products": group["product_class"].value_counts().index[0] or "(none)"})check = pd.DataFrame(rows)print("queries:", len(check))print("queries without a query_class:", check["query_class"].isna().sum())# most means more than half of a query's exact products carry the predicted classprint("queries where most exact products have the query_class:",      (check["share of exact products in it"] > 0.5).sum())missed = check[(check["share of exact products in it"] == 0) & check["query_class"].notna()]print("queries where no exact product has the query_class:", len(missed))print()print(missed.drop(columns="share of exact products in it").to_string(index=False))
輸出python predicted-class-check.py
queries: 80
queries without a query_class: 1
queries where most exact products have the query_class: 63
queries where no exact product has the query_class: 6

                   query               query_class commonest class of exact products
foutains with brick look                 Wallpaper                         Fountains
  70s inspired furniture                  Wall Art                            (none)
         7qt slow cooker Crock Pots & Slow Cookers                      Slow Cookers
 sheffield home bath set          Bath Rugs & Mats                          Vanities
      luau string lights             String Lights                  Holiday Lighting
             rustic twig     Faux Plants and Trees             Wall & Accent Mirrors

The predicted class usually agrees with the exact products. For 63 of the 80 queries, most exact products carry it, and for 6 queries, none does, and 1 query has no class at all. Read each row the example prints before counting it as an error. foutains with brick look is classed as Wallpaper, while its exact products are fountains, so the classifier went wrong. 7qt slow cooker is a naming mismatch instead: the query classes and the product classes use different names for the same thing.

Working rule: Read each query as an entity plus modifiers, and check that your ranker gets the entity right before you tune it for modifiers. Use query_class as a hint, for a feature or a filter you check against the judgments, and never as a label. When intent is ambiguous, as with cooktop, judge against the guidelines' reading and write that reading down.

以等級評估相關性,而非只分是或否

最近一次變動於第 2026.11.8 版

二元標籤只說明相關或不相關。分級標籤則說明相關的程度,例如 WANDS 的 exact(完全符合)、partial(部分符合)或 irrelevant(不相關)。各個結果的實用程度不同,因此最相關的結果應該被找出來並排在最前面。Järvelin and Kekäläinen 在 2002 年提出了採用分級的主張。在商店裡,差一點就符合的商品通常是還不錯的次要選擇。一張不是透明的浴簾,WANDS 將它評為 partial,因為它是對的商品類型,只是缺少查詢所要求的屬性(參見〈將搜尋意圖解讀為商品與其修飾語〉),它比浴簾掛鉤更接近購物者想要的東西。二元標籤會硬在各等級之間劃出一條線。只把 exact 算作相關,差一點符合的商品就和掛鉤一樣沒有價值。若把 exact 與 partial 都算作相關,它就和透明浴簾一樣有價值。

商店的查詢 clear shower curtain 呈現了這兩種缺陷。根據該查詢的判定結果,建立三組前 10 名排名:十件 exact 商品、十件被判定為 partial 的浴簾,以及十件被判定為 irrelevant 的浴簾掛鉤。接著以 nDCG@10 為每組排名評分(參見〈為單一排名評分〉),並採用三種標註方式:以 exact 為相關的二元標籤、以 exact 或 partial 為相關的二元標籤,以及本書的等級,也就是 exact 為 2、partial 為 1、irrelevant 為 0,並將它們當作增益。增益(gain)是指特定位置上的結果,在套用位置折減之前,為分數所增加的量。

binary-labels-hide-partial.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 通過
"""Scores three rankings of one query's judged products under three labelings, to show what each labeling counts asrelevant. The prose's point rests on the `grades` lines, since only the grades separate the three rankings."""from lib.metrics import ndcg_at_kfrom lib.store import load_products, qrelsclasses = load_products()["product_class"]judgments = qrels()[246]                                     # clear shower curtain: {product_id: grade}def first_ten(grade, product_class=None):    """Ten products judged with this grade for the query, in product_id order."""    # sorted(judgments) takes product_id order as the tie-break; None for product_class means any class    return [product_id for product_id in sorted(judgments) if judgments[product_id] == grade            and product_class in (None, classes[product_id])][:10]rankings = {"clear curtains": first_ten(2),                                   # exact            "other curtains": first_ten(1, "Shower Curtains"),               # partial: not clear            "hooks": first_ten(0, "Shower Curtain Hooks and Accessories")}   # irrelevantlabelings = {"exact only": {2: 1, 1: 0, 0: 0},              # binary: relevant means exact             "exact or partial": {2: 1, 1: 1, 0: 0},        # binary: relevant means exact or partial             "grades": {2: 2, 1: 1, 0: 0}}                  # graded: the book's grades as gainsprint("exact products for this query:", sum(grade == 2 for grade in judgments.values()))# the ideal top 10 holds the ten highest grades, so counting the 2s counts the exact products in itprint("exact products in the ideal top 10:", sorted(judgments.values(), reverse=True)[:10].count(2))for labeling, gain in labelings.items():    gains = {product_id: gain[grade] for product_id, grade in judgments.items()}    for name, ranking in rankings.items():        print(f"{name}, {labeling}: {ndcg_at_k(ranking, gains, 10):.3f}")
輸出python binary-labels-hide-partial.py
exact products for this query: 12
exact products in the ideal top 10: 10
clear curtains, exact only: 1.000
other curtains, exact only: 0.000
hooks, exact only: 0.000
clear curtains, exact or partial: 1.000
other curtains, exact or partial: 1.000
hooks, exact or partial: 0.000
clear curtains, grades: 1.000
other curtains, grades: 0.500
hooks, grades: 0.000

只有分級標籤能區分這三組排名。使用僅 exact 的標籤時,其他浴簾與掛鉤得分相同,分別是 0.000 和 0.000。使用 exact 或 partial 的標籤時,其他浴簾的得分為 1.000,與透明浴簾的 1.000 相同。使用等級時,透明浴簾得分為 1.000,其他浴簾為 0.500,掛鉤為 0.000。當理想的前 10 名全為 exact 時(如本例,其中有 10 件 exact 商品),由十件 partial 商品組成的排名,其得分等於 partial 增益與 exact 增益的比值。

實用原則: 我們建議商品搜尋採用分級判定,讓指標能為差一點符合的結果給予部分肯定,又不會把它誤當成完全符合。即使同時回報二元指標,也請收集等級:等級隨時可以合併成是或否,反過來則做不到。當差一點符合的結果對使用者毫無價值時,請使用二元標籤,例如查詢的是型號,而且只有該型號才能滿足需求。

刻意選擇增益值

最近一次變動於第 2026.11.8 版

等級只是標籤,增益則是每個標籤在指標中的價值,而這是一項選擇。本書直接以各等級作為其增益,因此部分相關的價值是完全相關的一半。WANDS 自己的論文將完全相關、部分相關與不相關的權重分別設為 1.0、0.5 和 0(Chen et al., 2022, Table 3),比例相同,因此 nDCG 也相同。Shopping Queries Dataset,簡稱 ESCI(Reddy et al., 2022),在 13 萬個不重複查詢上有 260 萬筆人工標註的查詢與商品判斷,其分級方式不同。它將替代品(substitute,可作為功能性替代的商品,例如以刷毛衣物替代毛衣查詢)與互補品(complement,可與完全相關商品搭配使用的商品)區分開來。其增益為:完全相關 1.0、替代品 0.1、互補品 0.01、不相關 0.0(Reddy et al., 2022, section 3.1)。WANDS 的部分相關與 ESCI 的替代品並不相同,但它是最接近的對應。

請以下列各種方案,為第 1 章中商店的兩組測試執行評分,也就是 BM25 與語意檢索對每個測試查詢所產生的排名。這些方案包括:以分級取代是非判斷中的兩種二元標註、本書的分級,以及仿照 ESCI 精神、部分相關增益為 0.1 的方案。範例也會針對各方案下語意檢索的領先幅度,執行第 3 章的排列檢定與成對 t 檢定,並印出兩個 p 值。與本書所有檢定一樣,這兩項領先幅度都屬於描述性結果。語意檢索的執行讀取開放模型 Snowflake/snowflake-arctic-embed-m-v1.5 已記錄的嵌入向量(以語意搜尋)。

gain-choice-changes-ndcg.py Python 3.12.15 · bm25s 0.3.16 · numpy 2.5.5 · pandas 3.1.1 · scipy 1.18.1 通過
"""Scores BM25 and semantic retrieval on the test queries under four gain schemes, with exploratory paired tests onsemantic retrieval's lead. The prose's point rests on the `semantic minus bm25` lines, which keep one sign across theschemes while their size changes."""from lib.lexical import LexicalSearchfrom lib.metrics import evaluatefrom lib.semantic import SemanticSearchfrom lib.significance import comparefrom lib.store import load_products, load_queries, qrelsqueries, judgments = load_queries("test"), qrels("test")# each run keeps the top 100 products per query; the scores below read only the top 10 of eachruns = {"bm25": LexicalSearch(load_products()).run(queries, k=100), "semantic": SemanticSearch().run(queries, k=100)}for name, run in runs.items():    # an unjudged product gives None here, which the count of 1 does not include    grades = [judgments[query_id].get(product_id) for query_id, ranking in run.items() for product_id in ranking[:10]]    print(f"partial share of the {name} top 10: {grades.count(1) / len(grades):.3f}")# each scheme turns a grade into the gain it is worthschemes = {"exact only": {2: 1, 1: 0, 0: 0},           "exact or partial": {2: 1, 1: 1, 0: 0},           "grades 2, 1, 0": {2: 2, 1: 1, 0: 0},       # the book's choice; WANDS's 1, 0.5, 0 gives the same nDCG           "partial gain 0.1": {2: 1, 1: 0.1, 0: 0}}   # close to the gains ESCI chose for its substitutesleads = 0for scheme, gain in schemes.items():    gains = {query_id: {product_id: gain[grade] for product_id, grade in per_query.items()}             for query_id, per_query in judgments.items()}    scores = {name: evaluate(run, gains)["ndcg@10"] for name, run in runs.items()}    test = compare(scores["semantic"], scores["bm25"])  # exploratory paired tests (chapter 3): no planned comparisons    print(f"bm25, {scheme}: {scores['bm25'].mean():.3f}")    print(f"semantic, {scheme}: {scores['semantic'].mean():.3f}")    print(f"semantic minus bm25, {scheme}: {test['mean difference']:.3f}")    print(f"permutation p, {scheme}: {test['permutation p']:.3f}")    print(f"t-test p, {scheme}: {test['t-test p']:.3f}")    print(f"tied queries, {scheme}: {test['ties']}")    # a lead is a positive mean difference, semantic minus BM25    leads += test["mean difference"] > 0print(f"schemes where semantic leads: {leads} of {len(schemes)}")
輸出python gain-choice-changes-ndcg.py
partial share of the bm25 top 10: 0.403
partial share of the semantic top 10: 0.380
bm25, exact only: 0.519
semantic, exact only: 0.648
semantic minus bm25, exact only: 0.130
permutation p, exact only: 0.000
t-test p, exact only: 0.002
tied queries, exact only: 3
bm25, exact or partial: 0.903
semantic, exact or partial: 0.983
semantic minus bm25, exact or partial: 0.080
permutation p, exact or partial: 0.010
t-test p, exact or partial: 0.026
tied queries, exact or partial: 19
bm25, grades 2, 1, 0: 0.725
semantic, grades 2, 1, 0: 0.830
semantic minus bm25, grades 2, 1, 0: 0.105
permutation p, grades 2, 1, 0: 0.001
t-test p, grades 2, 1, 0: 0.003
tied queries, grades 2, 1, 0: 2
bm25, partial gain 0.1: 0.564
semantic, partial gain 0.1: 0.688
semantic minus bm25, partial gain 0.1: 0.125
permutation p, partial gain 0.1: 0.000
t-test p, partial gain 0.1: 0.002
tied queries, partial gain 0.1: 2
schemes where semantic leads: 4 of 4

語意檢索在4 of 4種方案下領先,但領先幅度的大小取決於所作的選擇。部分相關的商品占 BM25 前 10 名位置的 0.403,占語意檢索前 10 名位置的 0.380,因此部分相關增益會大幅移動兩者的分數。BM25 在僅計完全相關的標註下得分為 0.519,在完全或部分相關的標註下為 0.903。在類似 ESCI、部分相關增益為 0.1 的情況下,BM25 得分為 0.564,語意檢索為 0.688。領先幅度在僅計完全相關的標註下為 0.130,在類似 ESCI 的增益下為 0.125,在本書的分級下為 0.105,在完全或部分相關的標註下為 0.080。範例並未檢定這些領先幅度之間是否有差異,因此在 30 個測試查詢的情況下,應將它們之間的差異視為可能只是雜訊。

完全或部分相關的標註也會出現飽和,使得能區分排序器的查詢變少。在這種標註下有 19 個測試查詢持平,在本書的分級下則為 2 個。

實用原則: 商品搜尋預設使用線性增益,也就是 2、1、0 的分級。當差一點就符合的商品對你的購物者價值不高時,請選擇較低的部分相關增益。無論選擇為何,都要在比較排序器之前定案,隨每個分數一併報告,且絕不要比較以不同增益計算的 nDCG。

檢視每個查詢的標籤組成

最近一次變動於第 2026.11.8 版

增益決定了每個等級的價值,卻沒有說明一個查詢中各等級各有多少。判斷集並不是單一的母體:每個查詢都有自己的一組已判斷商品,也有自己的等級組成,而這個組成決定了該查詢的完美排序是什麼樣子。廣泛的查詢,也就是指名某個許多商品都符合的類別的查詢,會有很多完全符合的商品。具體的查詢,也就是帶有數個修飾語的查詢,可能只有少數幾個。對查詢取平均會把這些不同的問題混在一起,所以在做任何平均之前,先檢視標籤組成。

商店中的查詢並不是從 WANDS 隨機抽出的。每個查詢都被選為擁有 60 到 300 筆已判斷商品、至少 3 筆完全符合與 20 筆部分符合的判斷,而且其預測類別不與其他任何被選中的查詢相同(資料卡)。這些計數描述的是樣本,而不是整個 WANDS,因為超出限制的查詢不會進入商店。請計算 80 個查詢各自的標籤數,並依完全符合商品的比例將查詢分組:

count-labels-per-query.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 通過
"""Counts each query's judgments by label, and groups the queries by their share of exact products. The prose's pointrests on the `queries with an exact share of 0.1 or less` line and on the histogram below it."""import pandas as pdfrom lib.store import load_labels, load_queriesqueries, labels = load_queries(), load_labels()counts = pd.crosstab(labels["query_id"], labels["label"])[["exact", "partial", "irrelevant"]]shares = counts.div(counts.sum(axis=1), axis=0).join(queries[["query", "split"]])print(f"partial share of all judgments: {(labels['label'] == 'partial').mean():.3f}")print(f"median exact share: {shares['exact'].median():.3f}")print(f"median irrelevant share: {shares['irrelevant'].median():.3f}")for label, which in [("exact", "lowest"), ("exact", "highest"), ("irrelevant", "highest")]:    query_id = shares[label].idxmin() if which == "lowest" else shares[label].idxmax()    print(f"{which} {label} share: {shares.at[query_id, label]:.3f}")    print(f"query with the {which} {label} share: {shares.at[query_id, 'query']}")print("queries with no irrelevant judgment:", (counts["irrelevant"] == 0).sum())print("fewest exact products for a query:", counts["exact"].min())print("most exact products for a query:", counts["exact"].max())# 10 is the size of a top 10: a query with fewer exact products can't fill a perfect top 10 with exact onesfew = counts["exact"] < 10print("queries with fewer than 10 exact products:", few.sum())print("test queries with fewer than 10 exact products:", (few & (shares["split"] == "test")).sum())print("queries with an exact share of 0.1 or less:", (shares["exact"] <= 0.1).sum())print()print("exact share  queries")bins = pd.cut(shares["exact"], [tenth / 10 for tenth in range(11)])     # no share is 0: each query has 3 or more exactfor interval, number in bins.value_counts(sort=False).items():    print(f"{interval.left:.1f} to {interval.right:.1f}  {number:>3}  {'#' * number}".rstrip())
輸出python count-labels-per-query.py
partial share of all judgments: 0.610
median exact share: 0.207
median irrelevant share: 0.060
lowest exact share: 0.014
query with the lowest exact share: 48 in entry table with side by side drawer
highest exact share: 0.749
query with the highest exact share: outdoor light fixtures
highest irrelevant share: 0.776
query with the highest irrelevant share: podium with locking cabinet
queries with no irrelevant judgment: 9
fewest exact products for a query: 3
most exact products for a query: 193
queries with fewer than 10 exact products: 23
test queries with fewer than 10 exact products: 7
queries with an exact share of 0.1 or less: 29

exact share  queries
0.0 to 0.1   29  #############################
0.1 to 0.2   10  ##########
0.2 to 0.3   12  ############
0.3 to 0.4   11  ###########
0.4 to 0.5    9  #########
0.5 to 0.6    1  #
0.6 to 0.7    7  #######
0.7 to 0.8    1  #
0.8 to 0.9    0
0.9 to 1.0    0

部分符合是最常見的標籤,占所有判斷的 0.610。可能的原因在於 WANDS 蒐集候選商品的方式,這部分在了解判斷從何而來中說明。完全符合的比例在各查詢之間差異很大,從 48 in entry table with side by side drawer 的 0.014,到 outdoor light fixtures 的 0.749,中位數為 0.207。範例印出的直方圖在低端最高:有 29 個查詢的完全符合比例在 0.1 以下。

多數查詢的不相關判斷很少,但並非全部如此:不相關比例的中位數為 0.060,然而 podium with locking cabinet 達到 0.776,並且有 9 個查詢完全沒有不相關的判斷。

這樣的分布改變了 nDCG@10 所衡量的內容。每個查詢的完全符合商品數從 3(抽樣規則的下限)到 193,有 23 個查詢的完全符合商品少於 10 個,其中 7 個是測試查詢。對這些查詢而言,理想的前 10 名包含部分符合的商品,因此排序可以在前 10 名含有部分符合商品的情況下得到滿分。對於 outdoor light fixtures 這類廣泛的查詢,要得到滿分則需要十個完全符合的商品。相同的 nDCG@10 對兩者的意義並不相同。

實用原則: 在對任何指標取平均之前,先印出每個查詢的標籤組成,並且兩端都要看。完全符合商品很少的查詢得到低分,與廣泛查詢得到低分,應視為不同的問題。每個平均值旁都要附上各查詢的分數;第 3 章說明了平均值會隱藏什麼。

將點擊視為有偏誤的證據

最近一次變動於第 2026.11.10 版

商店會記錄每一次點擊,而且蒐集點擊不需成本,因此點擊看起來像是免費的相關性判斷。事實並非如此。點擊只說明購物者選擇查看某項商品,並不說明該商品滿足了需求,而購物者是否點擊,取決於商品在何處、以何種方式呈現。

點擊如何誤導的證據來自網頁搜尋。Joachims 等人追蹤康乃爾大學大學生使用 Google 回答十個既定問題時的視線與點擊,分兩個階段進行,第一階段有 34 位學生,第二階段有 22 位。在第二階段中,部分學生看到前兩筆結果對調,或整份清單反向排列(Joachims et al., 2005)。他們的結論是,點擊「具有資訊性,但有偏誤」。

第一種偏誤是位置偏誤(position bias):人們較常點擊排名較高的結果,部分原因是信任排名。Joachims 等人將這種效應稱為「信任偏誤」,即使使用者看過的其他摘要更相關,排名靠前的連結仍會吸引點擊。作者也發現,相對偏好,例如被點擊的結果優於其上方被略過的結果,平均而言相當準確。

在一個大型網頁搜尋引擎的實驗中,Craswell 等人發現,串接模型(cascade model)最能解釋前幾名的位置偏誤。在串接模型中,使用者由上往下閱讀結果,並在第一份值得閱讀的文件處離開(Craswell et al., 2008)。

呈現偏誤(presentation bias)不只涉及位置:點擊取決於結果如何呈現,而不只是出現在哪裡。在一部分 Google 網頁搜尋流量上,標題更具吸引力的結果,也就是查詢字詞以粗體顯示得更多的結果,即使在控制位置與人工判斷的相關性之後,仍獲得較多點擊(Yue et al., 2010)。在商店中,圖片與價格最接近標題的對應物。這是我們的推論,並非實測結果。

WANDS 沒有點擊資料,因此商店無法測量這些偏誤。不過,你仍可以看出點擊紀錄會記下什麼。取商店對 clear shower curtain 的判斷,並以最嚴格的形式套用串接模型:購物者從頂端開始閱讀,點擊第一個完全符合的商品後離開。這個思想實驗使用一條固定規則,並非真實購物者的模型。第 11 章會以機率模擬點擊。請對 BM25 的排名與語意檢索的排名各執行一次這個思想實驗,並排閱讀兩份點擊紀錄:

cascade-clicks.py Python 3.12.15 · bm25s 0.3.16 · numpy 2.5.5 · pandas 3.1.1 · scipy 1.18.1 通過
"""Reads the top 10 of BM25 and of semantic retrieval for `clear shower curtain` as a cascade of clicks, which stops atthe first exact product. The prose's point rests on the `clicked rank` lines and on the `exact products shown below theclick` lines."""from lib.lexical import LexicalSearchfrom lib.semantic import SemanticSearchfrom lib.store import load_products, qrelsproducts = load_products()judgments = qrels()[246]                                     # clear shower curtain: {product_id: grade}# search returns scores indexed by product_id, so .index is the ranking; semantic search takes the query's id, 246rankings = {"bm25": list(LexicalSearch(products).search("clear shower curtain", k=10).index),            "semantic": list(SemanticSearch().search(246, k=10).index)}for name, ranking in rankings.items():    grades = [judgments.get(product_id, 0) for product_id in ranking]     # an unjudged product counts as grade 0    rank = grades.index(2) + 1             # cascade: read from the top, click the first exact product, stop    print(f"{name} grades, ranks 1 to 10: {' '.join(map(str, grades))}")    print(f"{name} clicked rank: {rank}")    # rank counts from 1 and the list from 0, so the clicked product sits at index rank - 1    print(f"{name} clicked product: {products.at[ranking[rank - 1], 'name']}")    # grades[rank:] starts just after the click, since rank is the clicked product's position counted from 1    print(f"{name} exact products shown below the click: {grades[rank:].count(2)}")
輸出python cascade-clicks.py
bm25 grades, ranks 1 to 10: 1 2 2 2 2 1 1 0 0 0
bm25 clicked rank: 2
bm25 clicked product: woolley vinyl single shower curtain
bm25 exact products shown below the click: 3
semantic grades, ranks 1 to 10: 2 2 1 1 2 1 2 2 1 1
semantic clicked rank: 1
semantic clicked product: woolley vinyl solid color single shower curtain
semantic exact products shown below the click: 4

點擊紀錄記下的,和購物者想要的一樣多,也記下排序器最先呈現的。在 BM25 的排名下,購物者點擊第 2 名的 woolley vinyl single shower curtain,從未到達其下方的 3 個完全符合的商品。在點擊紀錄中,這些完全符合的商品看起來,與 BM25 排名第 8 到 10 名、被判定為不相關的浴簾掛鉤(以 BM25 對商品排序)沒有差別,因為同樣沒有人點擊。

在語意檢索的排名下,點擊落在第 1 名的 woolley vinyl solid color single shower curtain。同一個商品目錄產生的兩種排名,會得出兩份歸功於不同商品的點擊紀錄。

WANDS 的作者以同樣的原因,也就是位置偏誤,說明為何不以點擊紀錄建構資料集(對照判斷檢查結果)。不過,點擊仍帶有資訊。在 WANDS 中,取自點擊日誌的候選項目有 62.47% 被判定為完全符合,而取自開源排序系統的候選項目只有 30.95%(Chen et al., 2022, Table 4)。

修正點擊的偏誤本身就是一個領域,稱為無偏誤學習排序(unbiased learning to rank)。在 Baidu 發布的大型搜尋資料集上,這些方法改善了點擊預測,但就專家的判斷而言,仍「難以穩定地改善排序表現」(Hager et al., 2024)。

實用原則: 離線評估應以判斷而非點擊作為參考標準。將點擊當作相對證據使用,即被點擊的商品優於排名在其上方、被略過的商品,且絕不可將沒有點擊解讀為商品不相關。若以點擊訓練模型,請對照判斷檢查結果。

了解判定的來源

最近一次變動於第 2026.11.10 版

沒有人會針對每個查詢判定每一項商品,因此測試集合改為判定一個候選池(pool),也就是為每個查詢蒐集而來的一組候選項目。在 TREC(Text REtrieval Conference)的共同評測中,某個主題的候選池是各參與者所檢索出排名最前面文件的聯集,通常取前 100 筆,而候選池之外的每份文件都視為不相關。Voorhees 主張,因為候選池涵蓋了足夠多的相關文件,所以即使把每份未判定文件都視為不相關,系統之間仍可公平比較。同一篇文章也指出,對於任意大的文件集合,候選池法無法產生可重複使用的測試集合(Voorhees, 2007)。Zobel 檢驗了這項公平性的主張。使用候選池判定時,系統之間的相對表現看起來可靠,但召回率會被高估,因為許多相關文件很可能從未被找到(Zobel, 1998)。

WANDS 分兩個步驟建立其候選池(Chen et al., 2022, sections 4.2 and 4.3)。第一步取用購物者點擊過或加入購物車的商品,再加上開源詞彙搜尋引擎 Solr 與一套神經檢索系統的結果。第二步用同樣的兩套系統再次探勘整個商品目錄,並加入查詢分類模型為該查詢預測的類別中的每一項商品。第二步產生了 WANDS 的大部分標籤,而且多半是接近但未命中的項目:其標籤中有 4.12% 為完全符合,63.99% 為部分符合(Table 5)。在本店的樣本中,部分符合同樣是最常見的標籤(讀懂每個查詢的標籤組成)。作者主張,完整性對商品搜尋很重要,因為「incompleteness in Product Search dataset does negatively impact the discriminative power of the evaluation」(section 3.1)。

候選池是屬於查詢的。某個查詢下沒有判定的商品,對該查詢而言是未判定(unjudged),而對某個查詢已判定的商品,對相近的另一個查詢可能是未判定。本店的兩個查詢可以說明這點:訓練查詢 trundle daybed 與測試查詢 full bed with trundle。請計算對前者判定為完全符合、但對後者沒有判定的商品數量,並查看評審如何為有判定的那些商品評分:

trundle-daybeds-unjudged.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 通過
"""Counts the products judged exact for the train query `trundle daybed` that have no judgment for the test query `fullbed with trundle`, and grades the ones that do. The prose's point rests on the `of them unjudged for full bed withtrundle` line and the `judged partial` line."""from collections import Counterfrom lib.store import load_products, load_queries, qrelsproducts, queries, judgments = load_products(), load_queries(), qrels()LABELS = {2: "exact", 1: "partial", 0: "irrelevant"}# 433 is the train query trundle daybed, and its exact products come out in product_id orderdaybeds = sorted(product_id for product_id, grade in judgments[433].items() if grade == 2)full_bed = judgments[436]                                    # {product_id: grade} for "full bed with trundle"print(f"433: {queries.at[433, 'query']} ({queries.at[433, 'split']})")print(f"436: {queries.at[436, 'query']} ({queries.at[436, 'split']})")print("exact for trundle daybed:", len(daybeds))unjudged = [product_id for product_id in daybeds if product_id not in full_bed]print("of them unjudged for full bed with trundle:", len(unjudged))print("of them judged for full bed with trundle:", len(daybeds) - len(unjudged))judged = Counter(LABELS[full_bed[product_id]] for product_id in daybeds if product_id in full_bed)for label in LABELS.values():    print(f"judged {label}: {judged[label]}")# daybeds is in product_id order, so the first two unjudged are the two lowest product_ids among themprint("first unjudged daybed:", products.at[unjudged[0], "name"])print("second unjudged daybed:", products.at[unjudged[1], "name"])
輸出python trundle-daybeds-unjudged.py
433: trundle daybed (train)
436: full bed with trundle (test)
exact for trundle daybed: 113
of them unjudged for full bed with trundle: 77
of them judged for full bed with trundle: 36
judged exact: 5
judged partial: 31
judged irrelevant: 0
first unjudged daybed: daybed with trundle
second unjudged daybed: pattonsburg daybed with trundle

把未判定商品計為 0 分的指標,等於是在猜測它,而在這裡這個猜測相當重要。依我們對指引規則的解讀,附拉床的沙發床往往是種類正確、但尺寸不符合查詢要求的床。對 trundle daybed 判定為完全符合的 113 項商品中,有 77 項對 full bed with trundle 沒有判定,其中包括商品編號最小的兩項,daybed with trundle 與 pattonsburg daybed with trundle。至於有判定的 36 項,評審判為完全符合的有 5 項,部分符合的有 31 項,不相關的有 0 項。如果未判定的沙發床與已判定的相似,其中大多數也會是部分符合,然而指標卻給每一項 0 分。

Zobel 警告,候選池判定可能低估未對候選池做出貢獻的系統(Zobel, 1998)。這使得判定對與建立候選池的系統相似的排序器最為公平:以 WANDS 而言,就是購物者的點擊、詞彙引擎、神經檢索器與查詢分類器。這是一種傾向,而不是定律。在此,與 WANDS 詞彙引擎最相似的排序器 BM25,其 judged@10 在兩個排序器中反而較低(計算未判定商品)。我們的推測是,本店的 BM25 讀取的欄位與候選池所用的引擎不同,因此排在前面的商品也不同。

實用原則: 在信任比較結果之前,先弄清楚你的候選池從何而來。當你評估的排序器與建立候選池的系統不相似時,預期它會呈現無人判定過的商品,並規劃去判定這些商品。

此部分尚未翻譯,因此以英文呈現。

Count unjudged products before you compare scores

最近一次變動於第 2026.11.10 版

A product nobody judged for a query is unjudged: its grade is unknown, not 0. Standard metrics can't tell the difference. They score an unjudged product exactly as they score one judged irrelevant, and Buckley and Voorhees warn that such metrics become unreliable when judgments are very incomplete (Buckley and Voorhees, 2004). Two diagnostics are standard in evaluation tools (ir-measures). Judged@k is the share of the top k that has a judgment of any grade:

judged@k = (number of the top k products that have a judgment for the query) / k

The second diagnostic is the condensed list, a ranking with its unjudged products dropped before scoring, so the judged ones move up. ir-measures offers it as the judged_only option. Buckley and Voorhees's bpref measure works in the same spirit: it uses judged documents only, and they found it more robust to incomplete judgments.

In chapter 1, BM25's mean judged@10 on the test queries is 0.927. The mean for semantic retrieval is 0.990. One property of the store's data keeps those numbers high. Its catalog is almost all judged: 11,917 of 12,000 products have a judgment for at least one query (Load the store's data). Expect a full catalog to put more unjudged products in the top 10s.

Build judged@10 by hand, check it against lib.metrics, and list where BM25's unjudged products are:

unjudged-in-bm25-top-10.py Python 3.12.15 · bm25s 0.3.16 · numpy 2.5.5 · pandas 3.1.1 · scipy 1.18.1 通過
"""Builds judged@10 by hand for BM25's test rankings, checks it against lib.metrics, and lists where BM25's unjudgedproducts sit in the top 10s. The prose's point rests on the `unjudged products in the top 10s` line and on the`irrelevant judgments` line for pull out sleeper loveseat."""import pandas as pdfrom lib.lexical import LexicalSearchfrom lib.metrics import evaluatefrom lib.store import load_products, load_queries, qrelsproducts, queries, judgments = load_products(), load_queries("test"), qrels("test")# 100 products per query are kept; this example reads only the top 10 of eachrun = LexicalSearch(products).run(queries, k=100)def judged_at_k(ranking, judged, k):    """The share of the top k that has a judgment of any grade."""    return sum(product_id in judged for product_id in ranking[:k]) / krows = []for query_id, ranking in run.items():    unjudged = [product_id for product_id in ranking[:10] if product_id not in judgments[query_id]]    # the name is cut to 45 characters so that the table fits on the page    rows.append({"query": queries.at[query_id, "query"], "judged@10": judged_at_k(ranking, judgments[query_id], 10),                 "unjudged": len(unjudged), "first unjudged product": products.at[unjudged[0], "name"][:45] if unjudged else ""})table = pd.DataFrame(rows)print(f"mean judged@10: {table['judged@10'].mean():.3f}")# lib.metrics' judged@10 per query, indexed by query_idlibrary = evaluate(run, judgments)["judged@10"]# 1e-12 allows for floating-point rounding when the two judged@10 values are comparedagree = sum(abs(row["judged@10"] - library[query_id]) < 1e-12 for query_id, row in zip(run, rows))print(f"queries where the hand-built judged@10 equals lib.metrics: {agree} of {len(rows)}")print("unjudged products in the top 10s:", table["unjudged"].sum())print("test queries with an unjudged product in the top 10:", (table["judged@10"] < 1).sum())sleeper = list(judgments[244].values())                      # pull out sleeper loveseatprint("judgments for pull out sleeper loveseat:", len(sleeper))print("irrelevant judgments for pull out sleeper loveseat:", sleeper.count(0))  # grade 0 is irrelevantprint()# judged@10 below 1 means the query has at least one unjudged product in its top 10print(table[table["judged@10"] < 1].sort_values("judged@10").to_string(index=False))
輸出python unjudged-in-bm25-top-10.py
mean judged@10: 0.927
queries where the hand-built judged@10 equals lib.metrics: 30 of 30
unjudged products in the top 10s: 22
test queries with an unjudged product in the top 10: 7
judgments for pull out sleeper loveseat: 143
irrelevant judgments for pull out sleeper loveseat: 0

                    query  judged@10  unjudged                        first unjudged product
   outdoor light fixtures        0.2         8  60 watt equivalent e12 led candle light bulb
     beach blue headboard        0.5         5 lemon frosting indoor/outdoor , removable cov
pull out sleeper loveseat        0.6         4    country oxford 87.75 '' rolled arm sleeper
    full bed with trundle        0.8         2 abel staircase twin over full bunk bed with t
   whimsical solar lights        0.9         1 luciole 0.12-watt led solar power dusk to daw
  bistro table and chairs        0.9         1              drayton patio 5 piece bistro set
     wooden chair outdoor        0.9         1 osterley 31.5 '' w writing desk and chair set

The hand-built judged@10 equals lib.metrics on 30 of 30 test queries, with a mean of 0.927. Unjudged products fill 22 of BM25's top-10 places, spread over 7 queries. By our reading, some are plainly irrelevant: light bulbs for outdoor light fixtures, an outdoor pillow for beach blue headboard. Others look like near misses or matches: a bunk bed with a trundle for full bed with trundle, a bistro set for bistro table and chairs, a sleeper sofa for pull out sleeper loveseat, which by the entity-and-modifier rule is probably a near miss: the judges called 0 of that query's 143 judged products irrelevant.

Score both rankers with condensed lists, which drop the unjudged products, to see how far the scores move:

condensed-ndcg.py Python 3.12.15 · bm25s 0.3.16 · numpy 2.5.5 · pandas 3.1.1 · scipy 1.18.1 通過
"""Scores BM25 and semantic retrieval on the test queries with standard and condensed nDCG@10, and runs exploratorytests on the change in the lead. The prose's point rests on the `semantic minus bm25` lines and the `share of bm25'sgain` line."""from lib.lexical import LexicalSearchfrom lib.metrics import evaluatefrom lib.semantic import SemanticSearchfrom lib.significance import comparefrom lib.store import load_products, load_queries, qrelsqueries, judgments = load_queries("test"), qrels("test")runs = {"bm25": LexicalSearch(load_products()).run(queries, k=100), "semantic": SemanticSearch().run(queries, k=100)}standard, condensed = {}, {}for name, run in runs.items():    # condensed: drop every unjudged product, so the judged ones move up into the top 10    judged_only = {query_id: [product_id for product_id in ranking if product_id in judgments[query_id]]                   for query_id, ranking in run.items()}    standard[name] = evaluate(run, judgments)["ndcg@10"]    condensed[name] = evaluate(judged_only, judgments)["ndcg@10"]    print(f"{name} ndcg@10: {standard[name].mean():.3f}")    print(f"{name} condensed ndcg@10: {condensed[name].mean():.3f}")gain = condensed["bm25"] - standard["bm25"]                   # bm25's change, per test query# the scores are indexed by query_id: 133 is outdoor light fixtures, and 46 is beach blue headboardprint(f"bm25 ndcg@10 for outdoor light fixtures: {standard['bm25'][133]:.3f}")print(f"bm25 condensed ndcg@10 for outdoor light fixtures: {condensed['bm25'][133]:.3f}")print(f"share of bm25's gain from outdoor light fixtures and beach blue headboard: {gain[[133, 46]].sum() / gain.sum():.3f}")lead, condensed_lead = standard["semantic"] - standard["bm25"], condensed["semantic"] - condensed["bm25"]change = compare(condensed_lead, lead)                       # is the change in the lead more than noise? (exploratory)print(f"semantic minus bm25: {lead.mean():.3f}")print(f"semantic minus bm25, condensed: {condensed_lead.mean():.3f}")print(f"change in the lead: {change['mean difference']:.3f}")print(f"95% interval for the change: {change['ci low']:.3f} to {change['ci high']:.3f}")print(f"t-test p for the change: {change['t-test p']:.3f}")print(f"permutation p for the change: {change['permutation p']:.3f}")condensed_test = compare(condensed["semantic"], condensed["bm25"])   # the condensed lead itself (exploratory)print(f"t-test p for the condensed lead: {condensed_test['t-test p']:.3f}")print(f"permutation p for the condensed lead: {condensed_test['permutation p']:.3f}")
輸出python condensed-ndcg.py
bm25 ndcg@10: 0.725
bm25 condensed ndcg@10: 0.778
semantic ndcg@10: 0.830
semantic condensed ndcg@10: 0.834
bm25 ndcg@10 for outdoor light fixtures: 0.286
bm25 condensed ndcg@10 for outdoor light fixtures: 1.000
share of bm25's gain from outdoor light fixtures and beach blue headboard: 0.612
semantic minus bm25: 0.105
semantic minus bm25, condensed: 0.056
change in the lead: -0.049
95% interval for the change: -0.138 to -0.015
t-test p for the change: 0.073
permutation p for the change: 0.036
t-test p for the condensed lead: 0.011
permutation p for the condensed lead: 0.009

Condensing lifts BM25 mostly by forgiving bad results, not by undoing a penalty on good ones. BM25 goes from 0.725 to 0.778, and semantic retrieval from 0.830 to 0.834. Of BM25's gain, a share of 0.612 comes from outdoor light fixtures and beach blue headboard, the two queries whose unjudged products are plainly irrelevant. For outdoor light fixtures, BM25 goes from 0.286 to 1.000: once the bulbs drop out, the judged lights below them move up.

Semantic retrieval's lead falls from 0.105 to 0.056 once the lists are condensed. Whether that change is more than noise is unsettled. Chapter 3's paired t-test gives p = 0.073 for the change, and its permutation test gives 0.036. The 95% bootstrap interval runs from -0.138 to -0.015. Both checks are exploratory. With the change carried by a few queries, treat it as possibly noise.

The condensed lead itself gets p = 0.009 from the permutation test and 0.011 from the t-test. Neither the standard score nor the condensed one is the truth. The standard score punishes unjudged products that may be good, and the condensed score hides unjudged products that are bad.

Working rule: Report judged@10 next to every nDCG@10, and compute condensed scores as a diagnostic, never as the headline. When two rankers' judged@10 differ, have the unjudged products in their top 10s judged before you compare them. That is the dependable cure, and on this sample it is a small job: 22 judgments for BM25's test run. On a full catalog, expect more.

正式環境中會出什麼問題

最近一次變動於第 2026.11.8 版

判斷會過時。 商店的商品目錄與購物者都在持續變動,相關性也不只取決於查詢本身。Tsagkias 等人指出使用者、時間、查詢與情境都會影響相關性,並舉例說明:查詢外套時,冬季與夏季的相關商品並不相同(Tsagkias et al., 2020, section 3.1.1)。新商品進入目錄時尚未經過判斷,而已下架的商品仍留在判斷集中。請持續追蹤正式環境排序器的 judged@10,在數值下降時重新進行判斷。

點擊標籤會確認你現有的排序。 以現行排序下記錄的點擊來訓練排序器,排序器學到的除了購物者的偏好,還包括該排序本身的順序。請保留人工判斷作為檢驗點擊的基準(把點擊視為有偏誤的證據)。

標註指引會形塑判斷。 WANDS 的標註人員花了四個月才達到近乎完美的一致性(Chen et al., 2022, Table 2)。指引中的小決定會影響整個資料集。WANDS 要求標註人員忽略拼字錯誤,並依查詢的字面內容判斷範例(指引),因此其判斷預設讀者能看懂 ligth bulb。無法做到這點的排序器,在這類查詢上得分會偏低,而這對購物者來說是公平的。第 8 章會建立一個拼字修正器,並說明它仍然會出錯的地方。

你的排序器看到的比判斷者少。 判斷者看得到圖片與商品頁面,而商店的排序器讀取的是不含屬性的文字(第 5 章測試加入屬性的效果)。任何大型語言模型(LLM)若依同樣的文字來判斷,看到的就是排序器看到的內容。這類系統與人工判斷之間的部分分歧,來自這個資訊落差,而非系統本身。在責怪排序器之前,請先檢查哪些欄位含有判斷者所獎勵的屬性。

用模型填補空缺仍然需要人工判斷。 Gienapp 等人針對每個查詢,以該查詢的人工判斷微調一個小型分類器,再用它來判斷未經判斷的商品。他們在 TREC Robust04 以及 2019 與 2020 年的 TREC Deep Learning 集合上回報,每個主題只要 128 筆初始人工判斷,就能提升模型之間的可比較性,而且這種做法比以提示詞驅動的 LLM 判斷者更可靠(Gienapp et al., 2025)。Soboroff 對 LLM 的建議則更直接:「不要使用 LLM 來為 TREC 風格的評估建立相關性判斷」(Soboroff, 2025)。第 4 章會先拿 LLM 的判斷與商店的人工判斷比較,再決定是否信任。

**實用原則:**為判斷標註日期,在正式環境中追蹤 judged@10,並為重新判斷編列預算。

本章的決定

最近一次變動於第 2026.11.8 版

除非訓練查詢上的測量結果另有說明,後續各章都沿用這些決定。

  • 以人工判斷作為基準。 本書所有主要分數都以人工判斷計算,包括以 LLM 建構的排序器的分數。點擊是關於購物者的證據,會受到位置與呈現方式的偏誤影響,因此絕不當作標籤使用。
  • 等級 2、1 和 0 就是增益。 因此部分相關的價值是完全相關的一半,與 WANDS 自己採用的 1.0、0.5 和 0 比例相同。二元指標只計入完全相關。
  • 每個配對只有一個等級。 當 WANDS 對同一個配對列出多次且標籤不同時,樣本保留較低的等級,這樣的取捨對排序器較不利。
  • query_class 只是提示, 絕不是標籤,也不是意圖的衡量方式。它的部分值是錯的。
  • 未判斷的視為等級 0,且每個 nDCG@10 旁都附上 judged@10。 精簡後的分數僅作為診斷之用。前 10 名中出現未判斷商品時,解決辦法是讓這些商品接受判斷。
  • 排序器讀取商品文字,而不是屬性。 第 5 章會測試將屬性作為一個欄位加入,並讓屬性不進入 BM25 文字。判斷者也看過圖片與商品頁面,因此對只使用文字的排序器而言,有些完全相關的商品幾乎沒有可供比對的內容。
  • 可以閱讀任何判斷,但只在訓練集上調校。 閱讀測試集的判斷不會改變任何排序器;依據這些判斷來選擇修正方式才會。
  • 本章呈現的結果: 分數的好壞取決於判斷的品質。

本章,於本版次中

版次 2026.11.10範例於 2026年11 月 19 通過測試本章 10 個範例 個範例中有 10 個通過

譯自英文書,版次 2026.11.10。

引用本節

格式

各種格式皆引用英文原書,本譯本即依循該原書。

每種格式都會標示本版。其連結會開啟目前版次,節標題會說明該節內文最後一次變動的時間。存取日期與擷取日期即為你開啟此頁的當日。

本頁內容

第 2 章,共 11 章: 相關性的意義

  1. Judge relevance against the shopper's need
  2. 預期評審者會在部分配對上意見不一
  3. Read intent as a product and its modifiers
  4. 以等級評估相關性,而非只分是或否
  5. 刻意選擇增益值
  6. 檢視每個查詢的標籤組成
  7. 將點擊視為有偏誤的證據
  8. 了解判定的來源
  9. Count unjudged products before you compare scores
  10. 正式環境中會出什麼問題
  11. 本章的決定

返回頂端所有章節