相关性的含义
阅读约需 78分钟 分钟,11节中含10个示例。
读完本章,你应当能够批判性地阅读一组相关性判断:说明每个等级对商品作出了怎样的断言,找出这些判断的来源及其缺漏之处,并决定应在多大程度上信任由它们算出的分数。
到目前为止,每一个分数都把商店的判断当作事实。第 1 章用 13,077 条判断为两种排序器打分,这些判断是人们给查询与商品所构成的配对打出的等级,其中查询是在商店中输入的,商品是商店所售的。在测试查询上,BM25的平均 nDCG@10 为 0.725。语义检索的得分为 0.830。这些数字都建立在判断所默默作出的若干选择之上:它们记录的是谁心目中的相关性,差一点就命中值多少,以及无人判断过的商品会怎样处理。
本章细读商店的判断。本章不改动任何排序器,因此对训练查询和测试查询一并阅读:阅读判断不是调优,本书中的每一项改进仍然只依据训练查询来选择。本章在测试查询上运行的少数检验属于探索性检验,因此它们只用于描述这些判断,并不能确立任何差异(见本书的规则)。第 3 章会把 BM25 与语义检索之间的差距作为一项预先计划的比较来检验。
这部分尚未翻译,因此以英文显示。
Judge relevance against the shopper's need
A search system returns products that match a query in the way the system defines a match: shared words for lexical retrieval, nearby vectors for semantic retrieval. Information science calls that system relevance. It separates that from topical relevance, whether a product is about the query's subject, and situational relevance, whether it helps with the task at hand (Saracevic, 2007). Some argue that system relevance is always perfect, because the system retrieved what the query asked for. Saracevic's answer is "Not so."
A judgment is a verdict, and this chapter asks what each grade claims. The store's judgments come from WANDS, a public product-search dataset, through a sample of its queries and products (Load the store's data). WANDS's judges label each pair exact, partial or irrelevant, which the book grades 2, 1 and 0. Exact means the product is the kind of thing the query asks for and has every attribute the query names. Partial means it is the right kind of thing but misses an attribute. Irrelevant means it isn't the kind of thing asked for (Read intent as a product and its modifiers). The judge grades the product against the information need behind the query's words, the thing the shopper wanted, not against the words. Manning et al. define a document as relevant when it addresses the stated information need, not merely because it contains the query's words (Manning et al., 2008, chapter 8). Evaluation then makes a bargain. It fixes a set of documents, a set of need statements and their judgments, and abstracts away the real users (Voorhees, 2007). The judgments stand in for shoppers here, and every score in this book inherits that bargain.
If matching words were the same as relevance, a product whose text holds every word of the query would be exact. You can test that on every judged pair in the store: for each pair, ask whether the product's name, class, category and description contain every word of the query, with the sample's descriptions cut to about 400 characters.
Of the judged pairs whose text holds every query word, what share did the judges call exact? Hold a number in mind.
1"""Checks whether a full word match makes a judged pair exact, and how many exact pairs have a text that misses a2query word, with and without attributes. The prose's point rests on the `of them exact` line and the3`of them, still missing one with attributes added` line."""4import re56from lib.store import load_labels, load_products, load_queries, product_text789def words(text):10 """Lowercase word tokens: a rough stand-in for what lexical search matches."""11 return set(re.findall(r"\w+", text.lower()))121314products, queries, labels = load_products(), load_queries(), load_labels()15text_words = {product_id: words(text) for product_id, text in product_text(products).items()}16attribute_words = {product_id: words(text) for product_id, text in products["attributes"].items()}17query_words = {query_id: words(query) for query_id, query in queries["query"].items()}18pairs = list(zip(labels["query_id"], labels["product_id"]))19# on sets, <= asks whether every query word is in the product's text20labels["in text"] = [query_words[query_id] <= text_words[product_id] for query_id, product_id in pairs]21# | is set union: the product's text words with its attribute words added22labels["in text or attributes"] = [query_words[query_id] <= text_words[product_id] | attribute_words[product_id]23 for query_id, product_id in pairs]2425matched = labels[labels["in text"]]26print(f"exact share of all judged pairs: {(labels['label'] == 'exact').mean():.3f}")27print(f"judged pairs whose text has every query word: {len(matched):,}")28for label in ["exact", "partial", "irrelevant"]:29 print(f"of them {label}: {(matched['label'] == label).mean():.3f}")3031exact = labels[labels["label"] == "exact"]32print(f"exact pairs: {len(exact):,}")33print(f"exact pairs whose text misses a query word: {(~exact['in text']).sum():,}")34print(f"of them, still missing one with attributes added: {(~exact['in text or attributes']).sum():,}")3536curtains = list(exact.loc[exact["query_id"] == 246, "product_id"]) # query 246: clear shower curtain37unsaid = [product_id for product_id in curtains if "clear" not in text_words[product_id]]38print("exact products for clear shower curtain:", len(curtains))39print("of them without the word clear in their text:", len(unsaid))40print("of those, with clear in their attributes:", sum("clear" in attribute_words[product_id] for product_id in unsaid))
python word-match-versus-judgment.pyexact share of all judged pairs: 0.249
judged pairs whose text has every query word: 1,177
of them exact: 0.777
of them partial: 0.211
of them irrelevant: 0.012
exact pairs: 3,252
exact pairs whose text misses a query word: 2,337
of them, still missing one with attributes added: 2,251
exact products for clear shower curtain: 12
of them without the word clear in their text: 8
of those, with clear in their attributes: 4
A full word match is a strong signal, but it doesn't settle the question. Of the 1,177 judged pairs whose text holds every query word, 0.777 are exact, 0.211 partial and 0.012 irrelevant, against an exact share of 0.249 over all judged pairs. The words can be present without the product being what the shopper meant, so the judges decide, not the words. A share near 1 would mean matching words is relevance, and the judges say it isn't.
The other direction is more striking. Most exact pairs, 2,337 of the 3,252, have a product text that misses at least one query word. Adding the attributes still leaves 2,251 of those missing a word, so attributes explain few misses overall.
Some misses are word forms and spelling, such as chairs against chair (chapter 1) or the query ligth bulb. Chapter 5 and chapter 8 take those up. Others are attributes the text doesn't state. WANDS records attributes such as color, material and size apart from the text (Chen et al., 2022, Table 1), and the store keeps a few of them in an attributes column, for example color: clear; material: vinyl (data card). The text the store's search reads, lib.store.product_text, leaves that column out.
For some queries, though, the attributes hold the answer. For clear shower curtain, 8 of the 12 exact products never say clear in their text, and 4 of those carry clear in their attributes, where the store's text search never looks. Chapter 5 tests adding attributes to lexical search. The rest of those curtains say clear nowhere, in their text or their attributes.
The judges had more to go on. WANDS's annotation guidelines show them each product's name, image and web page (guidelines). The photo or the page may be what told them, but that is our guess, because the data doesn't record which evidence a judge used.
Working rule: Treat a judgment as a person's verdict about a need, not a property of the product. Before you blame a ranker for missing an exact product, check whether any field the ranker reads says what the judges saw, and look at the fields it doesn't read, such as attributes. When your judges see more than your ranker does, give the ranker the same evidence where you can, or accept that some exact products will rank low for reasons no change to the ranking can fix.
预期评判者会在部分配对上意见不一
两个人可能得出不同的结论。WANDS 的每个标签已经是最多三名标注员的多数投票结果(Chen et al., 2022,第 5 节),而这些标注员起初只有中等程度的一致性,经过培训后才逐渐趋同。Cohen's kappa 会扣除偶然一致的部分,它从最初两个月的 0.467 上升到第三个月的 0.664,第四个月达到 0.826(表 2)。第 4 章会计算 kappa。
WANDS 的文件对商店的一些配对列出了不止一次,每一行都是一次多数投票,其中少数配对的投票结果不同。商店的样本每个配对只保留一行,精确匹配记为 2,部分匹配记为 1,不相关记为 0;各行不一致时,保留较低的等级(加载商店的数据)。下面是各行不一致的配对,以及样本保留的等级:
1"""Lists the pairs WANDS labelled two ways, with the label the sample kept, and counts the conflicts by kind. The2prose's point rests on the `conflicts between neighbouring grades` line."""3import pandas as pd45from lib.store import DATA, load_labels, load_products, load_queries67GRADES = {"exact": 2, "partial": 1, "irrelevant": 0}89products, queries, labels = load_products(), load_queries(), load_labels()10conflicts = pd.read_csv(DATA / "label_conflicts.csv") # pairs WANDS lists more than once, with different labels11# the sample's kept label for each pair, which is the lower grade wherever WANDS's rows differ12kept = labels.set_index(["query_id", "product_id"])["label"]1314for row in conflicts.itertuples():15 print(f"{queries.at[row.query_id, 'query']:<32} {products.at[row.product_id, 'name'][:40]:<41} "16 f"{row.labels:<19} kept: {kept[(row.query_id, row.product_id)]}")1718# each conflicting pair's two labels, from the column where "|" separates them19pairs = conflicts["labels"].str.split("|")20# a spread of 1 means neighbouring grades: exact and partial, or partial and irrelevant21spread = pairs.apply(lambda both: max(GRADES[label] for label in both) - min(GRADES[label] for label in both))22print()23print("pairs with conflicting labels:", len(conflicts))24# frozensets ignore which of the two labels comes first25print("exact against partial:", (pairs.apply(frozenset) == frozenset(["exact", "partial"])).sum())26print("partial against irrelevant:", (pairs.apply(frozenset) == frozenset(["partial", "irrelevant"])).sum())27print(f"conflicts between neighbouring grades: {(spread == 1).sum()} of {len(conflicts)}")
python conflicting-judgments.pywhimsical solar lights solar powered led pathway light pack exact|partial kept: partial
indoor multi tiered plant stand taranto multi-tiered plant stand exact|partial kept: partial
clear shower curtain shower curtain hooks irrelevant|partial kept: irrelevant
breakfast bar table counter height dining table exact|partial kept: partial
wood floor protector mat hard floor straight rectangular chair ma exact|partial kept: partial
malachi sled sled coffee table irrelevant|partial kept: irrelevant
kohler purist brushed bronze purist widespread bathroom faucet with d exact|partial kept: partial
pairs with conflicting labels: 7
exact against partial: 5
partial against irrelevant: 2
conflicts between neighbouring grades: 7 of 7
在已评判的配对中,有 7 存在冲突:其中 5 是精确与部分之争,2 是部分与不相关之争。每一处冲突都出现在相邻等级之间,即 7 of 7,所以薄弱之处在于两个等级之间的界线,而不是精确与不相关之间的差距。breakfast bar table 这一行就是这样一个例子:一张高吧台式餐桌是否完全符合该查询,是关于修饰词的合理问题,并不表示某一票误解了任务。
工作准则: 每个配对只保留一个等级,并写下选定该等级所依据的规则。我们建议像样本那样采用较低的等级:这样出现的偏差不利于对该商品排序的排序器,而不是偏向它。当评判者的分歧跨越两个等级,即精确与不相关之间时,应与他们一起重读标注指南,因为他们对相关性没有共同的定义。
这部分尚未翻译,因此以英文显示。
Read intent as a product and its modifiers
Every judgment starts from a reading of what the query asks for. A shopper's intent is what they want to do. A widely cited taxonomy of web search sorts intent into navigational (reach a particular site), informational (find information) and transactional (do something on the web), and its author makes no assumption that intent can be inferred from the query with any certainty (Broder, 2002). Product search reshapes those kinds. Tsagkias et al. describe navigational product queries as ones that name a serial number or a product, and informational ones as usually mapping to a leaf in the inventory taxonomy or asking about specific attributes. They add that "there is no single taxonomy for eCommerce customer intent", because it depends on the store's business and inventory (Tsagkias et al., 2020, section 2).
WANDS writes intent into its labels. A query names a target entity, the kind of product, and the query's modifiers are the attributes the shopper asked for. Exact satisfies both. Partial matches only the target entity and fails the modifiers (Chen et al., 2022, section 3.2). The guidelines' worked example is a query for a kids' chair: a product that is a chair but not for kids is partial (guidelines). Even with that rule, a broad query can be hard to judge. The authors' own example is a shopper who wants an induction cooktop and types cooktop (section 1).
The store's queries cover the range. ines shag rug w003096809 and kohler whitehaven farmhouse kitchen sink name a product line or a model, which makes them navigational. outdoor light fixtures names a category. clear shower curtain and 48 inch kitchen hood add a modifier to an entity. The store's queries also carry query_class, WANDS's predicted class for each query (Table 1). It is a guess at the entity, says nothing about modifiers, and isn't a judgment. Check how often it agrees with the products the judges called exact:
1"""For each query, measures how many of its exact products carry the predicted class, and lists the queries where none2does. The prose's point rests on the `queries where most exact products have the query_class` line and the3`queries where no exact product has the query_class` line."""4import pandas as pd56from lib.store import load_labels, load_products, load_queries78products, queries, labels = load_products(), load_queries(), load_labels()9exact = labels[labels["label"] == "exact"].join(products["product_class"], on="product_id")1011rows = []12for query_id, group in exact.groupby("query_id"):13 predicted = queries.at[query_id, "query_class"]14 classes = group["product_class"].str.split("|") # a product can have several classes15 # the commonest product_class value, which can join several classes, comes first in value_counts(); a missing class is an empty string, which prints as (none)16 rows.append({"query": queries.at[query_id, "query"], "query_class": predicted,17 "share of exact products in it": classes.apply(lambda names: predicted in names).mean(),18 "commonest class of exact products": group["product_class"].value_counts().index[0] or "(none)"})19check = pd.DataFrame(rows)2021print("queries:", len(check))22print("queries without a query_class:", check["query_class"].isna().sum())23# most means more than half of a query's exact products carry the predicted class24print("queries where most exact products have the query_class:",25 (check["share of exact products in it"] > 0.5).sum())26missed = check[(check["share of exact products in it"] == 0) & check["query_class"].notna()]27print("queries where no exact product has the query_class:", len(missed))28print()29print(missed.drop(columns="share of exact products in it").to_string(index=False))
python predicted-class-check.pyqueries: 80
queries without a query_class: 1
queries where most exact products have the query_class: 63
queries where no exact product has the query_class: 6
query query_class commonest class of exact products
foutains with brick look Wallpaper Fountains
70s inspired furniture Wall Art (none)
7qt slow cooker Crock Pots & Slow Cookers Slow Cookers
sheffield home bath set Bath Rugs & Mats Vanities
luau string lights String Lights Holiday Lighting
rustic twig Faux Plants and Trees Wall & Accent Mirrors
The predicted class usually agrees with the exact products. For 63 of the 80 queries, most exact products carry it, and for 6 queries, none does, and 1 query has no class at all. Read each row the example prints before counting it as an error. foutains with brick look is classed as Wallpaper, while its exact products are fountains, so the classifier went wrong. 7qt slow cooker is a naming mismatch instead: the query classes and the product classes use different names for the same thing.
Working rule: Read each query as an entity plus modifiers, and check that your ranker gets the entity right before you tune it for modifiers. Use query_class as a hint, for a feature or a filter you check against the judgments, and never as a label. When intent is ambiguous, as with cooktop, judge against the guidelines' reading and write that reading down.
用等级评定相关性,而不是简单的是或否
二元标签只说明相关或不相关。分级标签则说明相关到什么程度,例如 WANDS 的 exact(完全匹配)、partial(部分匹配)或 irrelevant(不相关)。各个结果的有用程度并不相同,因此最相关的结果应当被找出来并排在最前面。Järvelin 和 Kekäläinen在 2002 年论证了使用等级的理由。在商店里,差一点就符合的商品往往也是不错的次选。一块不透明的浴帘,被 WANDS 评为 partial,因为它是对的那类商品,只是缺少查询所要求的属性(把意图理解为商品及其修饰语),它比浴帘挂钩更接近购物者想要的东西。二元标签迫使我们在各个等级之间划一条线。只把 exact 算作相关,差一点就符合的商品与挂钩一样没有价值。把 exact 和 partial 都算作相关,它又与透明浴帘价值相同。
该商店的查询 clear shower curtain 同时显示了这两种失败。用它的评判结果构造三个前 10 名排序:十个 exact 商品,十个被评为 partial 的浴帘,以及十个被评为 irrelevant 的浴帘挂钩。然后在三种标注方式下,用 nDCG@10(为单个排序打分)给每个排序打分:以 exact 为相关的二元标注,以 exact 或 partial 为相关的二元标注,以及本书采用的等级,即 exact 为 2、partial 为 1、irrelevant 为 0,并将其用作增益。增益(gain)是指某个位置上的结果在应用位置折扣之前为得分增加的数值。
1"""Scores three rankings of one query's judged products under three labelings, to show what each labeling counts as2relevant. The prose's point rests on the `grades` lines, since only the grades separate the three rankings."""3from lib.metrics import ndcg_at_k4from lib.store import load_products, qrels56classes = load_products()["product_class"]7judgments = qrels()[246] # clear shower curtain: {product_id: grade}8910def first_ten(grade, product_class=None):11 """Ten products judged with this grade for the query, in product_id order."""12 # sorted(judgments) takes product_id order as the tie-break; None for product_class means any class13 return [product_id for product_id in sorted(judgments) if judgments[product_id] == grade14 and product_class in (None, classes[product_id])][:10]151617rankings = {"clear curtains": first_ten(2), # exact18 "other curtains": first_ten(1, "Shower Curtains"), # partial: not clear19 "hooks": first_ten(0, "Shower Curtain Hooks and Accessories")} # irrelevant20labelings = {"exact only": {2: 1, 1: 0, 0: 0}, # binary: relevant means exact21 "exact or partial": {2: 1, 1: 1, 0: 0}, # binary: relevant means exact or partial22 "grades": {2: 2, 1: 1, 0: 0}} # graded: the book's grades as gains2324print("exact products for this query:", sum(grade == 2 for grade in judgments.values()))25# the ideal top 10 holds the ten highest grades, so counting the 2s counts the exact products in it26print("exact products in the ideal top 10:", sorted(judgments.values(), reverse=True)[:10].count(2))27for labeling, gain in labelings.items():28 gains = {product_id: gain[grade] for product_id, grade in judgments.items()}29 for name, ranking in rankings.items():30 print(f"{name}, {labeling}: {ndcg_at_k(ranking, gains, 10):.3f}")
python binary-labels-hide-partial.pyexact products for this query: 12
exact products in the ideal top 10: 10
clear curtains, exact only: 1.000
other curtains, exact only: 0.000
hooks, exact only: 0.000
clear curtains, exact or partial: 1.000
other curtains, exact or partial: 1.000
hooks, exact or partial: 0.000
clear curtains, grades: 1.000
other curtains, grades: 0.500
hooks, grades: 0.000
只有等级才能把这三个排序区分开。在只认 exact 的标签下,其他浴帘与挂钩得分相同,分别为 0.000 和 0.000。在 exact 或 partial 的标签下,其他浴帘的得分为 1.000,与透明浴帘的 1.000 相同。使用等级时,透明浴帘得 1.000,其他浴帘得 0.500,挂钩得 0.000。当理想的前 10 名全是 exact 时,就像这里理想前 10 名中有 10 个 exact 商品那样,由十个 partial 商品组成的排序,其得分等于 partial 增益与 exact 增益之比。
**工作准则:**我们建议商品搜索使用分级评判,这样指标就能认可差一点就符合的结果,又不会把它误当作匹配。即使同时报告二元指标,也应收集等级:等级总能合并为是或否,反过来则不行。当差一点就符合的结果对用户毫无价值时,应使用二元标签,例如查询是型号,只有该型号才可以。
有意识地选择增益
分级只是标签,增益才是每个标签在指标中的价值,而这是一种选择。本书直接把每个等级作为其增益,因此部分相关的价值是完全相关的一半。WANDS 自己的论文把完全相关、部分相关和不相关的权重分别定为 1.0、0.5 和 0(Chen et al., 2022, Table 3),比例相同,因此 nDCG 也相同。Shopping Queries Dataset 简称 ESCI(Reddy et al., 2022),包含 260 万条人工标注的查询-商品判断,涉及 13 万条互不相同的查询,其分级方式不同。它把替代品(可作为功能替代的商品,例如毛衣查询对应的抓绒衣)与互补品(可与完全相关的商品搭配使用的商品)区分开来。它的增益是:完全相关为 1.0,替代品为 0.1,互补品为 0.01,不相关为 0.0(Reddy et al., 2022, section 3.1)。WANDS 的部分相关与 ESCI 的替代品并不相同,但它是最接近的对应项。
对第 1 章中该商店的两次测试运行打分,即 BM25 和语义检索对每条测试查询给出的排序,并在下列每种方案下进行:用分级相关代替是非判断中的两种二元标注、本书的分级,以及参照 ESCI 思路取部分相关增益 0.1 的方案。该示例还在每种方案下,对语义检索的领先幅度运行第 3 章的置换检验和配对 t 检验,并输出两个 p 值。与本书中的所有检验一样,这两个检验的结果都只是描述性的。语义检索的运行读取开放模型 Snowflake/snowflake-arctic-embed-m-v1.5 的已记录嵌入(按语义搜索)。
1"""Scores BM25 and semantic retrieval on the test queries under four gain schemes, with exploratory paired tests on2semantic retrieval's lead. The prose's point rests on the `semantic minus bm25` lines, which keep one sign across the3schemes while their size changes."""4from lib.lexical import LexicalSearch5from lib.metrics import evaluate6from lib.semantic import SemanticSearch7from lib.significance import compare8from lib.store import load_products, load_queries, qrels910queries, judgments = load_queries("test"), qrels("test")11# each run keeps the top 100 products per query; the scores below read only the top 10 of each12runs = {"bm25": LexicalSearch(load_products()).run(queries, k=100), "semantic": SemanticSearch().run(queries, k=100)}1314for name, run in runs.items():15 # an unjudged product gives None here, which the count of 1 does not include16 grades = [judgments[query_id].get(product_id) for query_id, ranking in run.items() for product_id in ranking[:10]]17 print(f"partial share of the {name} top 10: {grades.count(1) / len(grades):.3f}")1819# each scheme turns a grade into the gain it is worth20schemes = {"exact only": {2: 1, 1: 0, 0: 0},21 "exact or partial": {2: 1, 1: 1, 0: 0},22 "grades 2, 1, 0": {2: 2, 1: 1, 0: 0}, # the book's choice; WANDS's 1, 0.5, 0 gives the same nDCG23 "partial gain 0.1": {2: 1, 1: 0.1, 0: 0}} # close to the gains ESCI chose for its substitutes24leads = 025for scheme, gain in schemes.items():26 gains = {query_id: {product_id: gain[grade] for product_id, grade in per_query.items()}27 for query_id, per_query in judgments.items()}28 scores = {name: evaluate(run, gains)["ndcg@10"] for name, run in runs.items()}29 test = compare(scores["semantic"], scores["bm25"]) # exploratory paired tests (chapter 3): no planned comparisons30 print(f"bm25, {scheme}: {scores['bm25'].mean():.3f}")31 print(f"semantic, {scheme}: {scores['semantic'].mean():.3f}")32 print(f"semantic minus bm25, {scheme}: {test['mean difference']:.3f}")33 print(f"permutation p, {scheme}: {test['permutation p']:.3f}")34 print(f"t-test p, {scheme}: {test['t-test p']:.3f}")35 print(f"tied queries, {scheme}: {test['ties']}")36 # a lead is a positive mean difference, semantic minus BM2537 leads += test["mean difference"] > 038print(f"schemes where semantic leads: {leads} of {len(schemes)}")
python gain-choice-changes-ndcg.pypartial share of the bm25 top 10: 0.403
partial share of the semantic top 10: 0.380
bm25, exact only: 0.519
semantic, exact only: 0.648
semantic minus bm25, exact only: 0.130
permutation p, exact only: 0.000
t-test p, exact only: 0.002
tied queries, exact only: 3
bm25, exact or partial: 0.903
semantic, exact or partial: 0.983
semantic minus bm25, exact or partial: 0.080
permutation p, exact or partial: 0.010
t-test p, exact or partial: 0.026
tied queries, exact or partial: 19
bm25, grades 2, 1, 0: 0.725
semantic, grades 2, 1, 0: 0.830
semantic minus bm25, grades 2, 1, 0: 0.105
permutation p, grades 2, 1, 0: 0.001
t-test p, grades 2, 1, 0: 0.003
tied queries, grades 2, 1, 0: 2
bm25, partial gain 0.1: 0.564
semantic, partial gain 0.1: 0.688
semantic minus bm25, partial gain 0.1: 0.125
permutation p, partial gain 0.1: 0.000
t-test p, partial gain 0.1: 0.002
tied queries, partial gain 0.1: 2
schemes where semantic leads: 4 of 4
在 4 of 4 种方案下语义检索都领先,但领先幅度的大小取决于这一选择。部分相关的商品占据了 BM25 前 10 位中的 0.403,以及语义检索前 10 位中的 0.380,因此部分相关的增益会使两个得分都发生很大变化。BM25 在仅完全相关标注下得分为 0.519,在完全或部分相关标注下为 0.903。采用类似 ESCI 的 0.1 部分相关增益时,BM25 得分为 0.564,语义检索为 0.688。领先幅度在仅完全相关标注下为 0.130,在类似 ESCI 的增益下为 0.125,在采用分级时为 0.105,在完全或部分相关标注下为 0.080。该示例并未检验这些领先幅度之间是否存在差异,因此在 30 条测试查询上,应把它们之间的差别视为可能只是噪声。
完全或部分相关标注还会出现饱和,能区分排序器的查询更少。在这种标注下有 19 条测试查询打成平局,而在分级下为 2 条。
工作准则: 对商品搜索,默认使用线性增益,即分级 2、1 和 0。如果差一点命中对你的购物者价值不大,就选择较低的部分相关增益。无论怎样选择,都要在比较排序器之前确定下来,在每个得分旁一并报告,并且绝不要比较用不同增益算出的 nDCG。
读懂每个查询的标签构成
增益设定了每个等级的价值,却没有说明一个查询中每种等级各有多少。一个判定集并不是单一的总体:每个查询都有自己的一批已判定商品,也有自己的等级构成,而这一构成决定了该查询的完美排序是什么样子。宽泛查询指的是所指品类能被许多商品满足的查询,它有很多完全匹配的商品。具体查询带有若干修饰语,可能只有寥寥几个。对查询取平均,会把这些不同的问题混在一起,所以在求任何平均值之前,应先读懂这一构成。
商店中的查询并不是从 WANDS 中随机抽取的。每个查询都被选定为有 60 到 300 个已判定商品,至少有 3 个完全匹配判定和 20 个部分匹配判定,并且其预测类别不与任何其他入选查询重复(数据卡)。这些计数描述的是样本,而不是整个 WANDS,因为超出上述限制的查询不会进入商店。请为 80 个查询中的每一个统计标签数量,并按完全匹配商品所占的比例对查询分组:
1"""Counts each query's judgments by label, and groups the queries by their share of exact products. The prose's point2rests on the `queries with an exact share of 0.1 or less` line and on the histogram below it."""3import pandas as pd45from lib.store import load_labels, load_queries67queries, labels = load_queries(), load_labels()8counts = pd.crosstab(labels["query_id"], labels["label"])[["exact", "partial", "irrelevant"]]9shares = counts.div(counts.sum(axis=1), axis=0).join(queries[["query", "split"]])1011print(f"partial share of all judgments: {(labels['label'] == 'partial').mean():.3f}")12print(f"median exact share: {shares['exact'].median():.3f}")13print(f"median irrelevant share: {shares['irrelevant'].median():.3f}")14for label, which in [("exact", "lowest"), ("exact", "highest"), ("irrelevant", "highest")]:15 query_id = shares[label].idxmin() if which == "lowest" else shares[label].idxmax()16 print(f"{which} {label} share: {shares.at[query_id, label]:.3f}")17 print(f"query with the {which} {label} share: {shares.at[query_id, 'query']}")18print("queries with no irrelevant judgment:", (counts["irrelevant"] == 0).sum())19print("fewest exact products for a query:", counts["exact"].min())20print("most exact products for a query:", counts["exact"].max())21# 10 is the size of a top 10: a query with fewer exact products can't fill a perfect top 10 with exact ones22few = counts["exact"] < 1023print("queries with fewer than 10 exact products:", few.sum())24print("test queries with fewer than 10 exact products:", (few & (shares["split"] == "test")).sum())2526print("queries with an exact share of 0.1 or less:", (shares["exact"] <= 0.1).sum())27print()28print("exact share queries")29bins = pd.cut(shares["exact"], [tenth / 10 for tenth in range(11)]) # no share is 0: each query has 3 or more exact30for interval, number in bins.value_counts(sort=False).items():31 print(f"{interval.left:.1f} to {interval.right:.1f} {number:>3} {'#' * number}".rstrip())
python count-labels-per-query.pypartial share of all judgments: 0.610
median exact share: 0.207
median irrelevant share: 0.060
lowest exact share: 0.014
query with the lowest exact share: 48 in entry table with side by side drawer
highest exact share: 0.749
query with the highest exact share: outdoor light fixtures
highest irrelevant share: 0.776
query with the highest irrelevant share: podium with locking cabinet
queries with no irrelevant judgment: 9
fewest exact products for a query: 3
most exact products for a query: 193
queries with fewer than 10 exact products: 23
test queries with fewer than 10 exact products: 7
queries with an exact share of 0.1 or less: 29
exact share queries
0.0 to 0.1 29 #############################
0.1 to 0.2 10 ##########
0.2 to 0.3 12 ############
0.3 to 0.4 11 ###########
0.4 to 0.5 9 #########
0.5 to 0.6 1 #
0.6 to 0.7 7 #######
0.7 to 0.8 1 #
0.8 to 0.9 0
0.9 to 1.0 0
部分匹配是最常见的标签,占全部判定的 0.610。可能的原因在于 WANDS 收集候选商品的方式,详见了解判定从何而来。完全匹配的占比在各查询之间差异很大,从 48 in entry table with side by side drawer 的 0.014 到 outdoor light fixtures 的 0.749,中位数为 0.207。示例打印的直方图在低端最高:有 29 个查询的完全匹配占比不超过 0.1。
对大多数查询而言,不相关判定很少见,但并非全部如此:不相关占比的中位数为 0.060,而 podium with locking cabinet 的这一占比达到 0.776,另有 9 个查询完全没有不相关判定。
这种分布差异改变了 nDCG@10 所衡量的内容。各查询的完全匹配商品数从 3(抽样规则的下限)到 193 不等,有 23 个查询的完全匹配商品少于 10 个,其中 7 个是测试查询。对这些查询来说,理想的前 10 名包含部分匹配商品,因此前 10 名中有部分匹配商品的排序也可以得到满分。对于 outdoor light fixtures 这样的宽泛查询,要得满分则需要十个完全匹配商品。同样的 nDCG@10,对两者的含义并不相同。
**工作准则:**在对任何指标求平均之前,先打印每个查询的标签构成,并且两端都要看。完全匹配商品很少的查询得分低,与宽泛查询得分低,应视为不同的问题。每个均值旁都要保留各查询的得分;第 3 章说明了平均值会掩盖什么。
把点击当作有偏的证据
商店会记录每一次点击,而且收集点击不花任何成本,所以点击看起来像是免费的相关性判断。其实不然。点击只说明某位购物者选择查看了某件商品,并不说明该商品满足了需求;而购物者是否点击,取决于商品在何处、以何种方式展示。
关于点击如何误导,证据来自网页搜索。Joachims 等人追踪了康奈尔大学本科生使用 Google 回答十个既定问题时的眼动和点击,实验分两个阶段,第一阶段 34 名学生,第二阶段 22 名学生。在第二阶段,一部分学生看到前两条结果被互换,或整个列表被倒置(Joachims et al., 2005)。他们的结论是,点击“有信息量,但存在偏差”。
第一种偏差是位置偏差:人们更多地点击排名靠前的结果,部分原因是信任排序。Joachims 等人把这种效应称为“信任偏差”,它会把点击吸引到排名靠前的链接上,即使用户浏览过的其他摘要相关性更高。作者还发现,相对偏好,例如被点击的结果优于其上方被跳过的结果,平均而言相当准确。
在一个网页搜索引擎的大规模实验中,Craswell 等人发现,在靠前的排名位置上,级联模型对位置偏差的解释最好。在级联模型中,用户自上而下阅读结果,并在遇到第一个有价值的文档时离开(Craswell et al., 2008)。
呈现偏差不止于位置:点击取决于结果如何展示,而不只是展示在哪里。在 Google 网页搜索的一部分流量上,标题更有吸引力的结果,即查询词在标题中加粗更多的结果,获得了更多点击,即使控制了位置以及人工判断的相关性之后也是如此(Yue et al., 2010)。在商店中,图片和价格最接近标题的作用。这是我们的推断,不是实测结果。
WANDS 没有点击数据,因此商店无法度量这些偏差。不过,你仍然可以看到点击日志会记录什么。取商店对 clear shower curtain 的判断,并以最严格的形式应用级联模型:购物者从顶部开始阅读,点击第一件完全匹配的商品后离开。这个思想实验使用一条固定规则,不是真实购物者的模型。第 11 章会用概率来模拟点击。对 BM25 的排序和语义检索的排序分别运行这个思想实验,并把两份点击日志并排对照:
1"""Reads the top 10 of BM25 and of semantic retrieval for `clear shower curtain` as a cascade of clicks, which stops at2the first exact product. The prose's point rests on the `clicked rank` lines and on the `exact products shown below the3click` lines."""4from lib.lexical import LexicalSearch5from lib.semantic import SemanticSearch6from lib.store import load_products, qrels78products = load_products()9judgments = qrels()[246] # clear shower curtain: {product_id: grade}10# search returns scores indexed by product_id, so .index is the ranking; semantic search takes the query's id, 24611rankings = {"bm25": list(LexicalSearch(products).search("clear shower curtain", k=10).index),12 "semantic": list(SemanticSearch().search(246, k=10).index)}1314for name, ranking in rankings.items():15 grades = [judgments.get(product_id, 0) for product_id in ranking] # an unjudged product counts as grade 016 rank = grades.index(2) + 1 # cascade: read from the top, click the first exact product, stop17 print(f"{name} grades, ranks 1 to 10: {' '.join(map(str, grades))}")18 print(f"{name} clicked rank: {rank}")19 # rank counts from 1 and the list from 0, so the clicked product sits at index rank - 120 print(f"{name} clicked product: {products.at[ranking[rank - 1], 'name']}")21 # grades[rank:] starts just after the click, since rank is the clicked product's position counted from 122 print(f"{name} exact products shown below the click: {grades[rank:].count(2)}")
python cascade-clicks.pybm25 grades, ranks 1 to 10: 1 2 2 2 2 1 1 0 0 0
bm25 clicked rank: 2
bm25 clicked product: woolley vinyl single shower curtain
bm25 exact products shown below the click: 3
semantic grades, ranks 1 to 10: 2 2 1 1 2 1 2 2 1 1
semantic clicked rank: 1
semantic clicked product: woolley vinyl solid color single shower curtain
semantic exact products shown below the click: 4
点击日志记录的,既是购物者想要的,也是排序器最先展示的。在 BM25 的排序下,购物者点击第 2 位的 woolley vinyl single shower curtain,并且始终看不到其下方的 3 件完全匹配的商品。在点击日志中,这些完全匹配的商品与 BM25 排在第 8 到 10 位、被判为不相关的浴帘挂钩(用 BM25 对商品排序)看起来没有区别,因为同样没有人点击。
在语义检索的排序下,点击落在第 1 位的 woolley vinyl solid color single shower curtain 上。同一份商品目录产生的两种排序,会得到两份归功于不同商品的点击日志。
WANDS 的作者出于同样的原因,即位置偏差,没有基于日志构建数据集(对照判断检查结果)。不过,点击仍然带有信息。在 WANDS 中,取自点击日志的候选项里有 62.47% 被判为完全匹配,而取自开源排序系统的候选项中这一比例为 30.95%(Chen et al., 2022, Table 4)。
对点击进行偏差校正本身就是一个领域,即无偏学习排序。在百度发布的一个大型搜索数据集上,这些方法改善了点击预测,但按专家判断,它们“难以持续提升排序性能”(Hager et al., 2024)。
工作准则: 离线评估应以判断而非点击为基准。把点击当作相对证据使用,即被点击的商品优于排在其上方、被跳过的商品;绝不要把没有点击当作商品不相关的标志。如果用点击训练模型,请对照判断检查结果。
了解判断从何而来
没有人会针对每个查询去判断每一件商品,因此测试集合改为判断一个池(pool),即为每个查询收集的一组候选项。在 TREC(Text REtrieval Conference,文本检索会议)的共享评测中,一个主题的池是各参与者检索到的排名靠前文档的并集,通常取前 100 篇,池之外的每篇文档都算作不相关。Voorhees 认为,只要池中包含足够多的相关文档,把所有未判断文档都算作不相关,仍然可以公平地比较各个系统。同一篇文章也指出,对于任意大的文档集,池化无法产生可复用的测试集合(Voorhees, 2007)。Zobel 检验了这一公平性的说法。使用池化判断时,各系统的相对表现看起来是可靠的,但召回率被高估了,因为很多相关文档很可能从未被找到(Zobel, 1998)。
WANDS 分两步构建它的池(Chen et al., 2022, 第 4.2 和 4.3 节)。第一步取用购物者点击过或加入购物车的商品,再加上开源词法搜索引擎 Solr 和一个神经检索系统返回的结果。第二步用同样的两个系统再次在整个商品目录中挖掘,并加入查询分类模型为该查询预测的类别中的每一件商品。第二步产生了 WANDS 的大部分标签,而且多数是接近但不完全匹配的情形:其标签中 4.12% 为完全匹配,63.99% 为部分匹配(Table 5)。在本店的样本中,部分匹配同样是最常见的标签(读取每个查询的标签构成)。作者认为,完整性对商品搜索很重要,因为“incompleteness in Product Search dataset does negatively impact the discriminative power of the evaluation”(section 3.1)。
池属于某个查询。对某个查询没有判断的商品,对该查询而言是未判断的;而对一个查询已有判断的商品,对另一个相似的查询可能是未判断的。本店的两个查询说明了这一点:训练查询 trundle daybed 和测试查询 full bed with trundle。请统计对前者判为完全匹配、而对后者没有判断的商品数量,并查看判断者如何给有判断的那些商品评级:
1"""Counts the products judged exact for the train query `trundle daybed` that have no judgment for the test query `full2bed with trundle`, and grades the ones that do. The prose's point rests on the `of them unjudged for full bed with3trundle` line and the `judged partial` line."""4from collections import Counter56from lib.store import load_products, load_queries, qrels78products, queries, judgments = load_products(), load_queries(), qrels()9LABELS = {2: "exact", 1: "partial", 0: "irrelevant"}1011# 433 is the train query trundle daybed, and its exact products come out in product_id order12daybeds = sorted(product_id for product_id, grade in judgments[433].items() if grade == 2)13full_bed = judgments[436] # {product_id: grade} for "full bed with trundle"14print(f"433: {queries.at[433, 'query']} ({queries.at[433, 'split']})")15print(f"436: {queries.at[436, 'query']} ({queries.at[436, 'split']})")16print("exact for trundle daybed:", len(daybeds))1718unjudged = [product_id for product_id in daybeds if product_id not in full_bed]19print("of them unjudged for full bed with trundle:", len(unjudged))20print("of them judged for full bed with trundle:", len(daybeds) - len(unjudged))21judged = Counter(LABELS[full_bed[product_id]] for product_id in daybeds if product_id in full_bed)22for label in LABELS.values():23 print(f"judged {label}: {judged[label]}")2425# daybeds is in product_id order, so the first two unjudged are the two lowest product_ids among them26print("first unjudged daybed:", products.at[unjudged[0], "name"])27print("second unjudged daybed:", products.at[unjudged[1], "name"])
python trundle-daybeds-unjudged.py433: trundle daybed (train)
436: full bed with trundle (test)
exact for trundle daybed: 113
of them unjudged for full bed with trundle: 77
of them judged for full bed with trundle: 36
judged exact: 5
judged partial: 31
judged irrelevant: 0
first unjudged daybed: daybed with trundle
second unjudged daybed: pattonsburg daybed with trundle
把未判断商品计为 0 分的指标,是在对它进行猜测,而这里的猜测很重要。按我们对指南规则的理解,带拖床的沙发床往往是类型正确、但不符合查询所要求尺寸的床。在对 trundle daybed 判为完全匹配的 113 件商品中,有 77 件对 full bed with trundle 没有判断,其中包括商品编号最小的两件,即 daybed with trundle 和 pattonsburg daybed with trundle。在有判断的 36 件中,判断者判为完全匹配的有 5 件,部分匹配的有 31 件,不相关的有 0 件。如果未判断的沙发床与已判断的相似,它们大多也会是部分匹配,然而指标给每一件都记 0 分。
Zobel 警告说,池化判断可能低估没有为池做出贡献的系统(Zobel, 1998)。这使得判断对与构建池的系统相似的排序器最为公平:对 WANDS 而言,就是购物者的点击、一个词法引擎、一个神经检索器和一个查询分类器。这是一种倾向,而不是定律。在这里,与 WANDS 的词法引擎最相似的排序器 BM25,在两个排序器中 judged@10 较低(统计未判断商品)。我们的猜测是,本店的 BM25 读取的字段与池所用引擎不同,因此它把不同的商品排在了前面。
工作准则: 在信任一次比较之前,先弄清楚你的池来自哪里。当你评估一个与构建池的系统不同的排序器时,要预期它会呈现无人判断过的商品,并计划对这些商品进行判断。
这部分尚未翻译,因此以英文显示。
Count unjudged products before you compare scores
A product nobody judged for a query is unjudged: its grade is unknown, not 0. Standard metrics can't tell the difference. They score an unjudged product exactly as they score one judged irrelevant, and Buckley and Voorhees warn that such metrics become unreliable when judgments are very incomplete (Buckley and Voorhees, 2004). Two diagnostics are standard in evaluation tools (ir-measures). Judged@k is the share of the top k that has a judgment of any grade:
judged@k = (number of the top k products that have a judgment for the query) / k
The second diagnostic is the condensed list, a ranking with its unjudged products dropped before scoring, so the judged ones move up. ir-measures offers it as the judged_only option. Buckley and Voorhees's bpref measure works in the same spirit: it uses judged documents only, and they found it more robust to incomplete judgments.
In chapter 1, BM25's mean judged@10 on the test queries is 0.927. The mean for semantic retrieval is 0.990. One property of the store's data keeps those numbers high. Its catalog is almost all judged: 11,917 of 12,000 products have a judgment for at least one query (Load the store's data). Expect a full catalog to put more unjudged products in the top 10s.
Build judged@10 by hand, check it against lib.metrics, and list where BM25's unjudged products are:
1"""Builds judged@10 by hand for BM25's test rankings, checks it against lib.metrics, and lists where BM25's unjudged2products sit in the top 10s. The prose's point rests on the `unjudged products in the top 10s` line and on the3`irrelevant judgments` line for pull out sleeper loveseat."""4import pandas as pd56from lib.lexical import LexicalSearch7from lib.metrics import evaluate8from lib.store import load_products, load_queries, qrels910products, queries, judgments = load_products(), load_queries("test"), qrels("test")11# 100 products per query are kept; this example reads only the top 10 of each12run = LexicalSearch(products).run(queries, k=100)131415def judged_at_k(ranking, judged, k):16 """The share of the top k that has a judgment of any grade."""17 return sum(product_id in judged for product_id in ranking[:k]) / k181920rows = []21for query_id, ranking in run.items():22 unjudged = [product_id for product_id in ranking[:10] if product_id not in judgments[query_id]]23 # the name is cut to 45 characters so that the table fits on the page24 rows.append({"query": queries.at[query_id, "query"], "judged@10": judged_at_k(ranking, judgments[query_id], 10),25 "unjudged": len(unjudged), "first unjudged product": products.at[unjudged[0], "name"][:45] if unjudged else ""})26table = pd.DataFrame(rows)2728print(f"mean judged@10: {table['judged@10'].mean():.3f}")29# lib.metrics' judged@10 per query, indexed by query_id30library = evaluate(run, judgments)["judged@10"]31# 1e-12 allows for floating-point rounding when the two judged@10 values are compared32agree = sum(abs(row["judged@10"] - library[query_id]) < 1e-12 for query_id, row in zip(run, rows))33print(f"queries where the hand-built judged@10 equals lib.metrics: {agree} of {len(rows)}")34print("unjudged products in the top 10s:", table["unjudged"].sum())35print("test queries with an unjudged product in the top 10:", (table["judged@10"] < 1).sum())36sleeper = list(judgments[244].values()) # pull out sleeper loveseat37print("judgments for pull out sleeper loveseat:", len(sleeper))38print("irrelevant judgments for pull out sleeper loveseat:", sleeper.count(0)) # grade 0 is irrelevant39print()40# judged@10 below 1 means the query has at least one unjudged product in its top 1041print(table[table["judged@10"] < 1].sort_values("judged@10").to_string(index=False))
python unjudged-in-bm25-top-10.pymean judged@10: 0.927
queries where the hand-built judged@10 equals lib.metrics: 30 of 30
unjudged products in the top 10s: 22
test queries with an unjudged product in the top 10: 7
judgments for pull out sleeper loveseat: 143
irrelevant judgments for pull out sleeper loveseat: 0
query judged@10 unjudged first unjudged product
outdoor light fixtures 0.2 8 60 watt equivalent e12 led candle light bulb
beach blue headboard 0.5 5 lemon frosting indoor/outdoor , removable cov
pull out sleeper loveseat 0.6 4 country oxford 87.75 '' rolled arm sleeper
full bed with trundle 0.8 2 abel staircase twin over full bunk bed with t
whimsical solar lights 0.9 1 luciole 0.12-watt led solar power dusk to daw
bistro table and chairs 0.9 1 drayton patio 5 piece bistro set
wooden chair outdoor 0.9 1 osterley 31.5 '' w writing desk and chair set
The hand-built judged@10 equals lib.metrics on 30 of 30 test queries, with a mean of 0.927. Unjudged products fill 22 of BM25's top-10 places, spread over 7 queries. By our reading, some are plainly irrelevant: light bulbs for outdoor light fixtures, an outdoor pillow for beach blue headboard. Others look like near misses or matches: a bunk bed with a trundle for full bed with trundle, a bistro set for bistro table and chairs, a sleeper sofa for pull out sleeper loveseat, which by the entity-and-modifier rule is probably a near miss: the judges called 0 of that query's 143 judged products irrelevant.
Score both rankers with condensed lists, which drop the unjudged products, to see how far the scores move:
1"""Scores BM25 and semantic retrieval on the test queries with standard and condensed nDCG@10, and runs exploratory2tests on the change in the lead. The prose's point rests on the `semantic minus bm25` lines and the `share of bm25's3gain` line."""4from lib.lexical import LexicalSearch5from lib.metrics import evaluate6from lib.semantic import SemanticSearch7from lib.significance import compare8from lib.store import load_products, load_queries, qrels910queries, judgments = load_queries("test"), qrels("test")11runs = {"bm25": LexicalSearch(load_products()).run(queries, k=100), "semantic": SemanticSearch().run(queries, k=100)}1213standard, condensed = {}, {}14for name, run in runs.items():15 # condensed: drop every unjudged product, so the judged ones move up into the top 1016 judged_only = {query_id: [product_id for product_id in ranking if product_id in judgments[query_id]]17 for query_id, ranking in run.items()}18 standard[name] = evaluate(run, judgments)["ndcg@10"]19 condensed[name] = evaluate(judged_only, judgments)["ndcg@10"]20 print(f"{name} ndcg@10: {standard[name].mean():.3f}")21 print(f"{name} condensed ndcg@10: {condensed[name].mean():.3f}")2223gain = condensed["bm25"] - standard["bm25"] # bm25's change, per test query24# the scores are indexed by query_id: 133 is outdoor light fixtures, and 46 is beach blue headboard25print(f"bm25 ndcg@10 for outdoor light fixtures: {standard['bm25'][133]:.3f}")26print(f"bm25 condensed ndcg@10 for outdoor light fixtures: {condensed['bm25'][133]:.3f}")27print(f"share of bm25's gain from outdoor light fixtures and beach blue headboard: {gain[[133, 46]].sum() / gain.sum():.3f}")2829lead, condensed_lead = standard["semantic"] - standard["bm25"], condensed["semantic"] - condensed["bm25"]30change = compare(condensed_lead, lead) # is the change in the lead more than noise? (exploratory)31print(f"semantic minus bm25: {lead.mean():.3f}")32print(f"semantic minus bm25, condensed: {condensed_lead.mean():.3f}")33print(f"change in the lead: {change['mean difference']:.3f}")34print(f"95% interval for the change: {change['ci low']:.3f} to {change['ci high']:.3f}")35print(f"t-test p for the change: {change['t-test p']:.3f}")36print(f"permutation p for the change: {change['permutation p']:.3f}")37condensed_test = compare(condensed["semantic"], condensed["bm25"]) # the condensed lead itself (exploratory)38print(f"t-test p for the condensed lead: {condensed_test['t-test p']:.3f}")39print(f"permutation p for the condensed lead: {condensed_test['permutation p']:.3f}")
python condensed-ndcg.pybm25 ndcg@10: 0.725
bm25 condensed ndcg@10: 0.778
semantic ndcg@10: 0.830
semantic condensed ndcg@10: 0.834
bm25 ndcg@10 for outdoor light fixtures: 0.286
bm25 condensed ndcg@10 for outdoor light fixtures: 1.000
share of bm25's gain from outdoor light fixtures and beach blue headboard: 0.612
semantic minus bm25: 0.105
semantic minus bm25, condensed: 0.056
change in the lead: -0.049
95% interval for the change: -0.138 to -0.015
t-test p for the change: 0.073
permutation p for the change: 0.036
t-test p for the condensed lead: 0.011
permutation p for the condensed lead: 0.009
Condensing lifts BM25 mostly by forgiving bad results, not by undoing a penalty on good ones. BM25 goes from 0.725 to 0.778, and semantic retrieval from 0.830 to 0.834. Of BM25's gain, a share of 0.612 comes from outdoor light fixtures and beach blue headboard, the two queries whose unjudged products are plainly irrelevant. For outdoor light fixtures, BM25 goes from 0.286 to 1.000: once the bulbs drop out, the judged lights below them move up.
Semantic retrieval's lead falls from 0.105 to 0.056 once the lists are condensed. Whether that change is more than noise is unsettled. Chapter 3's paired t-test gives p = 0.073 for the change, and its permutation test gives 0.036. The 95% bootstrap interval runs from -0.138 to -0.015. Both checks are exploratory. With the change carried by a few queries, treat it as possibly noise.
The condensed lead itself gets p = 0.009 from the permutation test and 0.011 from the t-test. Neither the standard score nor the condensed one is the truth. The standard score punishes unjudged products that may be good, and the condensed score hides unjudged products that are bad.
Working rule: Report judged@10 next to every nDCG@10, and compute condensed scores as a diagnostic, never as the headline. When two rankers' judged@10 differ, have the unjudged products in their top 10s judged before you compare them. That is the dependable cure, and on this sample it is a small job: 22 judgments for BM25's test run. On a full catalog, expect more.
生产环境中会出什么问题
**判断会过时。**商店的商品目录和购物者都在不断变化,相关性也不只取决于查询。Tsagkias 等人指出,相关性取决于用户、时间、查询和上下文,并举了一个例子:查询“夹克”时,冬季与夏季的相关商品并不相同(Tsagkias et al., 2020, section 3.1.1)。新商品进入目录时没有经过判断,而已经下架的商品仍留在判断集中。请长期跟踪生产环境排序器的 judged@10,并在它下降时重新进行判断。
**点击标签只会印证现有的排序。**用当前排序下记录的点击来训练排序器,它学到的不仅是购物者的偏好,还有该排序本身的顺序。请把人工判断保留为检验点击数据的参照(把点击视为有偏的证据)。
**标注指南决定判断的形态。**WANDS 的标注人员用了四个月才达到近乎完美的一致性(Chen et al., 2022, Table 2)。标注指南中的一些小选择会影响整个数据集。WANDS 要求标注人员忽略拼写错误,并按原样写法判断该查询示例(标注指南),因此它的判断默认读者能够理解 ligth bulb。做不到这一点的排序器在这类查询上得分会偏低,这对购物者而言是公平的。第 8 章构建了一个拼写纠错器,并说明它仍然会犯哪些错误。
**排序器看到的比标注者少。**标注者看到了图片和商品页面,而商店的排序器读取的是不含属性的文本(第 5 章测试了加入属性的效果)。任何被要求根据同样文本进行判断的大语言模型(LLM),看到的内容与排序器相同。这类系统与人工判断之间的部分分歧来自这一差距,而非系统本身。在归咎于排序器之前,请先检查标注者所看重的属性存放在哪些字段中。
**用模型填补空缺仍然需要人工判断。**Gienapp 等人针对每个查询,用该查询的人工判断微调一个小型分类器,再用它来判断未被判断的商品。他们报告称,在 TREC Robust04 以及 2019 年和 2020 年的 TREC Deep Learning 集合上,每个主题只需 128 条初始人工判断,就能提高模型之间的可比性,并且这种方法比通过提示词驱动的 LLM 判断者更可靠(Gienapp et al., 2025)。Soboroff 对 LLM 的建议则更为直接:“不要使用 LLM 为 TREC 式评测创建相关性判断”(Soboroff, 2025)。第 4 章先将 LLM 的判断与商店的人工判断对照衡量,然后才决定是否信任它们。
**工作准则:**为判断标注日期,在生产环境中跟踪 judged@10,并为重新判断预留预算。
本章的决定
除非训练查询上的测量另有结论,后续各章沿用这些决定。
- 以人工判断为基准。 本书所有的主要得分都以人工判断计算,包括用 LLM 构建的排序器的得分。点击反映的是购物者的行为,会受位置和呈现方式的影响而产生偏差,因此绝不用作标签。
- **等级 2、1、0 即为增益值,**因此部分相关的价值是完全相关的一半,与 WANDS 自己的 1.0、0.5 和 0 比例相同。二元指标只把完全相关计为相关。
- 每个配对只取一个等级。 当 WANDS 对同一配对给出多个不同标签时,样本保留较低的等级,这样偏向于对排序器不利的一侧。
- **
query_class只是提示,**既不是标签,也不是意图的度量。它的部分取值是错的。 - 未经判断的商品计为等级 0,并且每个 nDCG@10 旁都列出 judged@10。 压缩后的得分只用于诊断。前 10 名中出现未经判断的商品,解决办法是让这些商品得到判断。
- 排序器读取商品文本,而不读取属性。 第 5 章测试了把属性作为一个字段加入的做法,并让它们不进入 BM25 文本。评判者还看过图片和商品页面,因此有些完全相关的商品,仅凭文本的排序器很难与之匹配。
- 可以查看任何判断,但只在训练集上调优。 查看测试集的判断不会改变任何排序器;依据这些判断来选择修复方式则会改变。
- **本章表明:**得分的可靠程度,取决于判断的质量。
本章,在本版次中
版次 2026.11.10示例于 2026年11月19日 通过本章 10个示例 个示例中有 10 个通过
译自英文版图书,版次 2026.11.10。