测试变更
阅读约需 67分钟 分钟,12节中含11个示例。
读完本章,你将能够判断一项搜索变更是否应当上线:逐条查询对比两个排序器的结果,避免测试查询让变更显得比实际更好,估算 A/B 测试的规模,并通过交错展示两个排序器的结果来比较它们。
本书一直围绕的这家商店是一家家具和家居用品的在线零售商(第 1 章),其搜索框仍以 BM25 作为基线。前面各章的离线工作已使语义检索相对于 BM25 取得了确定的领先优势(第 3 章)。切换本身是一项变更,此后的每一次调整也同样是变更:更廉价的嵌入、混合权重、重排序器。每项变更都会引出同样的四个问题。平均值是否上升,且幅度超过噪声?哪些查询变差了?评判数据能否为新的结果打分?购物者的行为会与评判数据的预测一致吗?本章按这个顺序回答这些问题。商店的 30 条测试查询成为其回归集,即每项变更都要在其上测试的固定查询,而所有选择都基于其 50 条训练查询做出。商店没有点击日志,因此在线部分对购物者进行模拟,并在每个模拟旁说明其假设。
本章计划在测试查询上进行一项比较:语义检索对比 BM25。依照第 3 章,一项计划内的比较保留本书完整的 0.05,因此在这里,差异的 p 值低于 0.05 即视为成立。本章在测试查询上的其他所有分数都属于探索性的。
逐个查询对比两次运行
前两个问题,即平均值是否上升、哪些查询变差了,要在回归集上回答。回归集由固定的查询及其相关性判断组成,每项改动在上线前都要在它上面运行。商店目前使用的排序器是基线,带有改动的排序器是候选排序器。两者各自生成一次运行,即它对集合中每个查询给出的排序结果。仅比较两次运行的平均值是不够的,因为一项改动可能提高平均值,却仍然让一些查询变差。文本检索会议(Text REtrieval Conference,TREC)对搜索系统开展共同评测,并在 2013 年的 Web 评测任务中围绕这一风险设计了一项任务,按查询统计各系统相对于基线的胜负(Collins-Thompson et al., 2014)。因此要对两次运行做差异比较:每个查询一行,列出基线的得分、候选的得分以及变化量。然后统计胜、负和平局的数量,并逐一阅读那些变差的查询。
基线是 BM25,候选是语义检索,两者都在测试查询上用 nDCG@10 评分。本示例用 pandas 构建差异表,其检验来自 lib.significance,该模块由第 3 章手工构建。这些检验是置换检验和配对 t 检验。与它们并列的是自助法区间,其置信水平与阈值相匹配。自助法和置换检验都进行随机重采样,因此 lib.significance 用 rng 为 SciPy 设定随机种子(第 3 章)。
1"""Scores each test query under BM25 and semantic retrieval, then runs the chapter's one planned comparison on the difference. The verdict rests on the 'permutation test p' and 'established' lines, read against 'threshold'."""2import pandas as pd34from lib.lexical import LexicalSearch5from lib.metrics import evaluate6from lib.semantic import SemanticSearch7from lib.significance import compare8from lib.store import load_products, load_queries, qrels910PLANNED_COMPARISONS = 1 # this chapter's tests on the test queries11threshold = 0.05 / PLANNED_COMPARISONS # Bonferroni: one comparison keeps 0.051213products, queries, judgments = load_products(), load_queries("test"), qrels("test")14baseline = evaluate(LexicalSearch(products).run(queries, k=100), judgments) # what the store runs15candidate = evaluate(SemanticSearch().run(queries, k=100), judgments) # the change1617diff = pd.DataFrame({"query": queries["query"], "bm25": baseline["ndcg@10"], "semantic": candidate["ndcg@10"]})18# rounded to 3 decimals, as lib.significance rounds before it counts wins, losses and ties19diff["change"] = (diff["semantic"] - diff["bm25"]).round(3) # one row per test query2021print(f"bm25 ndcg@10: {diff['bm25'].mean():.3f}")22print(f"semantic ndcg@10: {diff['semantic'].mean():.3f}")23result = compare(candidate["ndcg@10"], baseline["ndcg@10"], confidence=1 - threshold) # chapter 3's comparison24print(f"mean change: {result['mean difference']:.3f}")25print(f"threshold: {threshold:g}")26print(f"permutation test p: {result['permutation p']:.3f}")27print(f"paired t-test p: {result['t-test p']:.3f}")28print(f"{(1 - threshold) * 100:g}% interval: {result['ci low']:.3f} to {result['ci high']:.3f}")29print("established:", "yes" if result["permutation p"] < threshold else "no")30print("wins:", (diff["change"] > 0).sum())31print("losses:", (diff["change"] < 0).sum())32print("ties:", (diff["change"] == 0).sum())33print(f"largest loss: {diff['change'].min():.3f}")34print()35print(diff[diff["change"] < 0].sort_values("change").round(3).to_string())
python diff-semantic-against-bm25.pybm25 ndcg@10: 0.725
semantic ndcg@10: 0.830
mean change: 0.105
threshold: 0.05
permutation test p: 0.001
paired t-test p: 0.003
95% interval: 0.055 to 0.185
established: yes
wins: 21
losses: 7
ties: 2
largest loss: -0.092
query bm25 semantic change
query_id
292 decorative white pillow 0.467 0.375 -0.092
293 wood floor protector mat 0.653 0.568 -0.084
482 48 inch kitchen hood 1.000 0.921 -0.079
63 monthly calendar 1.000 0.931 -0.069
285 48" sliding single track , barn door for laundry 0.592 0.555 -0.037
72 fold in half table 1.000 0.965 -0.035
94 kohler whitehaven farmhouse kitchen sink 0.694 0.690 -0.004
语义检索把 nDCG@10 的平均值从 0.725 提高到 0.830,变化为 0.105。对照 0.05 的阈值,置换检验给出 0.001。这一变化是否成立?yes。与之并列,配对 t 检验给出 0.003,95% 区间为 0.055 to 0.185。
从变差的查询中才能学到东西。语义检索在 21 个查询上获胜,在 7 个上落败,在 2 个上打平,最大的一次落败是 -0.092。其中两个落败的查询给出了以英寸为单位的尺寸,即 48 inch kitchen hood 和 48" sliding single track , barn door for laundry:这是两个查询,而不是一种规律。第 6 章会在每个含数字的查询上检验这一规律。
**工作准则:**保留一个由已作判断的查询组成的回归集,绝不在其上做任何调优,并让每个候选都在该集合上与基线对比运行。先列出计划中的比较,并为每项比较报告其置换 p 值、t 检验结果和区间。统计胜、负和平局的数量,并在上线前阅读每一个变差的查询。当这些变差的查询似乎有共同的规律时,先在训练查询上检验它,再据此行动,然后修复这类查询,或把它们交给基线处理。
用 URisk 让损失比收益占更大权重
逐查询差异统计了变差的查询数,但平均变化仍然把一条变差 0.1 的查询和一条变好 0.1 的查询视为相互抵消。商店未必这样看:顾客昨天还能找到某件商品,今天却找不到了,这就是一次退化,别处排序略有改善并不能补偿他。风险敏感评估对损失的计权高于收益。
TREC 2013 Web track的风险敏感度量是 URisk。对每条查询 q,令 Δ(q) 为候选系统的得分减去基线的得分。把变好的查询上的增益加总,减去变差的查询上的损失(损失按 α + 1 倍计),再除以查询数 N:
URisk = (1 / N) × [ sum of Δ(q) over improved queries − (α + 1) × sum of |Δ(q)| over hurt queries ]
该评测将此描述为"将损失的权重定为成功的 α + 1 倍",而 α = 0 时得到的就是平均变化(Collins-Thompson 等,第 4.3 节)。评测概述在损失项中直接使用 Δ(q),而它在变差的查询上为负数。我们使用其绝对值 |Δ(q)|,因为只有这样理解,α = 0 才如概述所说得到平均值。α 是一种选择,而不是事实。我们取 α = 1,因此每次损失按两倍计。
以 BM25 为基线,比较两个候选系统。第一个是完整的语义检索,使用 Snowflake/snowflake-arctic-embed-m-v1.5 生成的每个嵌入的全部 768 个数值。第二个只保留前 128 个数值,即六分之一的内存。这低于该模型的说明卡所记载的 256 维,而且第 6 章在训练查询上否决了 128 维。我们之所以使用它,是因为在测试查询上它以一些查询上的收益换取另一些查询上的损失,而这正是 URisk 的用武之地。它是根据其在测试集上的表现选出的,因此属于探索性比较,而不是预先计划的比较。URisk 只需几行代码,对两个候选系统分别运行:
1"""URisk for full semantic retrieval and for the 128-number version, each against BM25, with every loss counted alpha + 1 times. The prose's point rests on the 'semantic 128 urisk with alpha 1' line, read against its mean change."""2from lib.lexical import LexicalSearch3from lib.metrics import evaluate4from lib.semantic import SemanticSearch5from lib.significance import compare6from lib.store import load_products, load_queries, qrels78ALPHA = 1 # each loss counts alpha + 1 times: here, double91011def urisk(change, alpha):12 """Mean change per query, with every loss counted alpha + 1 times (TREC 2013 Web track)."""13 # gains are the positive changes; losses are the negative ones, made positive14 wins, losses = change.clip(lower=0).sum(), -change.clip(upper=0).sum()15 return (wins - (alpha + 1) * losses) / len(change)161718products, queries, judgments = load_products(), load_queries("test"), qrels("test")19baseline = evaluate(LexicalSearch(products).run(queries, k=100), judgments)20candidates = {"semantic": SemanticSearch(), "semantic 128": SemanticSearch(dims=128)} # all 768 numbers, or the first 1282122print(f"bm25 judged@10: {baseline['judged@10'].mean():.3f}")23evaluated = {name: evaluate(search.run(queries, k=100), judgments) for name, search in candidates.items()}24for name, scores in evaluated.items():25 # rounded to 3 decimals, so the wins and losses count the printed changes26 change = (scores["ndcg@10"] - baseline["ndcg@10"]).round(3)27 print(f"{name} judged@10: {scores['judged@10'].mean():.3f}")28 print(f"{name} mean change: {change.mean():.3f}")29 print(f"{name} wins: {(change > 0).sum()}")30 print(f"{name} losses: {(change < 0).sum()}")31 print(f"{name} largest loss: {change.min():.3f}")32 print(f"{name} urisk with alpha {ALPHA}: {urisk(change, ALPHA):.3f}")3334# a demonstration, chosen for how it behaves on test: its tests describe it, and it is never established35result = compare(evaluated["semantic 128"]["ndcg@10"], baseline["ndcg@10"], confidence=0.95)36print(f"semantic 128 permutation test p: {result['permutation p']:.3f}")37print(f"semantic 128 paired t-test p: {result['t-test p']:.3f}")38print(f"semantic 128 interval: {result['ci low']:.3f} to {result['ci high']:.3f}")
python urisk-of-two-candidates.pybm25 judged@10: 0.927
semantic judged@10: 0.990
semantic mean change: 0.105
semantic wins: 21
semantic losses: 7
semantic largest loss: -0.092
semantic urisk with alpha 1: 0.092
semantic 128 judged@10: 0.920
semantic 128 mean change: 0.017
semantic 128 wins: 15
semantic 128 losses: 13
semantic 128 largest loss: -0.253
semantic 128 urisk with alpha 1: -0.033
semantic 128 permutation test p: 0.592
semantic 128 paired t-test p: 0.597
semantic 128 interval: -0.036 to 0.088
完整语义检索的平均变化为 0.105,其 URisk 为 0.092。它的 7 次损失都很小,最大为 -0.092,所以加倍计入后代价不大。
较廉价的排序器平均变化为 0.017,但损失按两倍计后,其 URisk 为 -0.033。它在 15 条查询上获胜,在 13 条上失败,最大损失为 -0.253。其平均值掩盖了这一权衡:一些查询的结果变好,代价是另一些查询的结果变差。
它的置换检验给出 0.592,配对 t 检验给出 0.597,95% 区间为 -0.036 to 0.088。
它的部分损失可能来自无人判定过的商品,因为其 judged@10 为 0.920,而 BM25 为 0.927,完整语义检索为 0.990。下一节对此进行检查。
URisk 只描述风险,并不检验风险。Dinçer 等人定义了一个相关的度量 TRisk,它"是 t 统计量的线性变换",因此可以像配对 t 检验那样进行检验(Dinçer 等,2014)。
工作准则: 对每个候选系统,在报告平均变化的同时报告 α = 1 时的 URisk。无论平均值如何,都不要上线 URisk 为负的候选系统,并通过查看其损失来找出原因。当失败代价高昂时,提高 α,并另行报告商店最高频查询上的 URisk。只有当一条查询上的损失并不比另一条查询上的收益更糟时,才使用 α = 0。
评判新排序器带出的商品
新排序器的失败案例,例如URisk 示例中那个较廉价排序器的失败,可能源于评判中的缺口,因为评判只覆盖了早期系统找到的商品。WANDS 是该商店评判数据的来源,它只评判了早期系统找到的商品(第 2 章)。与这些系统不同的排序器会带出无人评判过的商品,而指标会给每个这样的商品打上等级 0,因此它的得分可能偏低,也可能恰好正确。在有人对这些商品做出评判之前,你无法分辨是哪一种情况。WANDS 的作者提醒过,这类缺口会削弱评估:"incompleteness in Product Search dataset does negatively impact the discriminative power of the evaluation"(Chen et al., 2022)。
第 2 章用 judged@10 统计未评判的商品。在检验一项改动时,还有一步很有帮助:敏感性检查。它在两种情形下对两次运行重新打分,每种情形各偏向一方。在一种情形中,基线前 10 名里每个未评判的商品都算作完全相关(exact);在另一种情形中,候选方前 10 名里每个未评判的商品都算作完全相关。在每种情形下,两次运行都使用相同的评判来打分,因为 nDCG 的理想排序是由该查询的全部评判构建的:一个被评判为完全相关的商品会抬高理想排序,从而降低所有没有展示它的运行的得分。这两种情形并不是边界,因为真实的评判可能是两者的混合。如果领先者在两种情形之间发生变化,说明评判尚不能做出裁决。
URisk 示例中较廉价的语义检索只保留每个嵌入 768 个数值中的 128 个,它在商店的测试查询中有 13 个输给了 BM25。在从这些失败中读出任何结论之前,先运行这项检查:
1"""Mean nDCG@10 for BM25, the cheaper ranker and full semantic retrieval, when each run's unjudged top-10 products are judged exact in turn. The prose's point rests on the 'leader changes' line for semantic 128 against BM25, read against the scores printed before it."""2import numpy as np34from lib.lexical import LexicalSearch5from lib.metrics import ndcg_at_k6from lib.semantic import SemanticSearch7from lib.store import load_products, load_queries, qrels89products, queries, judgments = load_products(), load_queries("test"), qrels("test")10runs = {"bm25": LexicalSearch(products).run(queries, k=100),11 "semantic 128": SemanticSearch(dims=128).run(queries, k=100),12 "semantic": SemanticSearch().run(queries, k=100)}131415def unjudged(run):16 """(query_id, product_id) pairs in a run's top 10 that nobody judged."""17 return {(query_id, product_id) for query_id in judgments for product_id in run[query_id][:10]18 if product_id not in judgments[query_id]}192021def score(run, exact=()):22 """Mean nDCG@10 with the given unjudged pairs judged exact, for every run alike: the ideal moves too."""23 # a copy of the judgments, so each scenario changes only its own exact pairs24 scenario = {query_id: dict(grades) for query_id, grades in judgments.items()}25 # grade 2 is exact, the top grade26 for query_id, product_id in exact:27 scenario[query_id][product_id] = 228 return np.mean([ndcg_at_k(run[query_id], scenario[query_id], 10) for query_id in scenario])293031for name in ["bm25", "semantic 128"]:32 print(f"unjudged pairs in the top 10, {name}: {len(unjudged(runs[name]))}")33# the empty set is the judgments as they are34scenarios = {"as judged": set(), **{f"{name}'s unjudged exact": unjudged(runs[name]) for name in runs}}35scores = {(scenario, name): score(run, exact) for scenario, exact in scenarios.items() for name, run in runs.items()}36for scenario, name in [("as judged", "bm25"), ("as judged", "semantic 128"), ("bm25's unjudged exact", "bm25"),37 ("bm25's unjudged exact", "semantic 128"), ("bm25's unjudged exact", "semantic"),38 ("semantic 128's unjudged exact", "bm25"), ("semantic 128's unjudged exact", "semantic 128")]:39 print(f"{scenario}, {name}: {scores[scenario, name]:.3f}")40for candidate in ["semantic 128", "semantic"]:41 # a set holding both True and False means the leader changes between scenarios42 leaders = {scores[s, candidate] > scores[s, "bm25"] for s in scenarios}43 print(f"leader changes, {candidate} against bm25:", "yes" if len(leaders) > 1 else "no")44# | is the union: pairs either run shows, each counted once45print("pairs to judge, semantic 128 against bm25:", len(unjudged(runs["bm25"]) | unjudged(runs["semantic 128"])))
python unjudged-score-range.pyunjudged pairs in the top 10, bm25: 22
unjudged pairs in the top 10, semantic 128: 24
as judged, bm25: 0.725
as judged, semantic 128: 0.742
bm25's unjudged exact, bm25: 0.788
bm25's unjudged exact, semantic 128: 0.738
bm25's unjudged exact, semantic: 0.829
semantic 128's unjudged exact, bm25: 0.718
semantic 128's unjudged exact, semantic 128: 0.807
leader changes, semantic 128 against bm25: yes
leader changes, semantic against bm25: no
pairs to judge, semantic 128 against bm25: 46
领先者是否发生变化?yes。因此,评判目前还不能确定这次比较的结果。按已有评判计算,较廉价的排序器领先,为 0.742 对 0.725。
如果 BM25 的未评判商品都是完全相关的,BM25 将领先,为 0.788 对 0.738。如果较廉价排序器的未评判商品都是完全相关的,它将领先,为 0.807 对 0.718。
BM25 的前 10 名中有 22 个未评判的配对,较廉价排序器的前 10 名中有 24 个。这次比较要等待 46 个评判,与上线错误排序器的代价相比,这笔开销很小。
不要猜测评判者会给出什么结果。在一个旧的 TREC 集合上,Voorhees 等人对神经排序与 BM25 运行产生的新文档池做了评判,发现绝大多数新文档并不相关(Voorhees et al., 2022)。但那个集合是从许多系统中做了很深的池化,而 WANDS 的构建方式不同。人工评判是基准。大语言模型(LLM)的评判成本更低。第 4 章用它们填充商店中未评判的配对,作为一种诊断手段,并且在填充后的等级改变了某个结论或两个排序器的先后顺序时,由人来评判这些配对。Thomas 等人指出了一个在测试改动时值得注意的风险:由 LLM 生成的标签可能偏向本身使用 LLM 的排序器(Thomas et al., 2024)。
工作准则: 对每个候选方,计算两次运行的 judged@10,并在两种情形下重新打分。当领先者发生变化时,先评判未评判的配对再做比较,并把新的评判永久加入回归集,使其覆盖你测试过的每一个排序器。当领先者在两种情形下都保持不变时,未评判的商品不太可能扭转这次比较。
这部分尚未翻译,因此以英文显示。
Keep the test queries out of every choice
The test queries in the regression set are held out, and held-out queries give an honest score only if nothing was chosen by looking at them. Training on test queries is the obvious mistake. Choosing on them is the subtle one: try several settings, keep the one that scores best on them, and report that score. The winner is partly the best setting and partly the luckiest on those particular queries, so its score is too high. Dwork et al. state the risk: "Reusing a holdout set adaptively multiple times can easily lead to overfitting to the holdout set itself" (Dwork et al., 2015). The damage can be smaller than feared. When ImageNet's test set was rebuilt, classifiers lost accuracy, but Recht et al.'s results suggest that adaptivity doesn't cause the drops: the new images were harder (Recht et al., 2019). In our judgement, a set of 30 queries leaves little room, and every look counts.
The change is a blend of BM25 and semantic retrieval with one weight, the share given to semantic retrieval. Chapter 7 calls it α, a different α from URisk's, and tunes it on the train queries with weighted reciprocal rank fusion, over 0, 0.1, … 1. Choose the semantic weight both ways, the right way on train queries and the wrong way on test queries, and report each choice on the test queries. These test scores show the bias; they are exploratory, not a planned comparison. Then measure how much choosing on the reported queries inflates their score, without touching the test queries again: split the train queries at random, 1,000 times, into 30 to choose on and 20 to report on, and compare the 20 reported queries' score when the weight is chosen on the 30 with their score when it is chosen on the 20 themselves.
1"""Chooses the blend's semantic weight on train queries and on test queries, then measures how much choosing on the queries you report inflates their score. The prose's point rests on the 'average inflation' line, read against the two 'test ndcg@10' lines."""2import numpy as np34from lib.blend import best_alpha, scores_by_alpha, weighted_rrf5from lib.lexical import LexicalSearch6from lib.semantic import SemanticSearch7from lib.store import load_products, load_queries, qrels89queries, judgments = load_queries(), qrels() # all 80 queries, train and test10bm25 = LexicalSearch(load_products()).run(queries, k=100)11semantic = SemanticSearch().run(queries, k=100)12per_query = scores_by_alpha(lambda weight: weighted_rrf(bm25, semantic, weight), judgments) # nDCG@10, a column per weight13train = per_query.loc[load_queries("train").index]14test = per_query.loc[load_queries("test").index]1516# best_alpha breaks ties toward the smaller weight17chosen_on_train, chosen_on_test = best_alpha(train), best_alpha(test)18print("settings tried:", per_query.shape[1])19print(f"semantic alone, test ndcg@10: {test[1.0].mean():.3f}")20print("semantic weight chosen on train:", chosen_on_train)21print(f"test ndcg@10, weight chosen on train: {test[chosen_on_train].mean():.3f}")22print("semantic weight chosen on test:", chosen_on_test)23print(f"test ndcg@10, weight chosen on test: {test[chosen_on_test].mean():.3f}")2425# How much does choosing on the queries you report inflate the score? Split the train queries26# at random, 30 to choose on and 20 to report on, and compare the two ways of choosing.27# a fixed seed, so the random splits are the same on every run28rng = np.random.default_rng(2026)29scores = train.to_numpy()30inflation = []31for _ in range(1000):32 order = rng.permutation(len(scores))33 # the first 30 of the shuffled queries choose the weight; the other 20 report it34 choose, report = scores[order[:30]].mean(axis=0), scores[order[30:]].mean(axis=0)35 # argmax takes the first of equal means, the smaller weight, as best_alpha does36 inflation.append(report.max() - report[choose.argmax()])37print(f"average inflation, 20 reported queries: {np.mean(inflation):.3f}")
python tuning-on-test-inflates.pysettings tried: 11
semantic alone, test ndcg@10: 0.830
semantic weight chosen on train: 0.6
test ndcg@10, weight chosen on train: 0.825
semantic weight chosen on test: 0.8
test ndcg@10, weight chosen on test: 0.835
average inflation, 20 reported queries: 0.007
Choosing the weight on the test queries flatters the blend. Chosen there, the semantic weight is 0.8, and the blend scores 0.835.
Chosen on train, it is 0.6, and the blend scores 0.825 on the test queries, against 0.830 for semantic retrieval alone. Only that train-chosen score fairly estimates how the blend does on queries it wasn't chosen on. Chapter 7 tests the train-chosen blend against semantic retrieval and finds no improvement.
The random splits put a number on the bias. On report sets of 20 queries, choosing on the queries you report adds 0.007 to nDCG@10 on average, with 11 settings to choose from. That gain appears in the report and nowhere else. Each extra setting you try is another chance for a lucky winner.
Working rule: Choose every setting on train queries, including which candidate to take forward. Look at the test queries once per decision, and log each look: what you compared and what you chose. When a test set has guided many decisions, move it to the train side and judge a fresh set of test queries.
运行 A/B 测试
离线测试用判断来衡量排序结果,但商店的收入来自购物者。A/B 测试,即受控实验,把用户随机分配到现有系统或变更后的系统,并比较一个在测试开始前选定的指标(Kohavi et al., 2009)。现有系统是对照组,变更后的系统是实验组。
有三项检查可以让结果值得信赖。第一,A/A 测试中两组使用相同的系统,在 95% 置信水平下,它拒绝原假设的概率应约为 5%(Kohavi et al., 2009)。远高于 5% 说明实验设置有问题。第二,只在计划的样本量上读取一次结果。一旦结果看起来显著就停止测试,这称为偷看(peeking),会使第一类错误率大幅上升(Kohavi et al., 2022)。第三,检查各组的规模是否与计划一致。样本比例失衡(SRM)意味着某一组的用户丢失,或用户被不均匀地加入该组,而且“在不了解根本原因的情况下忽视 SRM,可能导致一个糟糕的产品修改看起来很好”(Fabijan et al., 2019)。
对于转化指标,即购买的购物者所占的比例,测试比较两个比例。设每组有 n 名购物者,转化率分别为 p_A 和 p_B,p̄ 为二者的平均值:
z = (p_B − p_A) / sqrt( p̄ × (1 − p̄) × 2 / n ), p-value = 2 × P(Z > |z|)
在明确的假设下模拟 2,000 次 A/A 测试,以观察偷看的影响:每组每天 2,000 名购物者,5% 的人购买,每次测试计划进行 14 天。对每次测试,既在计划结束时读取结果,也每天读取一次,并在 p 值首次低于 0.05 时停止。然后检查一次分流是否存在失衡。
1"""Simulates A/A tests, where both groups get the same search, read at the planned end and every day, and checks one split for a mismatch. The prose's point rests on the 'significant at the planned end' and 'significant on some day' lines, and on the sample ratio mismatch line."""2import numpy as np3from scipy import stats45# Assumptions for the simulation, not the store's data: 2,000 shoppers a day in each group,6# 5% of them buy, and the test is planned to run 14 days. Both groups get the same search (an A/A test).7TESTS, DAYS, SHOPPERS_PER_DAY, CONVERSION = 2000, 14, 2000, 0.0589# a fixed seed, so the simulated tests are the same on every run10rng = np.random.default_rng(2026)11buyers = rng.binomial(SHOPPERS_PER_DAY, CONVERSION, size=(TESTS, DAYS, 2)).cumsum(axis=1) # running totals12shoppers = SHOPPERS_PER_DAY * np.arange(1, DAYS + 1)[None, :] # per group, by day13rate_a, rate_b = buyers[..., 0] / shoppers, buyers[..., 1] / shoppers14pooled = (rate_a + rate_b) / 215# two-proportion z for each day, using the pooled rate of the two groups16z = (rate_b - rate_a) / np.sqrt(pooled * (1 - pooled) * 2 / shoppers)17# two-sided: a gap in either direction counts18p_value = 2 * stats.norm.sf(np.abs(z))1920print(f"simulated A/A tests: {TESTS:,}")21# the last day is the planned end, day 1422print(f"significant at the planned end: {(p_value[:, -1] < 0.05).mean():.3f}")23print(f"significant on some day, checked daily: {(p_value < 0.05).any(axis=1).mean():.3f}")2425# A sample ratio mismatch: the split was planned 50/50, but the groups came out unequal.26control, treatment = 28_000, 27_10027# the default expected share is one half, as in a 50/50 split28p = stats.binomtest(treatment, control + treatment).pvalue29print(f"sample ratio mismatch p: {p:.3f}" if p >= 0.001 else "sample ratio mismatch p: below 0.001")
python aa-tests-with-peeking.pysimulated A/A tests: 2,000
significant at the planned end: 0.053
significant on some day, checked daily: 0.231
sample ratio mismatch p: below 0.001
每天检查会使多得多的 A/A 测试看起来显著。在计划结束时读取,显著的比例为 0.053,接近阈值所承诺的 0.05。每天检查时,在某一天看起来显著的比例升至 0.231,而两组的搜索其实都没有变化。对于计划为 50/50、实际为 28,000 比 27,100 的分流,对 55,100 名购物者中的 27,100 人做针对 1/2 的二项检验,得到的 p 值为 below 0.001:偶然几乎不会产生这样大的差距。
p 值常被误读:0.05 并不表示结果有 5% 的概率是假阳性;Kohavi et al.(2022)称这是“一个非常常见的误解”。它是在变更没有任何作用时,出现至少如此极端的数据的概率。
工作准则: 测试之前,先选定主要指标和不得变差的护栏指标,并确定测试规模(关于功效)。先在同一套设置上运行一次 A/A 测试。在读取任何结果之前先检查样本比例,并且只在计划的样本量上读取一次结果。如果必须在测试进行时观察它,就使用为持续监控而设计的序贯方法,例如始终有效的 p 值(Johari et al., 2015)。
用功效确定测试规模
要在计划的规模下一次性读取A/B 测试的结果,就得先根据测试的功效(power)来规划这一规模。功效是指测试检测出给定大小的真实变化的概率。功效不足的测试会漏掉真实的变化;即便结果显著,也往往会夸大效应:"a statistically significant finding with low power is likely to highly exaggerate the size of the effect"(Kohavi et al., 2022)。因此,应在测试开始之前,根据两个数字来确定规模:指标在用户之间的变化程度,以及值得检测的最小变化。方差越大,所需用户越多。变化越小,所需用户要多得多,因为变化量是以平方的形式进入公式的。
对于两个组、95% 的置信度和 80% 的功效,Kohavi 等人给出了一条经验法则,其中 σ² 是指标的方差,Δ 是要检测的变化(Kohavi et al., 2009):
users per group ≈ 16 × σ² / Δ²
其中的 16 来自正态分布:2 × (97.5% 对应的 z 值 + 80% 对应的 z 值)²,约等于 2 × (1.96 + 0.84)²。对于 90% 的功效,他们把 16 换成 21。SciPy 计算这个系数,代码则用他们的两个计算示例来检验这条法则:每用户收入均值为 3.75 美元、标准差为 30 美元;转化率为 5%;两者都要检测 5% 的相对变化。商店的数据来自 WANDS,没有销售数字,所以代码假设商店的转化率同样是 5%。
1"""Kohavi et al.'s rule of thumb for users per group, its coefficient, and the users the store would need for smaller changes, assuming its 5% conversion rate. The prose's point rests on the two 'users per group for a change' lines, read against the coefficient line."""2from scipy import stats34SIGNIFICANCE, POWER = 0.05, 0.805z = stats.norm.ppf(1 - SIGNIFICANCE / 2) + stats.norm.ppf(POWER) # 1.96 + 0.846print(f"coefficient for two groups: {2 * z ** 2:.1f}") # rounds to the 16 of the rule of thumb789def users_per_group(variance, change):10 """Kohavi et al.'s rule of thumb: about 16 * variance / change**2 users in each group."""11 return 16 * variance / change ** 2121314# Their worked examples: revenue per user (mean $3.75, standard deviation $30), and a 5% conversion rate,15# each with a 5% relative change to detect.16print(f"users per group, revenue: {users_per_group(30 ** 2, 0.05 * 3.75):,.0f}")17# a conversion rate's variance is p times (1 - p), here with p = 5%; the change is 5% of that rate18print(f"users per group, conversion: {users_per_group(0.05 * 0.95, 0.05 * 0.05):,.0f}")1920# The store, assuming a 5% conversion rate like theirs: smaller changes need far more users.21for relative in [0.02, 0.01]:22 # the change in conversion, as an absolute rate: a relative change of this size in the 5% rate23 print(f"users per group for a {relative:.0%} change: {users_per_group(0.05 * 0.95, 0.05 * relative):,.0f}")
python sample-size-from-power.pycoefficient for two groups: 15.7
users per group, revenue: 409,600
users per group, conversion: 121,600
users per group for a 2% change: 760,000
users per group for a 1% change: 3,040,000
变化减半,所需用户就变成四倍:在商店假设的 5% 转化率下,2% 的变化每组需要 760,000 个用户,1% 的变化则需要 3,040,000 个。
系数为 15.7,收入示例和转化率示例每组分别需要 409,600 和 121,600 个用户,与论文给出的超过 409,000 和不足 122,000 相符。
同样的算法可以说明回归集能够检测到什么。两个排序器对同一批查询打分,所以测试处理的是一个由逐查询差值构成的样本,而不是两个组,系数因此减半,变为 7.8。σ 取两个相近排序器之间这些差值的离散程度:一个是语义检索,另一个是把它与 BM25 融合的混合运行。
1"""The smallest change the regression set detects with 80% power, from the spread of per-query changes, and a simulation that checks the power formula. The prose's point rests on the 'smallest change 30 queries detect' line and the simulated power lines."""2import numpy as np3from scipy import stats45from lib.fusion import reciprocal_rank_fusion6from lib.lexical import LexicalSearch7from lib.metrics import evaluate8from lib.semantic import SemanticSearch9from lib.store import load_products, load_queries, qrels1011SIGNIFICANCE, POWER = 0.05, 0.8012queries, judgments = load_queries("test"), qrels("test")13bm25 = LexicalSearch(load_products()).run(queries, k=100)14semantic = SemanticSearch().run(queries, k=100)15hybrid = reciprocal_rank_fusion([bm25, semantic])16change = evaluate(hybrid, judgments)["ndcg@10"] - evaluate(semantic, judgments)["ndcg@10"]17spread = change.std() # of the per-query differences1819z = stats.norm.ppf(1 - SIGNIFICANCE / 2) + stats.norm.ppf(POWER)20print(f"spread of per-query changes: {spread:.3f}")21print(f"coefficient for paired queries: {z ** 2:.1f}") # one sample of differences, so half of 1622# z standard errors of the mean change: the smallest change the test can detect23detectable = z * spread / np.sqrt(len(change))24print(f"smallest change 30 queries detect: {detectable:.3f}")25# n = z squared times spread squared, over the change squared, solved for n26print(f"queries needed for a change of 0.01: {z ** 2 * spread ** 2 / 0.01 ** 2:.0f}")2728# Check the arithmetic by simulation: draw 30 normally distributed changes with that mean and spread,29# 10,000 times, and test each. The real differences aren't normal, so this checks the formula, not the data.30rng = np.random.default_rng(2026)31# ttest_1samp tests each draw against popmean 0, the no-change null32power = stats.power(stats.ttest_1samp, lambda size: rng.normal(detectable, spread, size=size), (len(change),),33 significance=SIGNIFICANCE, kwargs={"popmean": 0})34# .item() takes the single value out of the result, which holds one simulated power35print(f"simulated power at that change: {power.power.item():.3f}")36print("simulated power below the planned 0.8:", "yes" if power.power.item() < POWER else "no")
python queries-needed-for-power.pyspread of per-query changes: 0.062
coefficient for paired queries: 7.8
smallest change 30 queries detect: 0.032
queries needed for a change of 0.01: 300
simulated power at that change: 0.779
simulated power below the planned 0.8: yes
逐查询变化的离散程度为 0.062,因此在有 30 个测试查询的情况下,回归集能以 80% 的功效检测到 nDCG@10 上 0.032 的变化。若要检测 0.01 的变化,则需要 300 个查询。
scipy.stats.power 通过模拟来检验这一算法,模拟中抽取服从正态分布的差值:在该变化下,配对 t 检验的模拟功效为 0.779。这是否低于计划的 0.8?yes,因为公式使用的是正态分布,而针对测试查询的 t 检验使用的是更宽的 t 分布。真实的差值并不服从正态分布,所以模拟检验的是公式,而不是数据。增加查询时,应多加查询,并对每个查询做浅层评判:Webber 等人发现,在评判工作量相同的情况下,大量浅层评判的查询比少量深度评判的查询带来更大的统计功效(Webber et al., 2008)。
工作准则: 每个 A/B 测试开始之前都要先确定规模。如果所需用户数超过商店几周内能够接触到的数量,就测试更大的变化,使用方差更小的指标,或者用每个用户在测试之前的数据来调整指标。Deng 等人报告说,在 Bing 的实验中,这"can reduce variance by about 50%"(Deng et al., 2013);能节省多少,取决于用户过去的行为对该指标的预测程度。离线时,计算回归集能够检测到的最小变化,当你的改动更小时就增加查询。要事先确定数量:不断增加查询,直到功效估计表明数量足够为止,这种做法会使你偏向于发现显著性(Webber et al.)。
用团队选秀法交错两个排序
A/B 测试向每一组展示不同的排序,因此购物者之间的差异会带来噪声。交错(interleaving)则向每位购物者展示一个混合了两个排序器结果的列表。Joachims 设计这种方法时,要求“合并后的排序对两个搜索引擎给出(几乎)相等的呈现偏差”(Joachims, 2003):两个排序器的商品都不会系统性地排得更靠前。商品获得更多点击的排序器赢得该次搜索。交错回答的问题比 A/B 测试更窄:它回答用户更喜欢两个排序中的哪一个,而不给出任何一方的绝对数值(Hofmann et al., 2016)。
团队选秀法(team draft)像为一场比赛挑选队员那样交错两个排序器的列表(Radlinski et al., 2008,算法 2)。已选人数较少的队伍先选;双方人数相同时,由抛硬币决定。每一次挑选的都是该队排名最高且尚未展示的商品。一次点击记在挑选该商品的队伍名下。Radlinski 等人证明,在团队选秀法下,“随机点击的用户在期望意义上总会为 A 和 B 产生同样多的偏好”,而较早的方法无法保证这一点。
针对测试查询 clear shower curtain,交错商店的语义检索和 BM25 的结果。团队选秀法只需几行代码,抛硬币使用带种子的生成器,代码还会检查 lib.experiments 返回的列表是否与之相同:
1"""Team draft over one query's semantic and BM25 rankings: the interleaved list, and the team that picked each place. The prose's point rests on the BM25 pick at the bottom of the list, and on the 'same list' line that checks lib.experiments."""2import numpy as np34from lib import experiments5from lib.lexical import LexicalSearch6from lib.semantic import SemanticSearch7from lib.store import load_products, load_queries, qrels89LABELS = {2: "exact", 1: "partial", 0: "irrelevant"}101112def team_draft(ranking_a, ranking_b, rng, length=10):13 """Like picking teams: the team with fewer picks goes next, a coin flip decides when they're level,14 and each team adds its best product that isn't shown yet."""15 shown, teams = [], []16 while len(shown) < length:17 picks_a, picks_b = teams.count("a"), teams.count("b")18 # the coin is drawn only when the teams are level19 a_picks = picks_a < picks_b or (picks_a == picks_b and rng.random() < 0.5)20 ranking, team = (ranking_a, "a") if a_picks else (ranking_b, "b")21 # the team's best product that isn't shown yet, from its own ranking22 shown.append(next(p for p in ranking if p not in shown))23 teams.append(team)24 return shown, teams252627products, queries, judgments = load_products(), load_queries("test"), qrels("test")[246] # clear shower curtain28semantic = list(SemanticSearch().search(246, k=100).index)29bm25 = list(LexicalSearch(products).search(queries.at[246, "query"], k=100).index)3031shown, teams = team_draft(semantic, bm25, np.random.default_rng(7))32# judgments.get returns None for a product nobody judged, and LABELS.get then gives 'unjudged'33# names are cut at 45 characters to keep the columns aligned34for position, (product_id, team) in enumerate(zip(shown, teams), start=1):35 name = {"a": "semantic", "b": "bm25"}[team]36 print(f"{position:>2} {name:<8} {LABELS.get(judgments.get(product_id), 'unjudged'):<10} {products.at[product_id, 'name'][:45]}")37# lib.experiments holds the same function, which the simulations use38print("same list from lib.experiments:", "yes" if experiments.team_draft(semantic, bm25, np.random.default_rng(7)) == (shown, teams) else "no")
python team-draft-one-query.py 1 bm25 partial rodas vinyl shower curtain
2 semantic exact woolley vinyl solid color single shower curta
3 bm25 exact woolley vinyl single shower curtain
4 semantic partial newburg check 3d vinyl single shower curtain
5 bm25 exact siniard single shower curtain
6 semantic partial 71 '' x71 '' 4pcs shower curtain sets beach t
7 semantic exact almazan lightweight single shower curtain
8 bm25 exact simmon vinyl single shower curtain liner
9 semantic partial vinyl striped shower curtain set
10 bm25 irrelevant gilded pineapple shower curtain hooks
same list from lib.experiments: yes
团队选秀法总是让每个排序器在十个位置的列表中占 5 个位置;抛硬币决定是哪几个位置。两个排序器都排得靠前的商品只出现一次,记在最先选中它的队伍名下,所以同一商品不会让任何一个排序器得到两次功劳。对这个查询,BM25 所排的一个浴帘挂钩(第 1 章)会进入列表,位于最底部,是 BM25 的最后一次挑选。购物者对语义检索所选商品的点击记给语义检索,对 BM25 所选商品的点击记给 BM25,点击更多的队伍赢得该次搜索。
工作准则: 当你需要知道用户更喜欢哪一个排序器,并希望用比 A/B 测试更少的搜索量得出结论时,使用团队选秀交错在线比较两个排序器。当你需要绝对数值(例如收入),或者改动不在排序本身时,改用 A/B 测试。按我们的推断,交错列表无法显示延迟、页面布局或结果数量不同所产生的影响。
用模拟购物者演练交错测试
WANDS 是本店数据的来源,其中没有点击记录,因此本店无法在真实购物者身上运行团队选秀交错,但可以用模拟购物者来演练。点击模型是一种概率模型,描述用户如何浏览并点击结果列表,既用于构建指标,也用于模拟用户(Chuklin et al., 2015)。级联模型认为,用户从列表顶部开始阅读,并“一看到有价值的文档就离开”(Craswell et al., 2008;参见第 2 章)。标准的模拟遵循这一模型:均匀随机抽取一个查询,沿其列表向下浏览,以取决于等级的概率点击每个商品,点击之后,再以另一个取决于等级的概率离开(Oosterhuis and de Rijke, 2018)。关于交错的研究正是这样检验其方法的,即用带标注的数据模拟点击(Hofmann et al., 2011)。
这些概率都是假设。Oosterhuis 和 de Rijke 的表 2 给出了一个针对五个等级的“导航型”用户,即只寻找一个高度相关结果的用户,我们称之为导航型购物者。我们把不相关、部分相关和完全相关分别对应到该用户的最低、中间和最高等级,与指标的做法一样,将未经评判的商品视为不相关,并让每个查询被抽到的次数相同。这些取值集中放在一处,即 lib.experiments:
| 等级 | 点击 | 点击后停止 |
|---|---|---|
| 完全相关 | 0.95 | 0.9 |
| 部分相关 | 0.5 | 0.5 |
| 不相关 | 0.05 | 0.2 |
手工让一位这样的购物者浏览针对 clear shower curtain 的团队选秀列表:
1"""One simulated navigational shopper walks the team-draft list for clear shower curtain, clicking and leaving by the cascade model. The prose's point rests on the two 'clicks for' lines and the verdict line, and on the 'same clicks' line that checks lib.experiments."""2import numpy as np34from lib.experiments import NAVIGATIONAL, cascade_clicks, team_draft5from lib.lexical import LexicalSearch6from lib.semantic import SemanticSearch7from lib.store import load_products, load_queries, qrels89LABELS = {2: "exact", 1: "partial", 0: "irrelevant"}10products, queries, judgments = load_products(), load_queries("test"), qrels("test")[246] # clear shower curtain11semantic = list(SemanticSearch().search(246, k=100).index)12bm25 = list(LexicalSearch(products).search(queries.at[246, "query"], k=100).index)13shown, teams = team_draft(semantic, bm25, np.random.default_rng(7)) # the team-draft example's list14team_names = {"a": "semantic", "b": "bm25"}1516# the same seed as the check at the end, so both draw the same clicks17rng = np.random.default_rng(2026)18clicks = []19for position, product_id in enumerate(shown):20 grade = judgments.get(product_id, 0) # unjudged counts as 021 # click and stop probabilities for the grade: 0 irrelevant, 1 partial, 2 exact22 clicked = rng.random() < NAVIGATIONAL["click"][grade]23 # the second draw happens only after a click, the order cascade_clicks uses24 leaves = clicked and rng.random() < NAVIGATIONAL["stop"][grade] # only a click can end the visit25 label = LABELS.get(judgments.get(product_id), "unjudged")26 print(f"{position + 1:>2} {team_names[teams[position]]:<8} {label:<10} {'clicks' if clicked else '':<6} {'leaves' if leaves else ''}")27 if clicked:28 clicks.append(position)29 if leaves:30 break3132credited = [team_names[teams[position]] for position in clicks] # each click counts for the team that picked it33for_semantic, for_bm25 = credited.count("semantic"), credited.count("bm25")34print("clicks for semantic:", for_semantic)35print("clicks for bm25:", for_bm25)36print("this search goes to:", "semantic" if for_semantic > for_bm25 else "bm25" if for_bm25 > for_semantic else "neither, a tie")37print("same clicks from lib.experiments:", "yes" if cascade_clicks(shown, judgments, np.random.default_rng(2026)) == clicks else "no")
python one-simulated-shopper.py 1 bm25 partial clicks
2 semantic exact clicks leaves
clicks for semantic: 1
clicks for bm25: 1
this search goes to: neither, a tie
same clicks from lib.experiments: yes
该购物者的点击数对语义检索为 1,对 BM25 为 1,所以这次搜索的结果是 neither, a tie。我们用固定的随机种子模拟 10,000 次交错搜索,比较两组:语义检索对 BM25,以及混合运行对语义检索。符号检验用来判断胜场是否比负场多出的数量超过偶然水平;Radlinski et al. 使用“二项符号检验”,并将平局排除在外。分别在 100、1,000 和 10,000 次搜索时读取结论:
1"""Sign tests on simulated interleaved searches, at 100, 1,000 and 10,000 searches, for two pairs of rankers and a third pair with BM25's ties reversed. The prose's point rests on the 'verdict changes' line and the 100-search lines for the hybrid against semantic retrieval."""2import numpy as np3from scipy import stats45from lib.experiments import NAVIGATIONAL, simulate_interleaving6from lib.fusion import reciprocal_rank_fusion7from lib.lexical import LexicalSearch8from lib.semantic import SemanticSearch9from lib.store import load_products, load_queries, qrels1011for grade, label in [(2, "exact"), (1, "partial"), (0, "irrelevant")]: # the assumptions, from lib.experiments12 print(f"{label} click probability: {NAVIGATIONAL['click'][grade]}")13 print(f"{label} stop probability: {NAVIGATIONAL['stop'][grade]}")1415# lexsort's last key sorts first: the score, highest first, then the product_id, highest first among ties16def ties_reversed(scores): # BM25's ranking, with equal scores in descending product_id order instead17 return scores.index[np.lexsort((-scores.index.to_numpy(), -scores.to_numpy()))].tolist()181920queries, judgments = load_queries("test"), qrels("test")21lexical = LexicalSearch(load_products())22bm25 = lexical.run(queries, k=100) # equal scores by product_id23semantic = SemanticSearch().run(queries, k=100)24hybrid = reciprocal_rank_fusion([bm25, semantic])25bm25_reversed = {q: ties_reversed(lexical.search(query, k=100)) for q, query in queries["query"].items()}2627pairs = {"semantic against bm25": (semantic, bm25), "hybrid against semantic": (hybrid, semantic),28 "hybrid with bm25 ties reversed against semantic": (reciprocal_rank_fusion([bm25_reversed, semantic]), semantic)}29below = {}30for name, (run_a, run_b) in pairs.items():31 # a fresh seed for each pair, so every pair sees the same 10,000 queries32 outcomes = simulate_interleaving(run_a, run_b, judgments, 10_000, np.random.default_rng(2026))33 for searches in [100, 1_000, 10_000]:34 # the first 100, 1,000 or 10,000 of the same searches, so the three verdicts share data35 wins, losses = (outcomes[:searches] == 1).sum(), (outcomes[:searches] == -1).sum()36 p = stats.binomtest(wins, wins + losses).pvalue # sign test; ties left out37 below[name, searches] = p < 0.0538 shown = f"{p:.3f}" if p >= 0.001 else "below 0.001"39 print(f"{name}, {searches:,} searches: {wins} wins, {losses} losses, p {shown}")40changes = below["hybrid against semantic", 100] != below["hybrid with bm25 ties reversed against semantic", 100]41print("hybrid against semantic, verdict at 100 searches changes with bm25 ties reversed:", "yes" if changes else "no")
python simulated-interleaving-verdict.pyexact click probability: 0.95
exact stop probability: 0.9
partial click probability: 0.5
partial stop probability: 0.5
irrelevant click probability: 0.05
irrelevant stop probability: 0.2
semantic against bm25, 100 searches: 51 wins, 36 losses, p 0.133
semantic against bm25, 1,000 searches: 519 wins, 347 losses, p below 0.001
semantic against bm25, 10,000 searches: 5222 wins, 3552 losses, p below 0.001
hybrid against semantic, 100 searches: 32 wins, 51 losses, p 0.048
hybrid against semantic, 1,000 searches: 413 wins, 447 losses, p 0.260
hybrid against semantic, 10,000 searches: 4180 wins, 4593 losses, p below 0.001
hybrid with bm25 ties reversed against semantic, 100 searches: 35 wins, 48 losses, p 0.187
hybrid with bm25 ties reversed against semantic, 1,000 searches: 410 wins, 463 losses, p 0.078
hybrid with bm25 ties reversed against semantic, 10,000 searches: 4174 wins, 4560 losses, p below 0.001
hybrid against semantic, verdict at 100 searches changes with bm25 ties reversed: yes
小样本会剧烈波动。语义检索对 BM25 在离线时差距很大,但结论在 100 次搜索时为 51 wins, 36 losses, p 0.133,在 1,000 次时为 519 wins, 347 losses, p below 0.001。混合对语义检索,在 100 次搜索时为 32 wins, 51 losses, p 0.048,在 1,000 次时为 413 wins, 447 losses, p 0.260,这正是要在检验开始前就固定搜索次数的原因。
在 10,000 次搜索时,混合的结论为 4180 wins, 4593 losses, p below 0.001。这并不推翻第 3 章“没有确立差异”的发现:在同样的 30 个查询上模拟越来越多的点击,符号检验衡量的只是点击噪声,因此任何固定的差距最终都会变得显著。
混合在 100 次搜索时的结论还取决于一个不携带任何信息的细节。混合融合了 BM25,而 lib.lexical 按 product_id 对 BM25 得分相同的商品排序。若改用相反的顺序,这是同样任意的规则,则相同的种子在 100 次搜索时读出 35 wins, 48 losses, p 0.187。在 0.05 的水平上结论是否改变?yes。反转 BM25 的平局顺序会改变混合的一些列表,于是模拟购物者看到的是其他商品,而第一个改变的列表之后的每一次随机抽取都可能随之变化。一个会被平局规则翻转的结论是运气,而不是关于排序器的发现。
有三点限制使这只是一次演练。点击来自 nDCG@10 所用的同一套等级,所以交错倾向于与 nDCG@10 一致;它没有给排序器增添任何证据。每次搜索都来自同样的 30 个查询,所以符号检验只能说明这些查询的情况。而且真实的购物者并不是这个模型。Deffayet 等人研究了产生点击日志的排序器发生变化时会出现什么情况,发现点击模型对这种变化很敏感(Deffayet et al., 2023)。在三个搜索引擎的真实流量上,只要二者都显著,交错与专家判断就是一致的,并且所需数据“比绝对指标少约 1–2 个数量级”(Chapelle et al., 2012)。依我们的推理,模拟无法显示这一优势,因为其中的购物者行为都一样。
**工作准则:**在模拟中演练交错测试,以检查你的代码和符号检验。对它给出的任何搜索次数,只应按所假设购物者的可靠程度来看待;在对模拟结论作任何解读之前,先把平局按相反方式处理,重新运行一遍。切勿把模拟结论当作关于某个排序器的证据报告。在真实流量上,要事先固定搜索次数,并用符号检验来作决定。
线下与线上结果不一致时,先找原因
模拟购物者按 nDCG@10 所用的同一套等级来点击(click),因此他们的行为往往与该指标一致;真实购物者则未必。线下指标有时能很好地预测线上结果。在一项针对 36 个线下指标的电商研究中,表现最好的指标在判断两个排序模型孰优孰劣时,以某项业务指标衡量,与线上结果的一致率最高达 97%(Wang et al., 2023)。另一个电商团队发现,线下排序指标"可能与线上表现存在显著不一致,甚至可能产生误导"(Huzhang et al., 2020)。这两项都是行业内基于自有流量的研究,外部人员无法复现。当两个系统表现大致相当时,不同指标之间往往也会相互矛盾(Hofmann et al., 2016)。
不一致是有原因的,而且可以检查。相关性判断可能漏掉了线上胜出者所展示的商品,敏感性检查查的正是这一点。改动也可能作用于排序之外,例如速度或版面。又或者,线上指标衡量的并不是成功。在一个学术搜索引擎上,Radlinski et al. (2008)发现,与另外七个绝对使用指标一样,每次查询的点击数在他们所研究的样本量下,并不能可靠地反映检索质量。
商店可以用交错测试演练中的导航型购物者来模拟一次 A/B 测试:以 BM25 作为对照组进行 5,000 次搜索,以语义检索作为实验组进行 5,000 次搜索,全部使用测试查询。度量每次搜索的点击数,以及点击了精确匹配商品的搜索所占的比例。然后换用表 2 中的"完美"用户,即完美购物者,按同样的方式映射。他会阅读全部十条结果,点击每一个精确匹配商品,并且从不提前离开。你预期哪种排序器的每次搜索点击数更多?所有购物者会得出一致的结论吗?
1"""Clicks per search and the share of searches with an exact click, for BM25 and semantic retrieval run as simulated A/B arms, for the navigational and the perfect shopper. The prose's point rests on the two 'reverses between shoppers' lines, read with the means and p-values printed before them."""2import numpy as np3from scipy import stats45from lib.experiments import SHOPPERS, simulate_searches6from lib.lexical import LexicalSearch7from lib.semantic import SemanticSearch8from lib.store import load_products, load_queries, qrels910queries, judgments = load_queries("test"), qrels("test")11arms = {"bm25": LexicalSearch(load_products()).run(queries, k=100), # control12 "semantic": SemanticSearch().run(queries, k=100)} # treatment13metrics = {"clicks per search": "clicks", "share of searches with an exact click": "exact_click"}1415significant_leaders = {label: set() for label in metrics} # rankers ahead at p below 0.0516for shopper_name, shopper in SHOPPERS.items(): # navigational, then perfect17 # one seed per shopper: the two rankers draw from it in turn, bm25 first18 rng = np.random.default_rng(2026)19 searches = {name: simulate_searches(run, judgments, 5_000, rng, shopper) for name, run in arms.items()}20 for label, column in metrics.items():21 means = {name: table[column].mean() for name, table in searches.items()}22 for name, mean in means.items():23 print(f"{shopper_name} shopper, {name} {label}: {mean:.3f}")24 # exact_click holds True and False; as floats they are 1 and 0 for the t-test25 p = stats.ttest_ind(searches["semantic"][column].astype(float), searches["bm25"][column].astype(float)).pvalue26 print(f"{shopper_name} shopper, {label} p: " + (f"{p:.3f}" if p >= 0.001 else "below 0.001"))27 if p < 0.05:28 significant_leaders[label].add(max(means, key=means.get))29for label in metrics:30 print(f"{label} reverses between shoppers:", "yes" if len(significant_leaders[label]) > 1 else "no")
python clicks-per-search-misleads.pynavigational shopper, bm25 clicks per search: 1.362
navigational shopper, semantic clicks per search: 1.269
navigational shopper, clicks per search p: below 0.001
navigational shopper, bm25 share of searches with an exact click: 0.695
navigational shopper, semantic share of searches with an exact click: 0.799
navigational shopper, share of searches with an exact click p: below 0.001
perfect shopper, bm25 clicks per search: 6.443
perfect shopper, semantic clicks per search: 7.451
perfect shopper, clicks per search p: below 0.001
perfect shopper, bm25 share of searches with an exact click: 0.898
perfect shopper, semantic share of searches with an exact click: 0.895
perfect shopper, share of searches with an exact click p: 0.646
clicks per search reverses between shoppers: yes
share of searches with an exact click reverses between shoppers: no
每次搜索的点击数会在不同购物者之间发生反转吗?yes。它的符号取决于购物者的行为,因此单凭它无法衡量成功。导航型购物者在 BM25 上每次搜索的点击数为 1.362,在语义检索上为 1.269,p 值为 below 0.001;而完美购物者在语义检索上为 7.451,BM25 为 6.443。在靠前位置找到精确匹配商品的购物者会点击它,通常随即离开;而看到的只是部分匹配商品的购物者会点击好几个,仍在继续寻找。
点击了精确匹配商品的搜索所占的比例会在不同购物者之间发生反转吗?no。对导航型购物者而言,它偏向语义检索,为 0.799,对比 0.695;对完美购物者而言,二者没有明显差异,p 值为 0.646,因为只要前 10 条中有精确匹配商品,该购物者就会点击它:在 BM25 的搜索中占 0.898,在语义检索的搜索中占 0.895。
若所有购物者都得出同一个答案,点击就可以作为质量的度量。但点击度量的是行为,而行为是因人而异的。
工作准则: 线下与线上结果不一致时,先找出原因,再决定相信哪一方。检查线上指标衡量的是否为成功,例如购买,而不是活跃度。检查线上胜出者结果的 judged@10,并对缺失的部分补做判断。然后依据事先选定的指标的线上结果做决定,并把查询或判断补充到回归集中,使它下次预测得更准。
决定是否上线
上线决定要综合离线检查与在线检查。请在看到任何结果之前就写下检查清单,这样就不会有哪一个数字能说服你做出变更。我们建议采用下列各项,每项都附有展示它的节:
- 在回归集上取得已确立的离线增益,依据计划内比较规则,附带其 t 检验与区间(逐查询差异),并且要对照你手头最强的系统。Armstrong 等人发现,在 TREC 数据集上十余年发表的增益中,“基线通常很弱,往往低于最初 TREC 系统的中位数”(Armstrong et al., 2009)。
- 已阅读损失查询,并且 α = 1 时的 URisk 不为负(URisk)。
- **判断覆盖两次运行:**无论把哪一次运行中未判断的商品视为完全匹配,领先者都保持不变;若领先者会改变,则需要新的判断(未判断商品)。
- 为基于 LLM 的排序器的核心得分提供人工评级(第 4 章)。
- 延迟与成本在预算之内(第 9 章)。
- 每一项设置都在训练查询上选定,且每一次查看测试查询都有记录(留出查询)。
- **事先写好的在线方案:**主要指标与护栏指标、由功效计算得出的样本量、A/A 测试、样本比例检查,以及不中途偷看结果(A/B 测试)。交错测试可以先筛选候选方案(团队选择法)。
以商店改用语义检索为例,看前三项。第 1 项要求已确立的增益,置换检验给出的答案是yes,平均变化为0.105,95% 区间为0.055 to 0.185。第 2 项,α = 1 时的 URisk 为0.092。
第 3 项,判断覆盖两次运行:如果 BM25 未判断的商品全部视为完全匹配,BM25 将得到0.788,语义检索将得到0.829。领先者会改变吗?no。在线各项要等流量到来。
**工作准则:**当所有离线项目都通过,并且事先确定样本量的在线测试在主要指标上显示出显著增益、且没有护栏指标变差时,才可上线。当某个离线项目未通过时,先修正它再进行在线测试,因为在线测试会耗费购物者的时间。当在线测试在计划样本量下仍无定论时,保留基线。
生产环境中会出什么问题
一家商店在真实流量上逐项测试各种改动,用真实的购物者代替模拟的购物者,并且每一步都使用同一套回归集。
看板会诱使人提前查看结果。 实验平台每天都会显示结果,而团队只要在结果第一次显得显著时就停止实验,得到的假阳性就会远高于 5%,正如 A/A 模拟所示。应事先确定样本量,或者使用序贯方法。
各组与计划不符。 Fabijan 等人把多家公司出现样本比例失配的原因归纳为一个分类体系。在阅读任何结果之前,先检查分组比例;在弄清分组为何出现偏差之前,不要相信结果。
回归集会被用旧。 每一个依据测试查询做出的决定,都会消耗这些查询的一部分独立性(Dwork 等人,2015)。应记录每一次查看,并在记录变长时,用新的查询和判断刷新测试集。
LLM 评判者变成了关卡。 LLM 的判断成本足够低,可以为每个候选方案打分。Otero 等人发现,对 LLM 判断做显著性检验,会出现"在统计差异方面极高的假阳性率"(Otero 等人,2025)。在依据 LLM 评判得到的提升上线之前,应先用人工判断加以确认(第 4 章)。
并列得分会改变结果。 得分相同的商品,会按照库的选择操作所留下的顺序返回(第 3 章),因此一项不触及任何得分的改动,仍然可能改变结果。BM25 并列项的顺序反转,会改变这家商店的 100 次搜索交错评测结论吗?yes。在每个排序器中都按 ID 打破并列,并且在依据某个接近阈值的结果采取行动之前,先把并列顺序反转后重新运行一遍。
交错评测看不到整个页面。 我们认为,速度更慢的排序器,或者改变了结果数量的排序器,可能在交错测试中获胜,却仍然流失购物者。对于这类改动,应运行 A/B 测试。
工作准则: 在实验平台中将 A/A 测试、样本比例检查和固定的测试样本量自动化,记录每一次对测试查询的查看,按 ID 打破并列,并用人工判断确认由 LLM 评判得到的提升。
本章做出的决定
每项默认设置的理由,见其所链接的那一节。
- **回归集由 30 个测试查询及其相关性判断构成,**所有选择都使用 50 个训练查询(留出查询)。
- **在测试查询上只做一次预先计划的比较,**即语义检索与 BM25 的比较,只有当置换检验的 p 值低于 0.05 时才算成立,同时附上 t 检验和 95% 区间(逐查询差异)。128 维排序器属于探索性方案,是根据它在测试集上的表现选出的(URisk)。
- **每个候选方案都要做逐查询差异分析,**并且人工阅读其退步的查询。
- **URisk 取 α = 1,**因此每次退步按两倍计。这是对退步代价的一种判断,失败代价高昂的商店应当调高该值(URisk)。
- **当领先者在两种未判断情形之间发生变化时,先做判断,再做比较,**并把新的判断保留在回归集中(未判断的商品)。
- **每做一次决定,测试查询只查看一次,**并且每次查看都要记录在案(留出查询)。
- **A/B 测试在开始之前按 16σ²/Δ² 确定样本量,**达到该样本量后只读取一次结果,并且事先要做 A/A 测试和样本比例检查(功效)。
- 交错测试采用团队选秀法和符号检验(团队选秀)。
- **模拟购物者遵循映射到三个等级上的导航型级联模型,**查询频率均匀,各项概率在
lib.experiments中作为假设陈述,每次模拟都使用固定的随机种子。他们的判定只用于演练方法,从不算作证据(演练)。 - **每次搜索的点击数本身不是成功指标。**应使用衡量成功的指标,例如购买(离线与在线)。
- **本章的启示:**在查看之前先确定什么会改变你的想法,然后只查看一次。
本章,在本版次中
版次 2026.11.10示例于 2026年11月19日 通过本章 11个示例 个示例中有 11 个通过
译自英文版图书,版次 2026.11.10。