Ranking
阅读约需 92分钟 分钟,12节中含14个示例。
这部分尚未翻译,因此以英文显示。
By the end of this chapter you can describe each product that retrieval returns with features, train a learning-to-rank model with LightGBM on your own judgments, rerank the top of the list with a cross-encoder or an LLM, measure an LLM reranker's positional bias, set the rerank depth by counting its costs, and distil a slow reranker into a fast one. You can also decide, with a planned comparison on the test queries, whether any of them beats your retrieval.
Semantic retrieval alone is still the ranker to beat at the store, where every search is ordered by one embedding score and nothing else. It ranks products by embeddings from Snowflake/snowflake-arctic-embed-m-v1.5, and scores 0.830 nDCG@10 on the 30 test queries. No change in chapters 5 to 8 has beaten it by an established margin. This chapter asks whether a second stage can: a model trained on the store's own judgments, a reranker that reads the query and the product together, or an LLM. Chapter 1's cross-encoder reranked the hybrid run, not semantic retrieval, and never saw the store's judgments.
So this chapter tries all three on semantic retrieval's top results, each scored against the store's judgments. Reranking reorders the top of retrieval's list with a slower model. The store's rerankers are recorded: each ran once and saved its outputs, which the examples read.
The store's LLM is the open model gemma4:12b-it-q4_K_M, licensed Apache-2.0. It ran locally with chapter 4's settings, which make it take its likeliest next word every time, plus thinking off, so it answers without writing out reasoning, and a JSON schema, a machine-readable description of the answer's shape. recordings/llm_rerank.py reranked semantic retrieval's top 20 for the test queries only, so nothing about the LLM rerankers can be tuned. Every score uses the human judgments, because in one study LLM judges ranked LLM rerankers too high (What goes wrong in production). These results describe one model and two prompts, so treat them as the store's, not as a verdict on LLM rerankers.
The chapter fixed its 5 planned comparisons on the test queries before it read any test result. Each sets one method against semantic retrieval's run on nDCG@10:
- LambdaMART, a learned ranker trained on the train queries' judgments, ordering each query's candidate products
- the cross-encoder reranking semantic retrieval's top 20
- the LLM scoring each of the top 20 on its own
- the LLM ordering the top 20 in one prompt
- a small distilled ranker trained on the cross-encoder's scores
The bar is the book's rule: 5 planned comparisons split its 0.05, so each needs a paired permutation test p-value below 0.01. The paired t-test and a 99% bootstrap interval stand beside it.
用特征描述每个候选商品
本章计划进行的比较中,第一项是学习式排序器,它不读取文本。学习式排序器根据一组数字(即特征)为每一对查询和商品打分,并从评判中学习如何对这些特征加权。查询相关特征同时描述查询和商品,例如 BM25 分数。查询无关特征只描述商品本身,例如有多少购物者为它评过分。网页搜索的特征集两类兼有:LETOR 4.0 是学习排序的一个基准,含有 46 个特征,其中有“BM25 of title”“PageRank”和“Number of slash in URL”(Qin and Liu, 2013)。排序器只为候选打分,候选是成本较低的检索阶段返回的商品,因此它的特征可以计算得较为昂贵。
商店每个查询的候选,是交叉编码器的记录所打过分的商品:即 BM25 前 100 名与语义检索前 100 名的并集。lib.ranking.feature_table为每个候选提供 8 个特征:
bm25:来自lib.lexical的 BM25 分数,该实现是本书未做词干提取的基线;若商品与查询没有共同的词项,则为 0similarity:查询嵌入与商品嵌入的余弦相似度cross_encoder:已记录的交叉编码器分数name_match和attribute_match:查询的词项(按第 5 章的方式分词)在商品名称和商品属性中出现的比例rating_count、average_rating和review_count,由商店的数据提供给每个商品
目录中有 2632 的商品没有评分,它们的三个评分特征保持缺失。该表还包含每个候选的人工等级;若无人评判,则留空。为训练查询构建该表,先查看一个查询,再按等级比较各个特征:
1"""Builds the feature table for the train queries, shows one query's candidates, and compares each feature's mean across2the human grades. The prose's point rests on the mean cross_encoder line for each grade and the mean of each feature by grade."""3import pandas as pd45from lib.ranking import FEATURES, feature_table6from lib.store import load_products, load_queries78products, queries = load_products(), load_queries("train") # train queries: features are design work9table = feature_table(products, queries)10print(f"features: {len(FEATURES)}")11print(f"train candidates: {len(table)}")12print(f"train candidates per query, median: {table.groupby('query_id').size().median():.0f}")13print(f"share of train candidates nobody judged: {table['grade'].isna().mean():.3f}")1415query_id = queries.index[queries["query"] == "laundry basket with wheels"][0]16one = table[table["query_id"] == query_id].sort_values("cross_encoder", ascending=False)17one = one.assign(name=products.loc[one["product_id"], "name"].str[:38].to_numpy())18pd.set_option("display.width", 200)19print(f"\ncandidates for laundry basket with wheels: {len(one)}")20print(one[["name", "bm25", "similarity", "cross_encoder", "name_match", "attribute_match", "rating_count", "grade"]]21 .head(6).round(3).to_string(index=False))22print(f"top 6 by cross-encoder, exact: {(one['grade'].head(6) == 2).sum()}")2324# grades 2, 1 and 0 are exact, partial and irrelevant; a missing grade means nobody judged the pair25label = table["grade"].map({2: "exact", 1: "partial", 0: "irrelevant"}).fillna("unjudged")26means = table[FEATURES].groupby(label).mean().loc[["exact", "partial", "irrelevant", "unjudged"]]27print("\nmean of each feature by grade, train candidates:")28print(means.round(3).to_string())29for grade in means.index:30 print(f"mean cross_encoder, {grade}: {means.at[grade, 'cross_encoder']:.3f}")31relevance = ["bm25", "similarity", "cross_encoder", "name_match"]32nearer = [name for name in relevance if abs(means.at["unjudged", name] - means.at["irrelevant", name])33 < abs(means.at["unjudged", name] - means.at["partial", name])]34print(f"relevance features where unjudged sits nearer irrelevant than partial: {len(nearer)} of {len(relevance)}")35medians = table[["rating_count", "review_count"]].groupby(label).median().loc[means.index] # counts are skewed36for grade in ["exact", "partial", "irrelevant"]:37 print(f"rating_count, {grade}, mean: {means.at[grade, 'rating_count']:.3f}")38 print(f"rating_count, {grade}, median: {medians.at[grade, 'rating_count']:.1f}")39print(f"products with no rating: {products['rating_count'].isna().sum()}")
python ranking-features-by-grade.pyfeatures: 8
train candidates: 6896
train candidates per query, median: 131
share of train candidates nobody judged: 0.286
candidates for laundry basket with wheels: 125
name bm25 similarity cross_encoder name_match attribute_match rating_count grade
laundry sorter basket on wheels 8.229 0.597 3.433 1.000 0.0 NaN 2.0
vintage laundry basket 6.270 0.572 2.995 0.667 0.0 638.0 2.0
rolling wicker laundry basket 5.494 0.588 2.974 0.667 0.0 110.0 2.0
aluminum frame laundry basket can be f 4.739 0.524 2.941 0.667 0.0 14.0 2.0
rebrilliant rolling laundry hamper 5.537 0.558 2.804 0.333 0.0 17.0 2.0
4-bag laundry hamper sorter cart baske 6.012 0.543 2.739 0.667 0.0 NaN 2.0
top 6 by cross-encoder, exact: 6
mean of each feature by grade, train candidates:
bm25 similarity cross_encoder name_match attribute_match rating_count average_rating review_count
grade
exact 4.317 0.517 2.191 0.562 0.024 466.879 4.537 333.589
partial 3.936 0.454 1.513 0.461 0.030 385.454 4.539 288.779
irrelevant 2.760 0.360 1.159 0.404 0.006 246.781 4.552 159.052
unjudged 2.323 0.369 1.050 0.140 0.018 339.904 4.520 237.027
mean cross_encoder, exact: 2.191
mean cross_encoder, partial: 1.513
mean cross_encoder, irrelevant: 1.159
mean cross_encoder, unjudged: 1.050
relevance features where unjudged sits nearer irrelevant than partial: 4 of 4
rating_count, exact, mean: 466.879
rating_count, exact, median: 43.0
rating_count, partial, mean: 385.454
rating_count, partial, median: 31.0
rating_count, irrelevant, mean: 246.781
rating_count, irrelevant, median: 35.5
products with no rating: 2632
训练查询共有 6896 个候选,每个查询的中位数为 131 个,其中 0.286 无人评判。
laundry basket with wheels有 125 个候选。按交叉编码器分数排在前六位的商品中,有 6 个是完全相关的商品。它们的attribute_match为 0,因为属性中记录的是颜色、材质和尺寸,而不是wheels这样的词。
在全部训练候选中,交叉编码器分数的均值从完全相关商品的 2.191 降到部分相关商品的 1.513 和不相关商品的 1.159,bm25、similarity和name_match也以同样方式下降。average_rating和attribute_match几乎不变。
rating_count的均值随等级下降,但中位数并非如此:少数评分很多的商品决定了均值。
| 等级 | 评分数均值 | 评分数中位数 |
|---|---|---|
| exact | 466.879 | 43.0 |
| partial | 385.454 | 31.0 |
| irrelevant | 246.781 | 35.5 |
未评判候选的交叉编码器分数平均为 1.050;在随等级下降的相关性特征中,有 4 of 4 个特征上,它们的均值更接近不相关商品,而不是部分相关商品。这关系到未评判的候选应当教给模型什么。
工作准则: 从你已经在计算的特征入手:每个检索器的分数,若有重排序器则加上它的分数,名称等字段上的匹配情况,以及热度信号。训练之前,先在训练查询上按等级比较每个特征,分数用均值,偏斜的计数用中位数。不随等级变化的特征,能教给模型的很少。训练和上线服务时,要用同一份代码计算特征。
这部分尚未翻译,因此以英文显示。
Learn a ranking from pairs of products
Learning to rank trains a model on judged queries to order candidates by their features. Its methods fall into three families, pointwise, pairwise and listwise (Liu, 2009). A pointwise model predicts each product's grade on its own, as a regression. A pairwise model learns which of two products should rank higher. A listwise model scores a whole ranking. Liu's 2009 comparison on the LETOR benchmark favoured the listwise approach, though tentatively. The obstacle for all three is the metric: nDCG changes only when two products swap places, so ranking metrics "are everywhere either discontinuous or flat, so gradient descent appears to be problematic" (Burges, 2010).
RankNet, a pairwise method, works around it with a smooth loss on pairs. For two products where i should rank above j, its gradient is large when the model scores them the wrong way round or nearly level, and small when it already has them well apart. LambdaRank keeps that gradient and multiplies it by how much nDCG would change if the two products swapped places (Burges, 2010, section 4.1). Burges writes the gradient with a minus sign, as the slope of a cost; here a positive λ pushes i up:
λ_ij = σ / (1 + exp(σ · (s_i − s_j))) · |ΔnDCG_ij| s: the model's scores, σ = 1
ΔnDCG_ij = (g_i − g_j) · (1 / log2(1 + r_i) − 1 / log2(1 + r_j)) / ideal DCG g: grades, r: positions
A product's total is its λ summed over the pairs where it should rank higher, minus those where it should rank lower. Burges (2010) also reads the λ's as forces, pushing products up and down the list.
LambdaMART uses LambdaRank's λ as the gradients for gradient-boosted trees, which add small decision trees one at a time, each fitted to the gradients of the trees before it: "LambdaMART is the boosted tree version of LambdaRank, which is based on RankNet". An ensemble of LambdaMART rankers won Track 1 of the Learning to Rank Challenge that Yahoo ran in 2010 (Burges, 2010).
One train query, outdoor waterproof chest, shows the arithmetic. Let semantic similarity be the model's scores so far, compute every pair's λ in its top 10 by hand, check each |ΔnDCG| by swapping the pair and recomputing nDCG with lib.metrics, and add up the forces:
1"""Works out LambdaRank's λ for every pair in one train query's top 10 by hand, and checks each pair's |ΔnDCG| against2swapping the pair in the metric. The prose's point rests on the strongest push up and down, and on the count of pairs3whose formula matches swapping."""4import math5from itertools import combinations67from lib.metrics import dcg_at_k, ndcg_at_k8from lib.semantic import SemanticSearch9from lib.store import load_queries, qrels1011queries, judgments = load_queries("train"), qrels("train")12query_id = queries.index[queries["query"] == "outdoor waterproof chest"][0]13judged = judgments[query_id]14scores = SemanticSearch().search(query_id, k=10) # the model so far: semantic similarity15ranking, similarity = list(scores.index), scores.to_numpy()16# an unjudged product counts as grade 0, as it does in the metric17grades = [judged.get(product_id, 0) for product_id in ranking]18# the ideal DCG takes every judged product, as lib.metrics does; sigma is 1, LightGBM's default sigmoid19ideal, sigma = dcg_at_k(sorted(judged.values(), reverse=True), 10), 1.020print("grades from the top:", " ".join(map(str, grades)))2122force, checks, pulls = [0.0] * len(ranking), [], []23for i, j in combinations(range(len(ranking)), 2): # i is above j24 if grades[i] == grades[j]:25 continue26 better, worse = (i, j) if grades[i] > grades[j] else (j, i)27 ranknet = 1 / (1 + math.exp(sigma * (similarity[better] - similarity[worse]))) # RankNet's pull28 # i and j count from 0, so position p is i + 1 and its discount is 1 / log2(p + 1), which is 1 / log2(i + 2)29 delta = abs((grades[i] - grades[j]) * (1 / math.log2(i + 2) - 1 / math.log2(j + 2))) / ideal30 swapped = ranking.copy()31 swapped[i], swapped[j] = swapped[j], swapped[i]32 checks.append(math.isclose(delta, abs(ndcg_at_k(swapped, judged, 10) - ndcg_at_k(ranking, judged, 10))))33 pulls.append(ranknet)34 pair_lambda = sigma * ranknet * delta # LambdaRank: scaled by |delta nDCG|35 force[better], force[worse] = force[better] + pair_lambda, force[worse] - pair_lambda3637print(f"pairs with different grades: {len(checks)}")38print(f"pairs where delta ndcg by formula matches swapping the pair: {sum(checks)}")39for position, push in enumerate(force, start=1):40 print(f"force at position {position}: {push:+.3f}")41# list indexes count from 0 and positions from 1, hence the + 142print(f"strongest push up, position: {force.index(max(force)) + 1}")43print(f"strongest push down, position: {force.index(min(force)) + 1}")44print(f"ranknet's pull, lowest: {min(pulls):.3f}\nranknet's pull, highest: {max(pulls):.3f}")
python lambdas-for-one-query.pygrades from the top: 1 2 2 0 1 2 0 2 0 0
pairs with different grades: 32
pairs where delta ndcg by formula matches swapping the pair: 32
force at position 1: +0.013
force at position 2: +0.159
force at position 3: +0.104
force at position 4: -0.083
force at position 5: -0.010
force at position 6: +0.063
force at position 7: -0.092
force at position 8: +0.063
force at position 9: -0.105
force at position 10: -0.111
strongest push up, position: 2
strongest push down, position: 10
ranknet's pull, lowest: 0.472
ranknet's pull, highest: 0.525
The formula's |ΔnDCG| matches the change from swapping the pair for 32 of the 32 pairs in the top 10 with different grades. The top 10's grades, from the top, are 1 2 2 0 1 2 0 2 0 0; an unjudged product counts as 0, as it does in the metric.
The strongest push up goes to position 2, an exact product, with +0.159; the strongest push down goes to position 10, an irrelevant product, with -0.111. The partial product at position 1 gets a net push of +0.013: the irrelevant products below push it up, and the exact ones push it down.
RankNet's factor in λ stays between 0.472 and 0.525, because the similarities lie close together, so |ΔnDCG| decides almost everything.
That example simplifies. It takes the ideal DCG from every judged product, as lib.metrics does, and looks at ten products. LightGBM takes the ideal from the query's training rows, and at 4.7.0 two defaults change the λ. lambdarank_truncation_level is 30, which limits training to the top 30 results: only pairs with a product in the current top 30 get a λ. lambdarank_norm is true, which normalizes the lambdas across queries; set it to false to get the original lambdarank algorithm (Parameters). Normalizing, as LightGBM's source does it, divides each pair's |ΔnDCG| by the gap between its scores plus 0.01, then rescales each query's λ (rank_objective.hpp). To check the formula against the library, write it as a custom LightGBM objective, grow five small trees on this query's candidates, and grow them again with LightGBM's lambdarank, with and without those defaults:
1"""Writes LambdaRank's gradients by hand as a custom LightGBM objective, then checks them against LightGBM's own lambdarank2on one query, with and without its defaults. The prose's point rests on the two printed differences: a rounding-sized gap3without the defaults, and a large one at them."""4import numpy as np5from lightgbm import LGBMRanker67from lib.ranking import DETERMINISTIC, FEATURES, feature_table8from lib.store import load_products, load_queries910queries = load_queries("train")11one = queries[queries["query"] == "outdoor waterproof chest"]12rows = feature_table(load_products(), one) # one query's candidates, in product_id order13# unjudged candidates count as grade 0, as in the metric14grades = rows["grade"].fillna(0).astype(int).to_numpy()15SIGMA = 1.0 # LightGBM's default sigmoid161718def lambdas(y_true, y_pred, weight, group):19 """The chapter's lambda for every pair with different grades, as gradients and second derivatives."""20 rank = np.empty(len(y_pred), dtype=int)21 rank[np.argsort(-y_pred, kind="stable")] = np.arange(len(y_pred)) # LightGBM's stable sort on score22 # rank counts from 0, so the product at position rank + 1 has the discount 1 / log2(rank + 2)23 discount, ideal = 1 / np.log2(rank + 2), np.sum(np.sort(y_true)[::-1] / np.log2(np.arange(len(y_true)) + 2))24 gradient, hessian = np.zeros(len(y_pred)), np.zeros(len(y_pred))25 for i in range(len(y_true)):26 for j in np.flatnonzero(y_true < y_true[i]): # i should rank above j27 delta = (y_true[i] - y_true[j]) * abs(discount[i] - discount[j]) / ideal28 pull = 1 / (1 + np.exp(SIGMA * (y_pred[i] - y_pred[j])))29 gradient[i], gradient[j] = gradient[i] - SIGMA * pull * delta, gradient[j] + SIGMA * pull * delta30 hessian[[i, j]] += SIGMA ** 2 * pull * (1 - pull) * delta31 return gradient, hessian323334# five small trees; min_child_samples is 5 where LightGBM's default is 20, so the trees can split this query's candidates35tree = {"n_estimators": 5, "num_leaves": 3, "min_child_samples": 5, **DETERMINISTIC}36# turns off normalization and sets truncation to every candidate, so every pair gets a λ, as the formula has it37plain = {"lambdarank_norm": False, "lambdarank_truncation_level": len(rows)} # LightGBM's lambdas, untouched38models = {"by hand": LGBMRanker(objective=lambdas, **tree),39 "lightgbm, no normalization or truncation": LGBMRanker(objective="lambdarank", label_gain=[0, 1, 2], **plain, **tree),40 "lightgbm at its defaults": LGBMRanker(objective="lambdarank", label_gain=[0, 1, 2], **tree)}41# raw scores: the values the gradients act on, before any sigmoid42scores = {name: model.fit(rows[FEATURES], grades, group=[len(rows)]).predict(rows[FEATURES], raw_score=True)43 for name, model in models.items()}44print(f"candidates: {len(rows)}")45for name in list(models)[1:]:46 difference = np.abs(scores[name] - scores["by hand"]).max()47 print(f"{name}, largest difference from the lambdas by hand after 5 trees: {difference:.6f}")
python lambdas-match-lightgbm.pycandidates: 180
lightgbm, no normalization or truncation, largest difference from the lambdas by hand after 5 trees: 0.000006
lightgbm at its defaults, largest difference from the lambdas by hand after 5 trees: 0.595667
Over the query's 180 candidates, LightGBM without normalization or truncation scores every product within 0.000006 of the custom objective's scores. The gap is rounding, since LightGBM reads its sigmoid from a precomputed table (rank_objective.hpp). At its defaults, scores differ by up to 0.595667.
Working rule: Train rankers with a gradient that weighs each pair by the metric you report, as LambdaMART does, rather than regressing on grades. Know which defaults of your library change that gradient, and check on train queries that it beats a pointwise model on your data, because the families' order in a benchmark doesn't transfer by itself.
用 LightGBM 训练 LambdaMART
LightGBM 通过 LGBMRanker 和 objective="lambdarank" 训练 LambdaMART。有四项设置需要留意,每一项在 4.7.0 中都有文档说明:
- 行按查询分组。“data should be ordered by the query”,而
group按该顺序给出每个查询的行数,而不是查询 ID,因此各组大小之和等于行数(sum(group) = n_samples;Parameters;LGBMRanker)。 - 增益默认为指数形式。
label_gain默认为0,1,3,7,15,31,63,...,因此“the gain of label2is3”,并且每个标签都必须小于增益的个数(Parameters)。这就是 Burges 的 DCG,增益为 2^grade − 1。本书的 nDCG 使用线性增益,所以应设置label_gain=[0, 1, 2]。 - **nDCG 默认在第 1 到第 5 位上评估。**将
eval_at=[10]设置为按 nDCG@10 提前停止,并且无论如何都用lib.metrics报告最终得分。 - **验证数据放在
eval_X和eval_y中。**在 4.7.0 中,eval_set为“Deprecated since version 4.7.0”,由eval_X和eval_y取代(LGBMRanker;发布说明)。这些示例传入eval_X、eval_y和eval_group。
提前停止在每棵树之后对留出的训练查询打分,当再增加 30 棵树仍无提升时,保留到最佳得分为止的那些树。为使训练能够精确重复,这些示例使用单线程、固定随机种子和 deterministic=True,它“should ensure the stable results when using the same data”(Parameters),同时使用 force_row_wise=True,同一条目推荐这样做,以避免数值问题带来的潜在不稳定。lib.ranking 保存这些设置,其 fit 负责应用分组和提前停止。
行的顺序同样重要。在第一棵树之前,每个得分都是 0,因此 LightGBM 把行的顺序当作计算第一个 λ 时的初始排序,而相同的得分需要一种平局裁决方式。交叉编码器的记录把每个查询的候选按从好到差列出,这会在模型学到任何东西之前就把交叉编码器的排序交给它。因此 feature_table 按 product_id 对候选排序,lib.ranking.ranked 按 product_id 裁决平局。
用训练查询上的交叉验证来选择树的大小:把它们分成五折,对每一折用在其余三折上拟合、并在第五折上提前停止的模型进行排序,使每个查询恰好被打分一次。尝试八种设置,叶子数从 3 到 31,配以两种学习率,并像该指标那样把未经评判的候选计为等级 0:
1"""Cross-validates LambdaMART on the train queries over eight tree sizes and learning rates, and names the best. The prose's2point rests on the best setting's score, and on the score beside it for LightGBM's default of 31 leaves at 0.1."""3from lightgbm import LGBMRanker45from lib.metrics import evaluate6from lib.ranking import DETERMINISTIC, FEATURES, LAMBDARANK, TREE_GRID, TREES, cross_validate, feature_table7from lib.semantic import SemanticSearch8from lib.store import load_products, load_queries, qrels910queries, judgments = load_queries("train"), qrels("train") # tuning: train queries only11candidates = feature_table(load_products(), queries)12candidates["grade or 0"] = candidates["grade"].fillna(0).astype(int) # unjudged counts as irrelevant, as in nDCG13print(f"train queries: {len(queries)}")14print(f"semantic, train ndcg@10: {evaluate(SemanticSearch().run(queries, k=100), judgments)['ndcg@10'].mean():.3f}")1516results = {}17for setting in TREE_GRID: # LightGBM's default is 31 leaves at 0.118 name = f"{setting['num_leaves']} leaves, learning rate {setting['learning_rate']}"19 # up to 500 trees; early stopping inside cross_validate keeps fewer20 model = lambda: LGBMRanker(**LAMBDARANK, n_estimators=500, **setting, **DETERMINISTIC)21 # one nDCG@10 per train query, each ranked by a model that never saw that query22 results[name] = cross_validate(model, candidates, candidates, "grade or 0", FEATURES, judgments)23 print(f"{name}, cross-validated ndcg@10: {results[name].mean():.3f}")24 # the trees early stopping kept in each fold25 print(f"{name}, trees kept per fold: {', '.join(map(str, results[name].attrs['trees']))}")26best = max(results, key=lambda name: results[name].mean())27print(f"settings tried: {len(results)}")28print(f"best setting: {best}")29print(f"lib.ranking.TREES holds it: {'yes' if best == f'{TREES['num_leaves']} leaves, learning rate {TREES['learning_rate']}' else 'no'}")
python lambdamart-tree-size-on-train.pytrain queries: 50
semantic, train ndcg@10: 0.772
3 leaves, learning rate 0.05, cross-validated ndcg@10: 0.780
3 leaves, learning rate 0.05, trees kept per fold: 18, 27, 52, 20, 85
3 leaves, learning rate 0.1, cross-validated ndcg@10: 0.798
3 leaves, learning rate 0.1, trees kept per fold: 99, 20, 8, 43, 38
7 leaves, learning rate 0.05, cross-validated ndcg@10: 0.744
7 leaves, learning rate 0.05, trees kept per fold: 2, 28, 30, 39, 55
7 leaves, learning rate 0.1, cross-validated ndcg@10: 0.772
7 leaves, learning rate 0.1, trees kept per fold: 80, 12, 13, 7, 20
15 leaves, learning rate 0.05, cross-validated ndcg@10: 0.720
15 leaves, learning rate 0.05, trees kept per fold: 4, 37, 175, 44, 120
15 leaves, learning rate 0.1, cross-validated ndcg@10: 0.728
15 leaves, learning rate 0.1, trees kept per fold: 19, 68, 136, 48, 53
31 leaves, learning rate 0.05, cross-validated ndcg@10: 0.679
31 leaves, learning rate 0.05, trees kept per fold: 15, 8, 91, 8, 22
31 leaves, learning rate 0.1, cross-validated ndcg@10: 0.737
31 leaves, learning rate 0.1, trees kept per fold: 11, 75, 166, 96, 58
settings tried: 8
best setting: 3 leaves, learning rate 0.1
lib.ranking.TREES holds it: yes
在 8 种设置中,最佳设置是 3 leaves, learning rate 0.1,得分为 0.798;LightGBM 默认的 31 个叶子、学习率 0.1(LGBMRanker)得分为 0.737。所有设置中的最佳者会高估自己,因此在留出的查询上应预期得分更低。
有一项选择取决于商店的评判。训练候选中有 0.286 的比例无人评判。模型可以把它们计为等级 0,也可以只从已评判的候选中学习;未评判并不等于不相关,但无论哪种做法,模型在查询时都要对它们排序。在选定的树大小下,用交叉验证比较两者,并在旁边放上对等级做逐点回归的结果:
1"""Compares three ways of training on the train queries, each by cross-validation: LambdaRank with unjudged candidates as 0,2LambdaRank on judged candidates only, and pointwise regression. The prose's point rests on the differences and the3permutation p-values printed under each setup."""4from lightgbm import LGBMRanker, LGBMRegressor56from lib.metrics import evaluate7from lib.ranking import DETERMINISTIC, FEATURES, LAMBDARANK, TREES, cross_validate, feature_table8from lib.semantic import SemanticSearch9from lib.significance import compare10from lib.store import load_products, load_queries, qrels1112queries, judgments = load_queries("train"), qrels("train") # tuning: train queries only13candidates = feature_table(load_products(), queries)14candidates["grade or 0"] = candidates["grade"].fillna(0).astype(int) # unjudged counts as irrelevant15# the rows the model learns from when unjudged candidates are left out; it still ranks every candidate at query time16judged_only = candidates.dropna(subset=["grade"]).astype({"grade": int})1718setups = { # (model, rows it learns from, label)19 "lambdarank, unjudged as 0": (lambda: LGBMRanker(**LAMBDARANK, **TREES, **DETERMINISTIC), candidates, "grade or 0"),20 "lambdarank, unjudged left out": (lambda: LGBMRanker(**LAMBDARANK, **TREES, **DETERMINISTIC), judged_only, "grade"),21 "pointwise regression, unjudged as 0": (lambda: LGBMRegressor(**TREES, **DETERMINISTIC), candidates, "grade or 0"),22}23semantic = evaluate(SemanticSearch().run(queries, k=100), judgments)["ndcg@10"]24print(f"semantic, train ndcg@10: {semantic.mean():.3f}")25scores = {name: cross_validate(model, rows, candidates, label, FEATURES, judgments)26 for name, (model, rows, label) in setups.items()}27chosen = scores["lambdarank, unjudged as 0"]28for name, per_query in scores.items():29 print(f"{name}, cross-validated ndcg@10: {per_query.mean():.3f}")30 if name != "lambdarank, unjudged as 0":31 # the mean difference is the first argument minus the second, tested by permutation over the queries32 result = compare(per_query, chosen)33 print(f"{name}, minus the first: {result['mean difference']:.3f}")34 print(f"{name}, minus the first, permutation p: {result['permutation p']:.3f}")35result = compare(chosen, semantic)36print(f"first setup minus semantic: {result['mean difference']:.3f}")37print(f"first setup minus semantic, permutation p: {result['permutation p']:.3f}")
python lambdamart-choices-on-train.pysemantic, train ndcg@10: 0.772
lambdarank, unjudged as 0, cross-validated ndcg@10: 0.798
lambdarank, unjudged left out, cross-validated ndcg@10: 0.733
lambdarank, unjudged left out, minus the first: -0.064
lambdarank, unjudged left out, minus the first, permutation p: 0.005
pointwise regression, unjudged as 0, cross-validated ndcg@10: 0.769
pointwise regression, unjudged as 0, minus the first: -0.029
pointwise regression, unjudged as 0, minus the first, permutation p: 0.007
first setup minus semantic: 0.026
first setup minus semantic, permutation p: 0.242
把未评判的候选计为 0 的得分高于将其排除,分别为 0.798 和 0.733,差值为 -0.064,置换检验 p 值为 0.005。一个在训练中从未见过未评判候选的模型,仍然要对它们排序,而且这些特征表明它们看起来像不相关的商品。
逐点回归的得分为 0.769,差值为 -0.029(p = 0.007),因此在商店的训练查询上,与逐点模型对照的调参检查更支持 LambdaMART。
与语义检索的 0.772 相比,所选方案的差值为 0.026,置换检验 p 值为 0.242:即使在选择了它的那些查询上,它也没有显示出训练查询能够与偶然区分开来的提升。
**工作准则:**按查询对行排序并传入 group,把 label_gain 设置为指标所用的增益,通过 eval_X 和 eval_y 在留出的训练查询上提前停止,并固定随机种子和线程数。按第 3 章的建议,把每个查询的候选按商品 ID 排序,并以同样的方式裁决得分平局。除非训练查询上的交叉验证另有结论,否则把未评判的候选计为等级 0。已评判的查询较少时,尝试小树。
这部分尚未翻译,因此以英文显示。
Report LambdaMART on the test queries
A model tuned on train queries is reported once on the test queries, by a comparison planned before looking. Feature importance shows what a model learned: LightGBM's "gain" importance is the "total gains of splits which use the feature" (Booster), so a feature with no gain plays no part in the score.
The first planned comparison uses the setup chosen on train: LambdaMART on all the features, with 3 leaves at a learning rate of 0.1, unjudged candidates as grade 0, and candidates in product_id order. It fits on 40 train queries, stops early on 10 others, and ranks each test query's candidates. A gain needs a permutation p-value below 0.01:
1"""Fits LambdaMART on the train queries with the chosen setup, reports its early stopping and gain shares, then reports its2planned comparison on the test queries against semantic retrieval. The prose's point rests on that comparison's permutation3p-value, printed last, against the threshold printed first."""4from lightgbm import LGBMRanker56from lib.metrics import evaluate7from lib.ranking import (DETERMINISTIC, FEATURES, LAMBDARANK, PLANNED_TEST_COMPARISONS, THRESHOLD, TREES, feature_table,8 fit, ranked, report_planned, stopping_split)9from lib.semantic import SemanticSearch10from lib.store import load_products, load_queries, qrels1112products, train, test = load_products(), load_queries("train"), load_queries("test")13print(f"planned test comparisons: {len(PLANNED_TEST_COMPARISONS)}")14print(f"p-value threshold: {THRESHOLD:.2f}")1516table = feature_table(products, train)17table["grade or 0"] = table["grade"].fillna(0).astype(int) # the setup chosen on train18# 10 train queries are held out, at random with a fixed seed, to stop on19fit_ids, stop_ids = stopping_split(train.index)20model = fit(LGBMRanker(**LAMBDARANK, **TREES, **DETERMINISTIC), table, "grade or 0", FEATURES, fit_ids, stop_ids)21print(f"queries fitted: {len(fit_ids)}\nqueries held out for stopping: {len(stop_ids)}")22curve = model.evals_result_["valid_0"]["ndcg@10"] # LightGBM's nDCG@10, one value per tree23print(f"trees tried: {len(curve)}\ntrees kept by early stopping: {model.best_iteration_}")24print(f"stopping queries' ndcg@10, first tree: {curve[0]:.3f}")25# best_iteration_ counts trees from 1 and the curve from 0, so the last tree kept is at index best_iteration_ - 126print(f"stopping queries' ndcg@10, trees kept: {curve[model.best_iteration_ - 1]:.3f}")27gain = model.booster_.feature_importance(importance_type="gain") # total gain of the splits on each feature28# a feature with no gain was never split on, so it plays no part in the score29print(f"features the trees use: {(gain > 0).sum()} of {len(FEATURES)}")30for name, share in sorted(zip(FEATURES, gain / gain.sum()), key=lambda pair: -pair[1]):31 print(f"share of gain, {name}: {share:.3f}")3233candidates, judgments = feature_table(products, test), qrels("test") # test queries: read once, to report34after = evaluate(ranked(candidates, model.predict(candidates[FEATURES])), judgments)35before = evaluate(SemanticSearch().run(test, k=100), judgments)36for name, frame in [("semantic", before), ("lambdamart", after)]:37 print(f"{name}, test ndcg@10: {frame['ndcg@10'].mean():.3f}")38 print(f"{name}, test judged@10: {frame['judged@10'].mean():.3f}")39 # each query's top 10 is 10 places, so (1 - judged@10) times 10 counts its unjudged places40 print(f"{name}, unjudged places in the test top 10s: {round((1 - frame['judged@10']).sum() * 10)}")41report_planned("lambdamart", after["ndcg@10"], before["ndcg@10"])
python lambdamart-on-test.pyplanned test comparisons: 5
p-value threshold: 0.01
queries fitted: 40
queries held out for stopping: 10
trees tried: 243
trees kept by early stopping: 213
stopping queries' ndcg@10, first tree: 0.568
stopping queries' ndcg@10, trees kept: 0.794
features the trees use: 8 of 8
share of gain, cross_encoder: 0.641
share of gain, similarity: 0.172
share of gain, name_match: 0.097
share of gain, bm25: 0.043
share of gain, review_count: 0.021
share of gain, rating_count: 0.021
share of gain, attribute_match: 0.002
share of gain, average_rating: 0.001
semantic, test ndcg@10: 0.830
semantic, test judged@10: 0.990
semantic, unjudged places in the test top 10s: 3
lambdamart, test ndcg@10: 0.817
lambdamart, test judged@10: 0.990
lambdamart, unjudged places in the test top 10s: 3
lambdamart, mean difference: -0.012
lambdamart, wins, losses, ties: 14, 12, 4
lambdamart, permutation p: 0.260
lambdamart, established at 0.01: no
lambdamart, t-test p: 0.278
lambdamart, t-test agrees: yes
lambdamart, 99% interval: -0.044 to 0.011
lambdamart, interval agrees: yes
Of 243 trees tried, early stopping keeps 213. On the queries held out for stopping, LightGBM's own nDCG@10, whose ideal comes from their candidates alone, rises from 0.568 after the first tree to 0.794 at the trees kept.
The trees use 8 of 8 features, but they lean on the cross-encoder, which takes a share of 0.641 of the gain, ahead of similarity at 0.172, name_match at 0.097 and bm25 at 0.043. So serving the model means a cross-encoder call for every candidate.
LambdaMART doesn't establish a gain on the test queries: it scores 0.817 nDCG@10, against semantic retrieval's 0.830, at a permutation p-value of 0.260 (all five comparisons). Judged@10 is 0.990 for LambdaMART and 0.990 for semantic retrieval.
Working rule: Adopt a learned ranker only when its planned test comparison is established. Read its gain importance before you trust it, because the features it leans on are the ones you'll pay for at query time. Early stopping on a few held-out queries keeps very different numbers of trees from one split to the next, from 99, 20, 8, 43, 38 across the cross-validation folds to 213 here. A rule of thumb, not a measurement: with this few queries, judge more queries before you tune further, and treat any one stopping point as noise.
用交叉编码器重排语义检索结果
语义检索使用嵌入模型,也称为双编码器。它分别读取查询和每个商品,因此商品的嵌入可以提前计算。交叉编码器则把查询和一个商品放在一起读取,输出一个分数(Nogueira and Cho, 2019)。它可以把查询中的每个词与商品中的每个词联系起来,但这要付出代价。它通常比 Sentence Transformer 模型慢,因为它针对每个查询与商品的配对计算,而不是针对每段文本计算,所以“交叉编码器常用于对前 k 个结果重新排序”(Sentence Transformers)。本店的交叉编码器(见开篇)按语言模型的标准来看很小,有 149M 个参数(模型卡)。它是开放模型 Alibaba-NLP/gte-reranker-modernbert-base,其清单(随记录一起保存的文件)给出的许可证是 Apache-2.0。我们选择它,是因为任何人都可以下载它并重新运行这份记录。recordings/cross_encoder.py 为每个查询的 BM25 与语义检索前 100 名的并集打了分(有两个查询,其语义检索列表与 lib.semantic 的列表在第 100 位上有一处近乎并列的差异)。
第 1 章重排的是混合检索的结果。这里由交叉编码器重排需要超越的排序器,即语义检索的前 20 名:这些商品与 LLM 重排器读取的相同,因此本章的三个重排器从相同的列表出发。本章在看到任何测试结果之前就固定了这一深度,并且不对它调优。按记录的分数对前 20 名重新排序,其余部分保持原位。这是第二项计划内比较,要求 p 值低于 0.01:
1"""Reranks semantic retrieval's top 20 with the recorded cross-encoder scores, and compares the two on the test queries.2The prose's point rests on the planned comparison's permutation p-value, and on the judged@10 lines, which show what the3rerank does to the judgments it can score."""4from lib.metrics import evaluate5from lib.ranking import DEPTH, rerank, report_planned6from lib.recorded import cross_encoder_scores, llm_listwise, manifest7from lib.semantic import SemanticSearch8from lib.store import load_queries, qrels910queries, judgments = load_queries("test"), qrels("test") # scored on the human judgments11semantic = SemanticSearch().run(queries, k=100)12recorded = cross_encoder_scores()13# the recorded score of each (query, product) pair14score = dict(zip(zip(recorded["query_id"], recorded["product_id"]), recorded["score"]))15print("cross-encoder model:", manifest("cross-encoder")["model"])16print("licence in the manifest:", manifest("cross-encoder")["model_licence"])1718# the set of products the LLM was shown for each query, from the listwise recording19shown = llm_listwise().groupby("query_id")["product_id"].apply(set) # what the LLM rerankers saw20same = sum(set(semantic[query_id][:DEPTH]) == shown[query_id] for query_id in queries.index)21print(f"rerank depth: {DEPTH}")22print(f"queries whose top {DEPTH} holds the products the llm rerankers saw: {same} of {len(queries)}")2324reranked = rerank(semantic, score, DEPTH) # top 20 reordered, the rest in place25before, after = evaluate(semantic, judgments), evaluate(reranked, judgments)26for name, frame in [("semantic", before), ("cross-encoder", after)]:27 print(f"{name}, test ndcg@10: {frame['ndcg@10'].mean():.3f}")28 print(f"{name}, test judged@10: {frame['judged@10'].mean():.3f}")29 print(f"{name}, unjudged places in the test top 10s: {round((1 - frame['judged@10']).sum() * 10)}")30# index 0 is the first place31moved = sum(reranked[query_id][0] != semantic[query_id][0] for query_id in queries.index)32print(f"queries whose first product changes: {moved}")33report_planned("cross-encoder", after["ndcg@10"], before["ndcg@10"])
python cross-encoder-on-semantic.pycross-encoder model: Alibaba-NLP/gte-reranker-modernbert-base
licence in the manifest: Apache-2.0
rerank depth: 20
queries whose top 20 holds the products the llm rerankers saw: 30 of 30
semantic, test ndcg@10: 0.830
semantic, test judged@10: 0.990
semantic, unjudged places in the test top 10s: 3
cross-encoder, test ndcg@10: 0.813
cross-encoder, test judged@10: 0.983
cross-encoder, unjudged places in the test top 10s: 5
queries whose first product changes: 23
cross-encoder, mean difference: -0.016
cross-encoder, wins, losses, ties: 11, 12, 7
cross-encoder, permutation p: 0.178
cross-encoder, established at 0.01: no
cross-encoder, t-test p: 0.171
cross-encoder, t-test agrees: yes
cross-encoder, 99% interval: -0.058 to 0.006
cross-encoder, interval agrees: yes
重排前 20 名并不能确立提升:重排后的分数为 0.813,语义检索为 0.830,置换检验的 p 值为 0.178(见决策表)。对于 30 of 30 个测试查询,前 20 名包含 LLM 重排器所见的商品,而重排器改变了其中 23 个查询的第一个商品。
Judged@10从 0.990 降到 0.983,测试集前 10 名中未经评判的位置从 3 增加到 5。
在训练查询上,同样的重排得分为 0.802,语义检索为 0.772(深度扫描)。训练查询是另一份样本,其结论重复了第 3 章的发现:两个接近的排序器的先后顺序,可能在不同的查询集之间发生翻转。
工作准则: 对你最强的检索结果做重排,重排深度要在查看测试结果之前固定,并且只有在留出查询上确立了提升时,才采用该重排器。本店的通用交叉编码器不进入流水线。
用 LLM 为每个商品打分
LLM 也可以做重排序,方式有三种。逐点式(pointwise):LLM 单独判断每个商品,返回一个标签或分数。成对式(pairwise):LLM 指出两个商品中哪一个更符合查询。列表式(listwise):LLM 在一次回答中对整个列表排序(Qin et al., 2024)。在一项研究中,研究者使用 BEIR(一组用于检验模型在未训练过的数据上表现的检索基准)中的八个数据集,发现逐点式提示在采用分级回答时比采用是或否回答时效果更好,原因是“缺少中间相关性标签选项,可能导致 LLM 给出含噪声或有偏的回答”(Zhuang et al., 2024a)。
商店使用的 LLM 是本章开头提到的那个开源模型。它每次只看到查询和一个商品:名称、类(class)、类别、描述和属性(见LLM 看到的内容)。提示要求给出一个标签,即 exact、partial 或 irrelevant,并给出“一个 0 到 100 的分数,表示其匹配程度”。按该分数对语义检索的前 20 个结果重新排序;两个商品分数相同时,保持语义检索的顺序,这一规则在录制时已经固定。该重排序是第三项计划内比较,要求 p 值低于 0.01才算有提升:
1"""Reads the pointwise LLM's answers for the test queries, one call per product, and reranks semantic retrieval's top 20 by2their scores. The prose's point rests on the distinct-scores line, which shows how coarse the scores are, and on the3planned comparison's p-value."""4from lib.metrics import evaluate5from lib.ranking import DEPTH, rerank, report_planned6from lib.recorded import llm_pointwise, manifest7from lib.semantic import SemanticSearch8from lib.store import load_queries, qrels910queries, judgments = load_queries("test"), qrels("test") # scored on the human judgments, not the LLM's11semantic = SemanticSearch().run(queries, k=100)12answers = llm_pointwise() # one call per product: a label and a score13print("llm:", manifest("llm-rerank")["model"])14print("licence in the manifest:", manifest("llm-rerank")["model_licence"])15print(f"calls per query: {len(answers) // answers['query_id'].nunique()}")16print(f"answers without a score: {answers['score'].isna().sum()}")17print(f"distinct scores per query, median: {answers.groupby('query_id')['score'].nunique().median():.0f}")18print(f"share of products scored 100: {(answers['score'] == 100).mean():.3f}")19# each answered product's human grade, or missing when nobody judged it20people = answers.apply(lambda row: judgments[row["query_id"]].get(row["product_id"]), axis=1)21# grades 2, 1 and 0 are exact, partial and irrelevant, the labels the LLM was asked for22for label, grade in [("exact", 2), ("partial", 1), ("irrelevant", 0)]:23 print(f"products the llm labels {label}: {(answers['label'] == label).sum()}")24 print(f"products people judged {label}: {(people == grade).sum()}")25print(f"products nobody judged: {people.isna().sum()}")2627score = dict(zip(zip(answers["query_id"], answers["product_id"]), answers["score"]))28reranked = rerank(semantic, score, DEPTH) # equal scores keep semantic retrieval's order29counts = answers.groupby("query_id")["score"].value_counts() # products per score, for each query30shared = {query_id: sum(counts[(query_id, score[(query_id, product_id)])] > 1 for product_id in reranked[query_id][:10])31 for query_id in queries.index} # top-10 products that share their score32# more than 5 of the top 10 share a score: most of the list is decided by the tie-break33print(f"queries where most of the reranked top 10 shares its score: {sum(n > 5 for n in shared.values())}")34before, after = evaluate(semantic, judgments), evaluate(reranked, judgments)35for name, frame in [("semantic", before), ("llm pointwise", after)]:36 print(f"{name}, test ndcg@10: {frame['ndcg@10'].mean():.3f}")37 print(f"{name}, test judged@10: {frame['judged@10'].mean():.3f}")38 print(f"{name}, unjudged places in the test top 10s: {round((1 - frame['judged@10']).sum() * 10)}")39report_planned("llm pointwise", after["ndcg@10"], before["ndcg@10"])
python llm-pointwise-rerank.pyllm: gemma4:12b-it-q4_K_M
licence in the manifest: Apache-2.0
calls per query: 20
answers without a score: 0
distinct scores per query, median: 4
share of products scored 100: 0.390
products the llm labels exact: 251
products people judged exact: 310
products the llm labels partial: 345
products people judged partial: 265
products the llm labels irrelevant: 4
products people judged irrelevant: 7
products nobody judged: 18
queries where most of the reranked top 10 shares its score: 30
semantic, test ndcg@10: 0.830
semantic, test judged@10: 0.990
semantic, unjudged places in the test top 10s: 3
llm pointwise, test ndcg@10: 0.837
llm pointwise, test judged@10: 0.983
llm pointwise, unjudged places in the test top 10s: 5
llm pointwise, mean difference: 0.008
llm pointwise, wins, losses, ties: 11, 8, 11
llm pointwise, permutation p: 0.241
llm pointwise, established at 0.01: no
llm pointwise, t-test p: 0.244
llm pointwise, t-test agrees: yes
llm pointwise, 99% interval: -0.007 to 0.026
llm pointwise, interval agrees: yes
录制时每个查询调用了 20 次,有 0 个回答没有给出分数。分数的粒度很粗。对于 20 个商品,中位数查询只得到 4 个不同的分数,所有商品中有 0.390 得分为 100。在 30 个查询中,重排序后前 10 名里的大多数商品与另一个商品分数相同,因此用来打破平局的语义检索顺序在很大程度上决定了列表的排法。
在这个提示和这些列表上,LLM 判为 exact 的商品比人工判断的少:
| 标签 | LLM | 人工 |
|---|---|---|
| exact | 251 | 310 |
| partial | 345 | 265 |
| irrelevant | 4 | 7 |
| 无人判断 | – | 18 |
在这些列表上,这一结论成立,但第 4 章表明,LLM 与人工判断的分歧方向会随所判断的查询-商品对和提示而变化,因此不要把这一结论推广到其他场合。
逐点重排序同样没有证明带来了提升:它的 test ndcg@10 为 0.837,语义检索为 0.830,置换检验 p 值为 0.241(见决策表)。
LLM 的排序使前 10 名中出现了更多未被判断的商品:为 5 个位置,而语义检索为 3 个,judged@10 从 0.990 降到 0.983。
工作准则: 如果用 LLM 做逐点重排序,就要求它给出分级标签或分数,统计每个查询给出的不同分数的个数,并事先确定打破平局的方式:沿用检索的顺序是稳妥的默认做法。评估结果时使用人工判断,不要使用同一个 LLM 给出的等级。
用一条 LLM 提示词对 20 个商品排序
第三种方法是列表式(listwise)提示词:给候选项编号,并要求 LLM 按相关性顺序返回这些编号。RankGPT 将这种做法作为排列生成(permutation generation)提出(Sun et al., 2023)。一条提示词能容纳的商品数量有限,因此较长的列表要用滑动窗口重排:RankGPT 对 BM25 的前 100 个结果重排,采用“窗口大小为 20、步长为 10”,从列表底部向上进行(第 3.2 节和第 6.1 节)。答案必须是一个有效的排序,而这一点可能做不到:在 2023 年的中等规模 LLM 上,例如 200 亿参数的 Flan-UL2,列表式提示词几乎总是失败,原因是排序有缺失、重复或被拒绝(Qin et al., 2024)。
商店的 LLM 每条提示词看到 20 个商品,即一个窗口。提示词把每个商品显示为一短行:名称、类别、属性和描述的前 160 个字符。最后一条指令写道:“Order all 20 products from the best match for the query to the worst. Return the product numbers, each exactly once, in JSON.”JSON 模式要求恰好 20 个数字,但不要求这 20 个数字互不相同,因此记录环节会修复任何含有重复或缺失编号的答案,并加以标记。按 LLM 在看到按语义顺序排列的商品时给出的答案,对语义检索的前 20 个结果重新排序。这次重排是第四项计划内比较,p 值需低于 0.01:
1"""Reads the listwise LLM's answers for the test queries, in both orders, and reranks semantic retrieval's top 20 by the2answer given in semantic order. The prose's point rests on the planned comparison's p-value, and on the repair count,3which shows how many answers needed a fix."""4from lib.metrics import evaluate5from lib.ranking import DEPTH, report_planned6from lib.recorded import llm_listwise, manifest7from lib.semantic import SemanticSearch8from lib.store import load_queries, qrels910queries, judgments = load_queries("test"), qrels("test") # scored on the human judgments, not the LLM's11semantic = SemanticSearch().run(queries, k=100)12answers = llm_listwise() # one call per query and order shown13# the answers hold one row per product, so keep one row per call: one call per query and order14calls = answers.drop_duplicates(["query_id", "order"])15print(f"listwise calls, both orders: {len(calls)}")16# valid is false where the recording repaired a repeated or missing number17print(f"answers that needed repair: {(~calls['valid']).sum()}")18print(f"products per call: {answers.groupby(['query_id', 'order']).size().max()}")19print("prompt asks for:", manifest("llm-rerank")["prompts"]["listwise"].splitlines()[-1].replace("{n}", str(DEPTH)))2021shown_in_semantic_order = answers[answers["order"] == "semantic"].sort_values(["query_id", "position"])22order = shown_in_semantic_order.groupby("query_id")["product_id"].apply(list)23# the reranked top 20, then the rest of semantic retrieval's list in place: ranking[DEPTH:] starts at the 21st product24reranked = {query_id: order[query_id] + ranking[DEPTH:] for query_id, ranking in semantic.items()}25before, after = evaluate(semantic, judgments), evaluate(reranked, judgments)26for name, frame in [("semantic", before), ("llm listwise", after)]:27 print(f"{name}, test ndcg@10: {frame['ndcg@10'].mean():.3f}")28 print(f"{name}, test judged@10: {frame['judged@10'].mean():.3f}")29 print(f"{name}, unjudged places in the test top 10s: {round((1 - frame['judged@10']).sum() * 10)}")30moved = sum(reranked[query_id][0] != semantic[query_id][0] for query_id in queries.index)31print(f"queries whose first product changes: {moved}")32report_planned("llm listwise", after["ndcg@10"], before["ndcg@10"])
python llm-listwise-rerank.pylistwise calls, both orders: 60
answers that needed repair: 0
products per call: 20
prompt asks for: Order all 20 products from the best match for the query to the worst. Return the product numbers, each exactly once, in JSON.
semantic, test ndcg@10: 0.830
semantic, test judged@10: 0.990
semantic, unjudged places in the test top 10s: 3
llm listwise, test ndcg@10: 0.819
llm listwise, test judged@10: 0.983
llm listwise, unjudged places in the test top 10s: 5
queries whose first product changes: 19
llm listwise, mean difference: -0.010
llm listwise, wins, losses, ties: 12, 12, 6
llm listwise, permutation p: 0.580
llm listwise, established at 0.01: no
llm listwise, t-test p: 0.498
llm listwise, t-test agrees: yes
llm listwise, 99% interval: -0.070 to 0.016
llm listwise, interval agrees: yes
记录共进行了 60 次列表式调用,每个测试查询两次,其中 0 个答案需要修复。重排使 19 个查询的第一个商品发生了变化。
列表式重排的收益并未得到确认:它的得分为 0.819,对照为 0.830,置换检验的 p 值为 0.580(见全部五项比较)。
Judged@10 在这里同样下降,从 0.990 降到 0.983,未评判位置有 5 个,而语义检索为 3 个。
工作准则: 使用每个列表式答案之前都要检查:统计缺失、重复和未知的编号,并记录有多少答案需要修复。如果运行环境支持,就用模式约束答案的形状。测量之前先固定窗口、步长和展示商品的顺序,因为顺序会改变答案,下一项检查会说明这一点。
这部分尚未翻译,因此以英文显示。
Reverse the input to measure positional bias
A listwise answer can suffer from positional bias: an LLM's tendency to let a product's place in the prompt affect its rank, not the click position bias of shoppers. "Large language models (LLMs) exhibit positional bias in how they use context, which especially complicates listwise ranking" (Tang et al., 2024). On TREC DL19, the 2019 queries of TREC's Deep Learning track, a passage-ranking benchmark, gpt-3.5-turbo reranked BM25's top 100 to an nDCG@10 of 65.80 when shown BM25's order, 32.77 when shown it reversed and 25.17 when shown a random order (Sun et al., 2023, Table 5). The direct check is to show the same products in two orders and compare the answers. Kendall's tau measures how alike two orderings are, from 1 for the same order to −1 for the reverse, by counting the pairs they put the same way round (chapter 4).
The recording sent each test query's listwise prompt to the same LLM twice. The first prompt showed semantic retrieval's top 20 in semantic order, as the listwise rerank used; the second showed them reversed. Out of the 30 test queries, how often will the LLM put the same product first both times? Write your number down.
1"""Shows the listwise LLM the same test products in two orders, and measures how far its answers agree and how far each2answer follows the order it was shown. The prose's point rests on the mean tau between the two answers, and on the two3taus between each answer and its own input order."""4import numpy as np5from scipy.stats import kendalltau67from lib.metrics import evaluate8from lib.ranking import DEPTH9from lib.recorded import llm_listwise10from lib.semantic import SemanticSearch11from lib.store import load_queries, qrels1213queries, judgments = load_queries("test"), qrels("test")14semantic = SemanticSearch().run(queries, k=DEPTH) # the order the first prompt showed15answers = llm_listwise().sort_values(["query_id", "order", "position"])16# the products in the order the LLM returned them, for each query and each order shown17answer = {key: list(group["product_id"]) for key, group in answers.groupby(["query_id", "order"])}1819def tau(first: list, second: list) -> float:20 """Kendall's tau between two orderings of the same products."""21 place = {product_id: rank for rank, product_id in enumerate(second)} # each product's place in the second ordering22 # the first ordering's positions, against where the second ordering puts each product23 return kendalltau(range(len(first)), [place[product_id] for product_id in first]).statistic2425between, shown_semantic, shown_reversed, overlap, same_first = [], [], [], [], 026for query_id, shown in semantic.items():27 forward, backward = answer[(query_id, "semantic")], answer[(query_id, "reversed")]28 between.append(tau(forward, backward))29 shown_semantic.append(tau(forward, shown))30 shown_reversed.append(tau(backward, shown[::-1])) # the reversed input, as the second prompt showed it31 overlap.append(len(set(forward[:10]) & set(backward[:10])))32 same_first += forward[0] == backward[0] # index 0 is the first place33print(f"test queries: {len(semantic)}")34print(f"tau between the two answers, mean: {np.mean(between):.3f}")35print(f"tau between the two answers, lowest: {min(between):.3f}\ntau between the two answers, highest: {max(between):.3f}")36print(f"top-10 products in common, mean: {np.mean(overlap):.3f}")37print(f"top-10 products in common, fewest: {min(overlap)}")38print(f"queries with the same first product: {same_first}")39print(f"share of queries with the same first product: {same_first / len(semantic):.3f}")40print(f"tau between the order shown and the answer, semantic order: {np.mean(shown_semantic):.3f}")41print(f"tau between the order shown and the answer, reversed order: {np.mean(shown_reversed):.3f}")42# an answer that ignored the order would repeat itself, so its tau with the reversed input is minus its tau with the semantic order43print(f"the same, if the order shown made no difference: {-np.mean(shown_semantic):.3f}")44reversed_run = {query_id: answer[(query_id, "reversed")] for query_id in semantic}45print(f"reversed answer, test ndcg@10, not a planned comparison: {evaluate(reversed_run, judgments)['ndcg@10'].mean():.3f}")
python listwise-position-bias.pytest queries: 30
tau between the two answers, mean: 0.311
tau between the two answers, lowest: -0.484
tau between the two answers, highest: 0.811
top-10 products in common, mean: 6.967
top-10 products in common, fewest: 3
queries with the same first product: 13
share of queries with the same first product: 0.433
tau between the order shown and the answer, semantic order: 0.414
tau between the order shown and the answer, reversed order: 0.045
the same, if the order shown made no difference: -0.414
reversed answer, test ndcg@10, not a planned comparison: 0.826
The same product comes first for 13 of the 30 queries, so the expectation that a capable model finds the same best match whichever way the list comes holds for only a share of 0.433 of the queries.
The two answers agree with a mean tau of 0.311, and per query the tau runs from -0.484 to 0.811. Their top 10s share 6.967 products on average, as few as 3.
The answers also lean toward the order shown. Each answer's tau against its own input order is 0.414 for the semantic order and 0.045 for the reversed one. An LLM that ignored the order would give the same answer both times, so its reversed answer would have a tau of -0.414 against the reversed input. The reversed answer sits far nearer its own input order than that: positional bias, for this model, prompt and these queries. The reversed answer scores 0.826 nDCG@10; it is not a planned comparison, so the chapter doesn't test it.
Three remedies exist. Pairwise ranking prompting asks the LLM about each pair twice, once in each order, and counts a disagreement as a tie (Qin et al., 2024). Permutation self-consistency reranks several shuffled orders and aggregates the answers, which raised scores by up to 7 to 18% for GPT-3.5 and 8 to 16% for LLaMA v2 (70B) (Tang et al., 2024). And rerankers fine-tuned for listwise ranking, such as the open RankZephyr, are reported to be "resilient against variations in initial document ordering" (Pradeep et al., 2023); the store didn't record one.
Working rule: Before you trust a listwise LLM reranker, run each query in at least two orders and report how much the answers agree. When agreement is low, aggregate several orders or ask each pair in both orders, and count those calls against your budget. In production, show the products in retrieval's order every time, so that the same list at least gets the same answer.
根据预算设定重排深度
搜索流水线分阶段运行,每个阶段都比上一阶段更慢,获得的候选项也更少,而本章中的每个重排器都会增加一个阶段。多阶段排序器提供了“通过控制候选项进入各流水线阶段的准入,在质量与延迟之间进行权衡的能力”(Nogueira et al., 2019)。因此,重排深度,即重排器对每个查询读取的商品数量,是一项预算决策。耗时会随硬件和运行时的选择而变化(Peng et al., 2025),所以本节转而统计决定耗时的因素:每个查询的模型调用次数和所读取的文本量,再乘以模型的规模。商店的交叉编码器有 149M 个参数(模型卡),其 LLM 约有 120 亿个参数(模型卡),约为前者的 80 倍。
在深度为 20 时,根据商品文本和已记录的提示词,统计每个重排器对每个测试查询的模型调用次数和读取的字符数,另外再统计商店未记录的两种设计所需的调用次数:
1"""Counts the characters each reranker reads per test query at depth 20, from the products' text and the recorded prompts,2and the model calls each design makes. The prose's point rests on the characters and calls lines, and on the two designs3the store didn't record."""4import numpy as np56from lib.agreement import as_the_llm_saw7from lib.ranking import DEPTH, listwise_line8from lib.recorded import manifest9from lib.semantic import SemanticSearch10from lib.store import load_products, load_queries, product_text1112products, queries = load_products(), load_queries("test")13semantic = SemanticSearch().run(queries, k=DEPTH) # the 20 products each reranker reads14prompts, text = manifest("llm-rerank")["prompts"], product_text(products)1516cross_encoder, pointwise, listwise = [], [], [] # characters read per query17for query_id, query in queries["query"].items():18 shown = products.loc[semantic[query_id]]19 # one call per product, each reading the query and that product's text together20 cross_encoder.append(sum(len(query) + len(text[product_id]) for product_id in semantic[query_id]))21 # the pointwise prompt, filled in with the query and the product as the LLM saw it22 pointwise.append(sum(len(prompts["pointwise"].format(query=query, product=as_the_llm_saw(product)))23 for _, product in shown.iterrows()))24 # products numbered from 1, as the listwise prompt shows them25 lines = "\n".join(f"[{number}] {listwise_line(product)}" for number, (_, product) in enumerate(shown.iterrows(), 1))26 listwise.append(len(prompts["listwise"].format(query=query, products=lines, n=DEPTH)))2728# the cross-encoder and the pointwise LLM make one call per product; the listwise LLM makes one call per query29for name, calls, characters in [("cross-encoder", DEPTH, cross_encoder), ("llm pointwise", DEPTH, pointwise),30 ("llm listwise", 1, listwise)]:31 print(f"{name}, model calls per query: {calls}")32 print(f"{name}, characters read per query, mean: {np.mean(characters):,.0f}")33print(f"llm pairwise, every pair in both orders, calls per query: {DEPTH * (DEPTH - 1)}")34# a sliding window of 20 products, moved 10 at a time: its starts are list positions 0, 10, ..., 8035window, step, depth = 20, 10, 100 # a sliding window over a deeper list36print(f"llm listwise over the top {depth}, window {window}, step {step}, calls per query: {(depth - window) // step + 1}")
python reranker-prompt-sizes.pycross-encoder, model calls per query: 20
cross-encoder, characters read per query, mean: 9,911
llm pointwise, model calls per query: 20
llm pointwise, characters read per query, mean: 17,864
llm listwise, model calls per query: 1
llm listwise, characters read per query, mean: 5,973
llm pairwise, every pair in both orders, calls per query: 380
llm listwise over the top 100, window 20, step 10, calls per query: 9
| 重排器,前 20 | 每个查询的模型调用次数 | 每个查询读取的字符数 |
|---|---|---|
| 交叉编码器 | 20 | 9,911 |
| LLM,逐点 | 20 | 17,864 |
| LLM,列表式 | 1 | 5,973 |
逐点方式会重复指令,并完整展示每个商品,因此读取的文本最多,且调用相互独立。列表式读取的文本最少,只需一次调用,但必须按顺序写出 20 个数字。若采用成对提示,对所有商品对按两种顺序各比较一次,则每个查询需要 380 次调用;若采用 RankGPT 的滑动窗口处理前 100 个候选项,则需要 9 次。在 TREC DL19 上的一项已发表测量中,使用 Flan-T5-XL 在一块 RTX A6000 GPU 上对 BM25 的前 100 个结果重排,逐点的“是或否”提示得到 0.650 的 nDCG@10,每个查询耗时 1.5 秒;对所有商品对的成对提示得到 0.713,耗时 254.9 秒,共 9,900 次 LLM 调用(Zhuang et al., 2024b)。以交叉编码器作为特征的 LambdaMART 需要为每个候选项计算交叉编码器得分,每个查询的商品对数中位数为 131。
对于交叉编码器,成本随深度同步增长,所以问题在于质量在何处停止增长。在训练查询上扫描不同的深度:
1"""Reranks semantic retrieval's top 5 to 75 with the recorded cross-encoder scores, on the train queries, and prints nDCG@102at each depth. The prose's point rests on the change from the depth before, which shows where the gain flattens and then3falls."""4from lib.metrics import evaluate5from lib.ranking import rerank6from lib.recorded import cross_encoder_scores7from lib.semantic import SemanticSearch8from lib.store import load_queries, qrels910queries, judgments = load_queries("train"), qrels("train") # choosing a depth is tuning: train queries11semantic = SemanticSearch().run(queries, k=100)12recorded = cross_encoder_scores()13# the recorded cross-encoder score of each (query, product) pair14score = dict(zip(zip(recorded["query_id"], recorded["product_id"]), recorded["score"]))15train_pairs = recorded[recorded["query_id"].isin(queries.index)]1617print(f"depth 0 (semantic alone), train ndcg@10: {evaluate(semantic, judgments)['ndcg@10'].mean():.3f}")18previous = round(evaluate(semantic, judgments)["ndcg@10"].mean(), 3) # changes between the printed values19for depth in [5, 10, 20, 30, 50, 75]:20 mean = round(evaluate(rerank(semantic, score, depth), judgments)["ndcg@10"].mean(), 3)21 print(f"depth {depth}, train ndcg@10: {mean:.3f}")22 print(f"depth {depth}, change from the depth before: {mean - previous:+.3f}")23 previous = mean24# every candidate needs its own cross-encoder score, so this counts the recorded pairs per query25print(f"pairs per query to score every candidate, median: {train_pairs.groupby('query_id').size().median():.0f}")
python rerank-depth-on-train.pydepth 0 (semantic alone), train ndcg@10: 0.772
depth 5, train ndcg@10: 0.782
depth 5, change from the depth before: +0.010
depth 10, train ndcg@10: 0.784
depth 10, change from the depth before: +0.002
depth 20, train ndcg@10: 0.802
depth 20, change from the depth before: +0.018
depth 30, train ndcg@10: 0.804
depth 30, change from the depth before: +0.002
depth 50, train ndcg@10: 0.793
depth 50, change from the depth before: -0.011
depth 75, train ndcg@10: 0.788
depth 75, change from the depth before: -0.005
pairs per query to score every candidate, median: 131
交叉编码器的增益在 20 到 30 之间趋于平缓,接近为与 LLM 记录相匹配而确定的深度。从语义检索的 0.772 起,重排前 5 个得到 0.782,重排前 20 个得到 0.802。前 30 个再增加 +0.002,所需的商品对数则比之前多出一半。再往深处,结果下降:深度 50 时为 0.793,深度 75 时为 0.788。
在测试集上,深度为 20 的重排并未证实有增益,所以对这家商店而言,成本最低的重排器,也就是不使用重排器,同样是应当保留的方案。
工作准则: 对你考虑的每个重排器,统计每个深度下每个查询的调用次数和字符数,并在训练查询上设定深度。这是经验法则,而非测量结果:重排深度只取到训练集 nDCG@10 不再提升的位置,因为超过该位置的每个商品成本相同,却没有任何收益。然后在你自己的硬件和流量上为该配置计时,只有在它满足延迟预算时才保留这一阶段。
将交叉编码器蒸馏为小型排序器
对预算而言过慢的重排器,可以用来训练一个快速的排序器。蒸馏(distillation)训练一个小模型,即学生模型(student),去模仿一个大模型,即教师模型(teacher)。Hinton 等人将其描述为把一组集成模型的知识蒸馏进单个模型(Hinton et al., 2015)。在排序任务中,教师模型的分数成为学生模型的标签。当两个模型的分数量级不同时(不同架构的分数往往如此),学生模型可以改为学习两个商品分数之间的差值,所用的损失称为 Margin-MSE(Hofstätter et al., 2020)。蒸馏也可以从大语言模型向下进行。RankGPT 报告称,一个经蒸馏的 440M 模型在 BEIR 基准上优于一个 3B 的有监督模型(Sun et al., 2023;BEIR)。在一项针对含最高级表达的查询的商品搜索研究中,轻量模型接管了大语言模型的解读,而直接用大语言模型重排的延迟高到无法接受(Zhu et al., 2025)。教师模型不需要人工标签,并且为每个候选项打标签。
教师模型是交叉编码器。学生模型是一个 LightGBM 回归模型,使用7特征,这些特征除检索本身已经进行的那一次调用外,不需要再调用任何模型:即除 cross_encoder 之外的全部特征。它的标签是教师模型的原始分数,树回归可以在教师模型自身的尺度上拟合这一分数。它对同样的候选项按 product_id 的顺序排序,树的规模取为 LambdaMART 所选的大小。首先,通过对训练查询做交叉验证,将学生模型与一个使用相同特征、在人工评分上训练的 LambdaMART 模型进行比较:
1"""Trains a student LightGBM regression on the cross-encoder's scores, and a LambdaMART model on human grades with the same2features, both by cross-validation on the train queries. The prose's point rests on the student's score, and on its3difference from the human-graded ranker."""4from lightgbm import LGBMRanker, LGBMRegressor56from lib.metrics import evaluate7from lib.ranking import CHEAP_FEATURES, DETERMINISTIC, LAMBDARANK, TREES, cross_validate, feature_table, ranked8from lib.semantic import SemanticSearch9from lib.significance import compare10from lib.store import load_products, load_queries, qrels1112queries, judgments = load_queries("train"), qrels("train") # design work: train queries only13candidates = feature_table(load_products(), queries)14# unjudged candidates count as grade 0, as in the metric15candidates["grade or 0"] = candidates["grade"].fillna(0).astype(int)1617student = cross_validate(lambda: LGBMRegressor(**TREES, **DETERMINISTIC), candidates, candidates,18 "cross_encoder", CHEAP_FEATURES, judgments) # labels from the teacher19human = cross_validate(lambda: LGBMRanker(**LAMBDARANK, **TREES, **DETERMINISTIC), candidates, candidates,20 "grade or 0", CHEAP_FEATURES, judgments) # labels from people21# the teacher's own order of the candidates, ties broken by product_id22teacher = evaluate(ranked(candidates, candidates["cross_encoder"]), judgments)["ndcg@10"]23semantic = evaluate(SemanticSearch().run(queries, k=100), judgments)["ndcg@10"]24for name, scores in [("semantic", semantic), ("teacher", teacher), ("student, teacher's labels", student),25 ("lambdarank, human grades, same features", human)]:26 print(f"{name}, train ndcg@10: {scores.mean():.3f}")27# the student's score minus the human-graded ranker's, per query, tested by permutation28result = compare(student, human)29print(f"student minus the human-graded ranker: {result['mean difference']:.3f}")30print(f"student minus the human-graded ranker, permutation p: {result['permutation p']:.3f}")
python distillation-on-train.pysemantic, train ndcg@10: 0.772
teacher, train ndcg@10: 0.785
student, teacher's labels, train ndcg@10: 0.791
lambdarank, human grades, same features, train ndcg@10: 0.747
student minus the human-graded ranker: 0.044
student minus the human-graded ranker, permutation p: 0.002
在这些训练查询和这些特征上,教师模型对每个候选项给出的标签训练出的排序器,优于人工评分训练出的排序器,其中未经评判的候选项按 0 计。学生模型的得分为 0.791,教师模型自身对候选项的排序得分为 0.785。
在人工评分上训练的模型得分为 0.747,低于语义检索的 0.772,学生模型减去它所得的差为 0.044,置换检验的 p 值为 0.002。这是在训练查询上做的调参检查,不构成证据。
然后在训练查询上拟合学生模型,并使用与 LambdaMART 相同的留出训练查询做早停,再进行最后一项计划内比较,阈值同为 0.01:
1"""Fits a student LightGBM regression on the teacher's scores for the train queries, then reports its gain shares, how2closely its order follows similarity and the teacher, and its planned comparison on the test queries. The prose's point3rests on that comparison and on the taus printed for each ranking."""4import numpy as np5from lightgbm import LGBMRegressor6from scipy.stats import kendalltau78from lib.metrics import evaluate9from lib.ranking import CHEAP_FEATURES, DETERMINISTIC, TREES, feature_table, fit, ranked, report_planned, stopping_split10from lib.semantic import SemanticSearch11from lib.store import load_products, load_queries, qrels1213products, train, test = load_products(), load_queries("train"), load_queries("test")14table = feature_table(products, train) # the teacher's score is the label: no human grades15print(f"train candidates the teacher labels: {len(table)}")16print(f"of them, judged by nobody: {table['grade'].isna().sum()}")17print(f"student features: {len(CHEAP_FEATURES)}")18# early stopping on the same 10 held-out train queries that LambdaMART stops on19student = fit(LGBMRegressor(**TREES, **DETERMINISTIC), table, "cross_encoder", CHEAP_FEATURES, *stopping_split(train.index))20print(f"trees kept by early stopping: {student.best_iteration_}")2122candidates, judgments = feature_table(products, test), qrels("test") # test queries: read once, to report23predicted = student.predict(candidates[CHEAP_FEATURES])24gain = student.booster_.feature_importance(importance_type="gain")25# the three features with the most gain26for name, share in sorted(zip(CHEAP_FEATURES, gain / gain.sum()), key=lambda pair: -pair[1])[:3]:27 print(f"student's share of gain, {name}: {share:.3f}")28for name in ["cross_encoder", "similarity"]: # whose order does the student follow?29 # candidates has a default index, so group.index picks this query's rows out of predicted30 tau = [kendalltau(group[name], predicted[group.index]).statistic for _, group in candidates.groupby("query_id")]31 print(f"tau between student and {name}, mean over test queries: {np.mean(tau):.3f}")32teacher = evaluate(ranked(candidates, candidates["cross_encoder"]), judgments)33print(f"teacher's own order of the candidates, test ndcg@10, not a planned comparison: {teacher['ndcg@10'].mean():.3f}")34after = evaluate(ranked(candidates, predicted), judgments)35before = evaluate(SemanticSearch().run(test, k=100), judgments)36for name, frame in [("semantic", before), ("distilled", after)]:37 print(f"{name}, test ndcg@10: {frame['ndcg@10'].mean():.3f}")38 print(f"{name}, test judged@10: {frame['judged@10'].mean():.3f}")39 # each query's top 10 is 10 places, so (1 - judged@10) times 10 counts its unjudged places40 print(f"{name}, unjudged places in the test top 10s: {round((1 - frame['judged@10']).sum() * 10)}")41report_planned("distilled", after["ndcg@10"], before["ndcg@10"])
python distilled-ranker-on-test.pytrain candidates the teacher labels: 6896
of them, judged by nobody: 1975
student features: 7
trees kept by early stopping: 202
student's share of gain, similarity: 0.812
student's share of gain, name_match: 0.126
student's share of gain, bm25: 0.055
tau between student and cross_encoder, mean over test queries: 0.466
tau between student and similarity, mean over test queries: 0.753
teacher's own order of the candidates, test ndcg@10, not a planned comparison: 0.782
semantic, test ndcg@10: 0.830
semantic, test judged@10: 0.990
semantic, unjudged places in the test top 10s: 3
distilled, test ndcg@10: 0.826
distilled, test judged@10: 0.987
distilled, unjudged places in the test top 10s: 4
distilled, mean difference: -0.004
distilled, wins, losses, ties: 10, 13, 7
distilled, permutation p: 0.720
distilled, established at 0.01: no
distilled, t-test p: 0.694
distilled, t-test agrees: yes
distilled, 99% interval: -0.034 to 0.017
distilled, interval agrees: yes
教师模型为全部 6896 个训练候选项打了标签,其中 1975 个无人评判,早停保留了 202 棵树。
学生模型主要依赖 similarity,它占增益的 0.812,其后是 name_match(0.126)和 bm25(0.055)。在测试候选项上,它的分数与语义相似度在各查询上的平均 tau 为 0.753,与教师模型的一致程度为 0.466。可见学生模型学到的主要是语义检索的排序,并在名称匹配和 BM25 匹配的推动下向教师模型的排序靠拢。
学生模型并未证明有所提升:蒸馏排序器的 nDCG@10 为 0.826,语义检索为 0.830,置换检验的 p 值为 0.720(见决策表)。教师模型对测试候选项的自身排序,本章并未计划检验,得分为 0.782。
学生模型的 Judged@10 为 0.987,语义检索为 0.990;学生模型的前 10 名中有 4 个未经评判的位置,语义检索则为 3。
工作准则: 当重排器带来了已确立的提升、但每次查询的成本过高时,就进行蒸馏:用教师模型为训练查询的每个候选项打标签,用无需教师模型即可计算的特征训练学生模型,并在留出查询上将学生模型与你的检索做对比检验。查看学生模型的增益重要性:当检索自身的分数在其中占主导时,学生模型基本上就是你的检索。当学生模型无法匹配教师模型的尺度时,使用成对商品之间的差值。商店目前还没有胜过语义检索的教师模型,因此保留代码,不上线学生模型。
生产环境中会出什么问题
线上排序器在购物者等待时计算特征并调用模型,而它所用的代码和模型版本,可能与训练和测试时所用的版本不一致。
特征在训练与服务之间发生漂移。 如果线上系统的分词方式不同、对 BM25 重新缩放,或升级了嵌入模型,已学到的权重面对的就是不同的数值,而且不会有任何报错。应通过同一条共享的代码路径计算特征,并将特征与每次返回的排序一并记录。
行顺序会泄漏到训练中。 LightGBM 的 lambdarank 从输入行的顺序出发,而排序总要以某种方式打破并列。按某个重排序器的得分排好的文件,会把该重排序器的排序交给模型。在每次拟合和每次打分之前,应按 ID 对候选排序,或用固定种子将其打乱。
判断来自一个候选池。 WANDS 只对其自身系统给出的商品做了判断(第 2 章)。把未经判断的候选视为不相关,会教会模型压低那些系统没有找到的商品。对你加入的每个排序器,都应检查 judged@10,并在相信某项提升之前,先让其未经判断的商品得到判断。
LLM 评判者偏向 LLM 重排序器。 在一项研究中,基于 LLM 评估者构建的排名,平均将基于 LLM 的重排序器高估了 9 至 17 位(Fröbe et al., 2025),而同时用 LLM 排序和判断是循环论证,“说明了对基于 LLM 的评估的依赖如何扭曲系统排名”(Clarke and Dietz, 2024)。应像本章这样,用人工判断为 LLM 重排序器打分。
LLM 重排序器对应唯一的模型版本和唯一的提示词。 不同的摘要值(标识确切权重的哈希)、量化方式(权重存储的精度)或提示词措辞,都可能改变每一个答案;再加上位置偏差,检索顺序的变化同样可能如此。LLM 记录的清单逐字记录了摘要值和两个提示词;两者中任何一个变化时,都应重新度量。
不熟悉的查询可能表现更差。 一项对 22 种重排序方法的比较发现,LLM 重排序器在熟悉的查询上表现更好,而“它们对新查询的泛化能力不一”(Abdallah et al., 2025)。应留意涉及比模型训练数据更新的商品和风格的查询。
每个重排序器都会在购物者的路径上增加一次调用。 应像预算中那样统计每次查询的调用次数和文本量,设置超时,并在重排序器未能及时响应时回退到检索的顺序。
工作准则: 随每次排序记录特征和模型版本,将候选按不携带任何信息的顺序排列,依据人工判断打分,检查 judged@10,并为每个重排序阶段设定时限和回退方案。
本章所做的决定
本章的每一项选择都是在训练查询上做出的,只有在测试查询上成立的计划内比较,才能改变商店的搜索。
5 项计划内比较均以 nDCG@10 与语义检索的 0.830 相比较,其成立所依据的 p 值低于 0.01:
| 指标 | LambdaMART | 交叉编码器 | LLM 逐点式 | LLM 列表式 | 蒸馏模型 |
|---|---|---|---|---|---|
| nDCG@10 | 0.817 | 0.813 | 0.837 | 0.819 | 0.826 |
| 平均差值 | -0.012 | -0.016 | 0.008 | -0.010 | -0.004 |
| 胜 / 负 / 平 | 14, 12, 4 | 11, 12, 7 | 11, 8, 11 | 12, 12, 6 | 10, 13, 7 |
| 置换检验 p 值 | 0.260 | 0.178 | 0.241 | 0.580 | 0.720 |
| 是否成立 | no | no | no | no | no |
| t 检验 p 值 | 0.278 | 0.171 | 0.244 | 0.498 | 0.694 |
| 99% 区间 | -0.044 to 0.011 | -0.058 to 0.006 | -0.007 to 0.026 | -0.070 to 0.016 | -0.034 to 0.017 |
| t 检验是否一致 | yes | yes | yes | yes | yes |
| 区间是否一致 | yes | yes | yes | yes | yes |
- 5 项计划内比较:与语义检索相比,基于人工判断,成立所依据的 p 值低于 0.01。
- 候选集:BM25 与语义检索各自前 100 名的并集,按
product_id顺序排列,得分相同时按product_id决定先后。 - LambdaMART:
label_gain=[0, 1, 2],未标注判断的候选按等级 0 处理,3 个叶子,学习率 0.1,通过在训练集上的交叉验证选定,并借助eval_X与eval_y,在留出的训练查询上提前停止。它的增益在测试集上未能成立,因此不予采用。 - 重排深度:语义检索的前 20 名,为与 LLM 的记录保持一致而固定;在训练集上,交叉编码器的曲线在 20 到 30 之间趋于平缓。没有任何一个重排器的增益在测试集上成立,因此都不采用。
- LLM 重排器只依据人工判断打分,列表式重排器还在两种输入顺序下做了检查。它的回答取决于所展示的顺序。
- 成本只计数,不计时:统计每个查询的调用次数和字符数。
- 蒸馏:以交叉编码器的得分训练 LightGBM 学生模型,除检索本身外不再调用模型。它学到的主要是语义检索的排序,没有显示出成立的增益,因此不予采用。
- 需要超越的排序器仍然只是语义检索,以未做词干提取的 BM25 作为基线。
- **本章的结论:**没有任何第二阶段证明了自己的价值;它们都没有显示出成立的增益。
本章,在本版次中
版次 2026.11.10示例于 2026年11月19日 通过本章 14个示例 个示例中有 14 个通过
译自英文版图书,版次 2026.11.10。