Ranking

阅读约需 92分钟 分钟,12节中含14个示例。

版次 2026.11.10 示例于 2026年11月19日 通过 132个示例中有132个通过 变化内容

AI 翻译。由人工以英文核验。 我们如何制作图书

译自英文版图书,版次 2026.11.10。 阅读原文

这部分尚未翻译,因此以英文显示。

By the end of this chapter you can describe each product that retrieval returns with features, train a learning-to-rank model with LightGBM on your own judgments, rerank the top of the list with a cross-encoder or an LLM, measure an LLM reranker's positional bias, set the rerank depth by counting its costs, and distil a slow reranker into a fast one. You can also decide, with a planned comparison on the test queries, whether any of them beats your retrieval.

Semantic retrieval alone is still the ranker to beat at the store, where every search is ordered by one embedding score and nothing else. It ranks products by embeddings from Snowflake/snowflake-arctic-embed-m-v1.5, and scores 0.830 nDCG@10 on the 30 test queries. No change in chapters 5 to 8 has beaten it by an established margin. This chapter asks whether a second stage can: a model trained on the store's own judgments, a reranker that reads the query and the product together, or an LLM. Chapter 1's cross-encoder reranked the hybrid run, not semantic retrieval, and never saw the store's judgments.

So this chapter tries all three on semantic retrieval's top results, each scored against the store's judgments. Reranking reorders the top of retrieval's list with a slower model. The store's rerankers are recorded: each ran once and saved its outputs, which the examples read.

The store's LLM is the open model gemma4:12b-it-q4_K_M, licensed Apache-2.0. It ran locally with chapter 4's settings, which make it take its likeliest next word every time, plus thinking off, so it answers without writing out reasoning, and a JSON schema, a machine-readable description of the answer's shape. recordings/llm_rerank.py reranked semantic retrieval's top 20 for the test queries only, so nothing about the LLM rerankers can be tuned. Every score uses the human judgments, because in one study LLM judges ranked LLM rerankers too high (What goes wrong in production). These results describe one model and two prompts, so treat them as the store's, not as a verdict on LLM rerankers.

The chapter fixed its 5 planned comparisons on the test queries before it read any test result. Each sets one method against semantic retrieval's run on nDCG@10:

  1. LambdaMART, a learned ranker trained on the train queries' judgments, ordering each query's candidate products
  2. the cross-encoder reranking semantic retrieval's top 20
  3. the LLM scoring each of the top 20 on its own
  4. the LLM ordering the top 20 in one prompt
  5. a small distilled ranker trained on the cross-encoder's scores

The bar is the book's rule: 5 planned comparisons split its 0.05, so each needs a paired permutation test p-value below 0.01. The paired t-test and a 99% bootstrap interval stand beside it.

用特征描述每个候选商品

最近一次修改见于版次 2026.11.10

本章计划进行的比较中,第一项是学习式排序器,它不读取文本。学习式排序器根据一组数字(即特征)为每一对查询和商品打分,并从评判中学习如何对这些特征加权。查询相关特征同时描述查询和商品,例如 BM25 分数。查询无关特征只描述商品本身,例如有多少购物者为它评过分。网页搜索的特征集两类兼有:LETOR 4.0 是学习排序的一个基准,含有 46 个特征,其中有“BM25 of title”“PageRank”和“Number of slash in URL”(Qin and Liu, 2013)。排序器只为候选打分,候选是成本较低的检索阶段返回的商品,因此它的特征可以计算得较为昂贵。

商店每个查询的候选,是交叉编码器的记录所打过分的商品:即 BM25 前 100 名与语义检索前 100 名的并集。lib.ranking.feature_table为每个候选提供 8 个特征:

  • bm25:来自 lib.lexical 的 BM25 分数,该实现是本书未做词干提取的基线;若商品与查询没有共同的词项,则为 0
  • similarity:查询嵌入与商品嵌入的余弦相似度
  • cross_encoder:已记录的交叉编码器分数
  • name_match和attribute_match:查询的词项(按第 5 章的方式分词)在商品名称和商品属性中出现的比例
  • rating_count、average_rating和review_count,由商店的数据提供给每个商品

目录中有 2632 的商品没有评分,它们的三个评分特征保持缺失。该表还包含每个候选的人工等级;若无人评判,则留空。为训练查询构建该表,先查看一个查询,再按等级比较各个特征:

ranking-features-by-grade.py Python 3.12.15 · bm25s 0.3.16 · lightgbm 4.7.0 · numpy 2.5.5 · pandas 3.1.1 · pystemmer 3.1.0 · scikit-learn 1.9.1 · scipy 1.18.1 通过
"""Builds the feature table for the train queries, shows one query's candidates, and compares each feature's mean acrossthe human grades. The prose's point rests on the mean cross_encoder line for each grade and the mean of each feature by grade."""import pandas as pdfrom lib.ranking import FEATURES, feature_tablefrom lib.store import load_products, load_queriesproducts, queries = load_products(), load_queries("train")      # train queries: features are design worktable = feature_table(products, queries)print(f"features: {len(FEATURES)}")print(f"train candidates: {len(table)}")print(f"train candidates per query, median: {table.groupby('query_id').size().median():.0f}")print(f"share of train candidates nobody judged: {table['grade'].isna().mean():.3f}")query_id = queries.index[queries["query"] == "laundry basket with wheels"][0]one = table[table["query_id"] == query_id].sort_values("cross_encoder", ascending=False)one = one.assign(name=products.loc[one["product_id"], "name"].str[:38].to_numpy())pd.set_option("display.width", 200)print(f"\ncandidates for laundry basket with wheels: {len(one)}")print(one[["name", "bm25", "similarity", "cross_encoder", "name_match", "attribute_match", "rating_count", "grade"]]      .head(6).round(3).to_string(index=False))print(f"top 6 by cross-encoder, exact: {(one['grade'].head(6) == 2).sum()}")# grades 2, 1 and 0 are exact, partial and irrelevant; a missing grade means nobody judged the pairlabel = table["grade"].map({2: "exact", 1: "partial", 0: "irrelevant"}).fillna("unjudged")means = table[FEATURES].groupby(label).mean().loc[["exact", "partial", "irrelevant", "unjudged"]]print("\nmean of each feature by grade, train candidates:")print(means.round(3).to_string())for grade in means.index:    print(f"mean cross_encoder, {grade}: {means.at[grade, 'cross_encoder']:.3f}")relevance = ["bm25", "similarity", "cross_encoder", "name_match"]nearer = [name for name in relevance if abs(means.at["unjudged", name] - means.at["irrelevant", name])          < abs(means.at["unjudged", name] - means.at["partial", name])]print(f"relevance features where unjudged sits nearer irrelevant than partial: {len(nearer)} of {len(relevance)}")medians = table[["rating_count", "review_count"]].groupby(label).median().loc[means.index]  # counts are skewedfor grade in ["exact", "partial", "irrelevant"]:    print(f"rating_count, {grade}, mean: {means.at[grade, 'rating_count']:.3f}")    print(f"rating_count, {grade}, median: {medians.at[grade, 'rating_count']:.1f}")print(f"products with no rating: {products['rating_count'].isna().sum()}")
输出python ranking-features-by-grade.py
features: 8
train candidates: 6896
train candidates per query, median: 131
share of train candidates nobody judged: 0.286

candidates for laundry basket with wheels: 125
                                  name  bm25  similarity  cross_encoder  name_match  attribute_match  rating_count  grade
       laundry sorter basket on wheels 8.229       0.597          3.433       1.000              0.0           NaN    2.0
                vintage laundry basket 6.270       0.572          2.995       0.667              0.0         638.0    2.0
         rolling wicker laundry basket 5.494       0.588          2.974       0.667              0.0         110.0    2.0
aluminum frame laundry basket can be f 4.739       0.524          2.941       0.667              0.0          14.0    2.0
    rebrilliant rolling laundry hamper 5.537       0.558          2.804       0.333              0.0          17.0    2.0
4-bag laundry hamper sorter cart baske 6.012       0.543          2.739       0.667              0.0           NaN    2.0
top 6 by cross-encoder, exact: 6

mean of each feature by grade, train candidates:
             bm25  similarity  cross_encoder  name_match  attribute_match  rating_count  average_rating  review_count
grade
exact       4.317       0.517          2.191       0.562            0.024       466.879           4.537       333.589
partial     3.936       0.454          1.513       0.461            0.030       385.454           4.539       288.779
irrelevant  2.760       0.360          1.159       0.404            0.006       246.781           4.552       159.052
unjudged    2.323       0.369          1.050       0.140            0.018       339.904           4.520       237.027
mean cross_encoder, exact: 2.191
mean cross_encoder, partial: 1.513
mean cross_encoder, irrelevant: 1.159
mean cross_encoder, unjudged: 1.050
relevance features where unjudged sits nearer irrelevant than partial: 4 of 4
rating_count, exact, mean: 466.879
rating_count, exact, median: 43.0
rating_count, partial, mean: 385.454
rating_count, partial, median: 31.0
rating_count, irrelevant, mean: 246.781
rating_count, irrelevant, median: 35.5
products with no rating: 2632

训练查询共有 6896 个候选,每个查询的中位数为 131 个,其中 0.286 无人评判。

laundry basket with wheels有 125 个候选。按交叉编码器分数排在前六位的商品中,有 6 个是完全相关的商品。它们的attribute_match为 0,因为属性中记录的是颜色、材质和尺寸,而不是wheels这样的词。

在全部训练候选中,交叉编码器分数的均值从完全相关商品的 2.191 降到部分相关商品的 1.513 和不相关商品的 1.159,bm25、similarity和name_match也以同样方式下降。average_rating和attribute_match几乎不变。

rating_count的均值随等级下降,但中位数并非如此:少数评分很多的商品决定了均值。

等级 评分数均值 评分数中位数
exact 466.879 43.0
partial 385.454 31.0
irrelevant 246.781 35.5

未评判候选的交叉编码器分数平均为 1.050;在随等级下降的相关性特征中,有 4 of 4 个特征上,它们的均值更接近不相关商品,而不是部分相关商品。这关系到未评判的候选应当教给模型什么。

工作准则: 从你已经在计算的特征入手:每个检索器的分数,若有重排序器则加上它的分数,名称等字段上的匹配情况,以及热度信号。训练之前,先在训练查询上按等级比较每个特征,分数用均值,偏斜的计数用中位数。不随等级变化的特征,能教给模型的很少。训练和上线服务时,要用同一份代码计算特征。

这部分尚未翻译,因此以英文显示。

Learn a ranking from pairs of products

最近一次修改见于版次 2026.11.10

Learning to rank trains a model on judged queries to order candidates by their features. Its methods fall into three families, pointwise, pairwise and listwise (Liu, 2009). A pointwise model predicts each product's grade on its own, as a regression. A pairwise model learns which of two products should rank higher. A listwise model scores a whole ranking. Liu's 2009 comparison on the LETOR benchmark favoured the listwise approach, though tentatively. The obstacle for all three is the metric: nDCG changes only when two products swap places, so ranking metrics "are everywhere either discontinuous or flat, so gradient descent appears to be problematic" (Burges, 2010).

RankNet, a pairwise method, works around it with a smooth loss on pairs. For two products where i should rank above j, its gradient is large when the model scores them the wrong way round or nearly level, and small when it already has them well apart. LambdaRank keeps that gradient and multiplies it by how much nDCG would change if the two products swapped places (Burges, 2010, section 4.1). Burges writes the gradient with a minus sign, as the slope of a cost; here a positive λ pushes i up:

λ_ij    = σ / (1 + exp(σ · (s_i − s_j))) · |ΔnDCG_ij|      s: the model's scores, σ = 1
ΔnDCG_ij = (g_i − g_j) · (1 / log2(1 + r_i) − 1 / log2(1 + r_j)) / ideal DCG      g: grades, r: positions

A product's total is its λ summed over the pairs where it should rank higher, minus those where it should rank lower. Burges (2010) also reads the λ's as forces, pushing products up and down the list.

LambdaMART uses LambdaRank's λ as the gradients for gradient-boosted trees, which add small decision trees one at a time, each fitted to the gradients of the trees before it: "LambdaMART is the boosted tree version of LambdaRank, which is based on RankNet". An ensemble of LambdaMART rankers won Track 1 of the Learning to Rank Challenge that Yahoo ran in 2010 (Burges, 2010).

One train query, outdoor waterproof chest, shows the arithmetic. Let semantic similarity be the model's scores so far, compute every pair's λ in its top 10 by hand, check each |ΔnDCG| by swapping the pair and recomputing nDCG with lib.metrics, and add up the forces:

lambdas-for-one-query.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 通过
"""Works out LambdaRank's λ for every pair in one train query's top 10 by hand, and checks each pair's |ΔnDCG| againstswapping the pair in the metric. The prose's point rests on the strongest push up and down, and on the count of pairswhose formula matches swapping."""import mathfrom itertools import combinationsfrom lib.metrics import dcg_at_k, ndcg_at_kfrom lib.semantic import SemanticSearchfrom lib.store import load_queries, qrelsqueries, judgments = load_queries("train"), qrels("train")query_id = queries.index[queries["query"] == "outdoor waterproof chest"][0]judged = judgments[query_id]scores = SemanticSearch().search(query_id, k=10)       # the model so far: semantic similarityranking, similarity = list(scores.index), scores.to_numpy()# an unjudged product counts as grade 0, as it does in the metricgrades = [judged.get(product_id, 0) for product_id in ranking]# the ideal DCG takes every judged product, as lib.metrics does; sigma is 1, LightGBM's default sigmoidideal, sigma = dcg_at_k(sorted(judged.values(), reverse=True), 10), 1.0print("grades from the top:", " ".join(map(str, grades)))force, checks, pulls = [0.0] * len(ranking), [], []for i, j in combinations(range(len(ranking)), 2):      # i is above j    if grades[i] == grades[j]:        continue    better, worse = (i, j) if grades[i] > grades[j] else (j, i)    ranknet = 1 / (1 + math.exp(sigma * (similarity[better] - similarity[worse])))      # RankNet's pull    # i and j count from 0, so position p is i + 1 and its discount is 1 / log2(p + 1), which is 1 / log2(i + 2)    delta = abs((grades[i] - grades[j]) * (1 / math.log2(i + 2) - 1 / math.log2(j + 2))) / ideal    swapped = ranking.copy()    swapped[i], swapped[j] = swapped[j], swapped[i]    checks.append(math.isclose(delta, abs(ndcg_at_k(swapped, judged, 10) - ndcg_at_k(ranking, judged, 10))))    pulls.append(ranknet)    pair_lambda = sigma * ranknet * delta                         # LambdaRank: scaled by |delta nDCG|    force[better], force[worse] = force[better] + pair_lambda, force[worse] - pair_lambdaprint(f"pairs with different grades: {len(checks)}")print(f"pairs where delta ndcg by formula matches swapping the pair: {sum(checks)}")for position, push in enumerate(force, start=1):    print(f"force at position {position}: {push:+.3f}")# list indexes count from 0 and positions from 1, hence the + 1print(f"strongest push up, position: {force.index(max(force)) + 1}")print(f"strongest push down, position: {force.index(min(force)) + 1}")print(f"ranknet's pull, lowest: {min(pulls):.3f}\nranknet's pull, highest: {max(pulls):.3f}")
输出python lambdas-for-one-query.py
grades from the top: 1 2 2 0 1 2 0 2 0 0
pairs with different grades: 32
pairs where delta ndcg by formula matches swapping the pair: 32
force at position 1: +0.013
force at position 2: +0.159
force at position 3: +0.104
force at position 4: -0.083
force at position 5: -0.010
force at position 6: +0.063
force at position 7: -0.092
force at position 8: +0.063
force at position 9: -0.105
force at position 10: -0.111
strongest push up, position: 2
strongest push down, position: 10
ranknet's pull, lowest: 0.472
ranknet's pull, highest: 0.525

The formula's |ΔnDCG| matches the change from swapping the pair for 32 of the 32 pairs in the top 10 with different grades. The top 10's grades, from the top, are 1 2 2 0 1 2 0 2 0 0; an unjudged product counts as 0, as it does in the metric.

The strongest push up goes to position 2, an exact product, with +0.159; the strongest push down goes to position 10, an irrelevant product, with -0.111. The partial product at position 1 gets a net push of +0.013: the irrelevant products below push it up, and the exact ones push it down.

RankNet's factor in λ stays between 0.472 and 0.525, because the similarities lie close together, so |ΔnDCG| decides almost everything.

That example simplifies. It takes the ideal DCG from every judged product, as lib.metrics does, and looks at ten products. LightGBM takes the ideal from the query's training rows, and at 4.7.0 two defaults change the λ. lambdarank_truncation_level is 30, which limits training to the top 30 results: only pairs with a product in the current top 30 get a λ. lambdarank_norm is true, which normalizes the lambdas across queries; set it to false to get the original lambdarank algorithm (Parameters). Normalizing, as LightGBM's source does it, divides each pair's |ΔnDCG| by the gap between its scores plus 0.01, then rescales each query's λ (rank_objective.hpp). To check the formula against the library, write it as a custom LightGBM objective, grow five small trees on this query's candidates, and grow them again with LightGBM's lambdarank, with and without those defaults:

lambdas-match-lightgbm.py Python 3.12.15 · bm25s 0.3.16 · lightgbm 4.7.0 · numpy 2.5.5 · pandas 3.1.1 · pystemmer 3.1.0 · scikit-learn 1.9.1 · scipy 1.18.1 通过
"""Writes LambdaRank's gradients by hand as a custom LightGBM objective, then checks them against LightGBM's own lambdarankon one query, with and without its defaults. The prose's point rests on the two printed differences: a rounding-sized gapwithout the defaults, and a large one at them."""import numpy as npfrom lightgbm import LGBMRankerfrom lib.ranking import DETERMINISTIC, FEATURES, feature_tablefrom lib.store import load_products, load_queriesqueries = load_queries("train")one = queries[queries["query"] == "outdoor waterproof chest"]rows = feature_table(load_products(), one)                  # one query's candidates, in product_id order# unjudged candidates count as grade 0, as in the metricgrades = rows["grade"].fillna(0).astype(int).to_numpy()SIGMA = 1.0                                                   # LightGBM's default sigmoiddef lambdas(y_true, y_pred, weight, group):    """The chapter's lambda for every pair with different grades, as gradients and second derivatives."""    rank = np.empty(len(y_pred), dtype=int)    rank[np.argsort(-y_pred, kind="stable")] = np.arange(len(y_pred))   # LightGBM's stable sort on score    # rank counts from 0, so the product at position rank + 1 has the discount 1 / log2(rank + 2)    discount, ideal = 1 / np.log2(rank + 2), np.sum(np.sort(y_true)[::-1] / np.log2(np.arange(len(y_true)) + 2))    gradient, hessian = np.zeros(len(y_pred)), np.zeros(len(y_pred))    for i in range(len(y_true)):        for j in np.flatnonzero(y_true < y_true[i]):                      # i should rank above j            delta = (y_true[i] - y_true[j]) * abs(discount[i] - discount[j]) / ideal            pull = 1 / (1 + np.exp(SIGMA * (y_pred[i] - y_pred[j])))            gradient[i], gradient[j] = gradient[i] - SIGMA * pull * delta, gradient[j] + SIGMA * pull * delta            hessian[[i, j]] += SIGMA ** 2 * pull * (1 - pull) * delta    return gradient, hessian# five small trees; min_child_samples is 5 where LightGBM's default is 20, so the trees can split this query's candidatestree = {"n_estimators": 5, "num_leaves": 3, "min_child_samples": 5, **DETERMINISTIC}# turns off normalization and sets truncation to every candidate, so every pair gets a λ, as the formula has itplain = {"lambdarank_norm": False, "lambdarank_truncation_level": len(rows)}   # LightGBM's lambdas, untouchedmodels = {"by hand": LGBMRanker(objective=lambdas, **tree),          "lightgbm, no normalization or truncation": LGBMRanker(objective="lambdarank", label_gain=[0, 1, 2], **plain, **tree),          "lightgbm at its defaults": LGBMRanker(objective="lambdarank", label_gain=[0, 1, 2], **tree)}# raw scores: the values the gradients act on, before any sigmoidscores = {name: model.fit(rows[FEATURES], grades, group=[len(rows)]).predict(rows[FEATURES], raw_score=True)          for name, model in models.items()}print(f"candidates: {len(rows)}")for name in list(models)[1:]:    difference = np.abs(scores[name] - scores["by hand"]).max()    print(f"{name}, largest difference from the lambdas by hand after 5 trees: {difference:.6f}")
输出python lambdas-match-lightgbm.py
candidates: 180
lightgbm, no normalization or truncation, largest difference from the lambdas by hand after 5 trees: 0.000006
lightgbm at its defaults, largest difference from the lambdas by hand after 5 trees: 0.595667

Over the query's 180 candidates, LightGBM without normalization or truncation scores every product within 0.000006 of the custom objective's scores. The gap is rounding, since LightGBM reads its sigmoid from a precomputed table (rank_objective.hpp). At its defaults, scores differ by up to 0.595667.

Working rule: Train rankers with a gradient that weighs each pair by the metric you report, as LambdaMART does, rather than regressing on grades. Know which defaults of your library change that gradient, and check on train queries that it beats a pointwise model on your data, because the families' order in a benchmark doesn't transfer by itself.

用 LightGBM 训练 LambdaMART

最近一次修改见于版次 2026.11.8

LightGBM 通过 LGBMRanker 和 objective="lambdarank" 训练 LambdaMART。有四项设置需要留意,每一项在 4.7.0 中都有文档说明:

  • 行按查询分组。“data should be ordered by the query”,而 group 按该顺序给出每个查询的行数,而不是查询 ID,因此各组大小之和等于行数(sum(group) = n_samples;Parameters;LGBMRanker)。
  • 增益默认为指数形式。label_gain 默认为 0,1,3,7,15,31,63,...,因此“the gain of label 2 is 3”,并且每个标签都必须小于增益的个数(Parameters)。这就是 Burges 的 DCG,增益为 2^grade − 1。本书的 nDCG 使用线性增益,所以应设置 label_gain=[0, 1, 2]。
  • **nDCG 默认在第 1 到第 5 位上评估。**将 eval_at=[10] 设置为按 nDCG@10 提前停止,并且无论如何都用 lib.metrics 报告最终得分。
  • **验证数据放在 eval_X 和 eval_y 中。**在 4.7.0 中,eval_set 为“Deprecated since version 4.7.0”,由 eval_X 和 eval_y 取代(LGBMRanker;发布说明)。这些示例传入 eval_X、eval_y 和 eval_group。

提前停止在每棵树之后对留出的训练查询打分,当再增加 30 棵树仍无提升时,保留到最佳得分为止的那些树。为使训练能够精确重复,这些示例使用单线程、固定随机种子和 deterministic=True,它“should ensure the stable results when using the same data”(Parameters),同时使用 force_row_wise=True,同一条目推荐这样做,以避免数值问题带来的潜在不稳定。lib.ranking 保存这些设置,其 fit 负责应用分组和提前停止。

行的顺序同样重要。在第一棵树之前,每个得分都是 0,因此 LightGBM 把行的顺序当作计算第一个 λ 时的初始排序,而相同的得分需要一种平局裁决方式。交叉编码器的记录把每个查询的候选按从好到差列出,这会在模型学到任何东西之前就把交叉编码器的排序交给它。因此 feature_table 按 product_id 对候选排序,lib.ranking.ranked 按 product_id 裁决平局。

用训练查询上的交叉验证来选择树的大小:把它们分成五折,对每一折用在其余三折上拟合、并在第五折上提前停止的模型进行排序,使每个查询恰好被打分一次。尝试八种设置,叶子数从 3 到 31,配以两种学习率,并像该指标那样把未经评判的候选计为等级 0:

lambdamart-tree-size-on-train.py Python 3.12.15 · bm25s 0.3.16 · lightgbm 4.7.0 · numpy 2.5.5 · pandas 3.1.1 · pystemmer 3.1.0 · scikit-learn 1.9.1 · scipy 1.18.1 通过
"""Cross-validates LambdaMART on the train queries over eight tree sizes and learning rates, and names the best. The prose'spoint rests on the best setting's score, and on the score beside it for LightGBM's default of 31 leaves at 0.1."""from lightgbm import LGBMRankerfrom lib.metrics import evaluatefrom lib.ranking import DETERMINISTIC, FEATURES, LAMBDARANK, TREE_GRID, TREES, cross_validate, feature_tablefrom lib.semantic import SemanticSearchfrom lib.store import load_products, load_queries, qrelsqueries, judgments = load_queries("train"), qrels("train")          # tuning: train queries onlycandidates = feature_table(load_products(), queries)candidates["grade or 0"] = candidates["grade"].fillna(0).astype(int)  # unjudged counts as irrelevant, as in nDCGprint(f"train queries: {len(queries)}")print(f"semantic, train ndcg@10: {evaluate(SemanticSearch().run(queries, k=100), judgments)['ndcg@10'].mean():.3f}")results = {}for setting in TREE_GRID:                                            # LightGBM's default is 31 leaves at 0.1    name = f"{setting['num_leaves']} leaves, learning rate {setting['learning_rate']}"    # up to 500 trees; early stopping inside cross_validate keeps fewer    model = lambda: LGBMRanker(**LAMBDARANK, n_estimators=500, **setting, **DETERMINISTIC)    # one nDCG@10 per train query, each ranked by a model that never saw that query    results[name] = cross_validate(model, candidates, candidates, "grade or 0", FEATURES, judgments)    print(f"{name}, cross-validated ndcg@10: {results[name].mean():.3f}")    # the trees early stopping kept in each fold    print(f"{name}, trees kept per fold: {', '.join(map(str, results[name].attrs['trees']))}")best = max(results, key=lambda name: results[name].mean())print(f"settings tried: {len(results)}")print(f"best setting: {best}")print(f"lib.ranking.TREES holds it: {'yes' if best == f'{TREES['num_leaves']} leaves, learning rate {TREES['learning_rate']}' else 'no'}")
输出python lambdamart-tree-size-on-train.py
train queries: 50
semantic, train ndcg@10: 0.772
3 leaves, learning rate 0.05, cross-validated ndcg@10: 0.780
3 leaves, learning rate 0.05, trees kept per fold: 18, 27, 52, 20, 85
3 leaves, learning rate 0.1, cross-validated ndcg@10: 0.798
3 leaves, learning rate 0.1, trees kept per fold: 99, 20, 8, 43, 38
7 leaves, learning rate 0.05, cross-validated ndcg@10: 0.744
7 leaves, learning rate 0.05, trees kept per fold: 2, 28, 30, 39, 55
7 leaves, learning rate 0.1, cross-validated ndcg@10: 0.772
7 leaves, learning rate 0.1, trees kept per fold: 80, 12, 13, 7, 20
15 leaves, learning rate 0.05, cross-validated ndcg@10: 0.720
15 leaves, learning rate 0.05, trees kept per fold: 4, 37, 175, 44, 120
15 leaves, learning rate 0.1, cross-validated ndcg@10: 0.728
15 leaves, learning rate 0.1, trees kept per fold: 19, 68, 136, 48, 53
31 leaves, learning rate 0.05, cross-validated ndcg@10: 0.679
31 leaves, learning rate 0.05, trees kept per fold: 15, 8, 91, 8, 22
31 leaves, learning rate 0.1, cross-validated ndcg@10: 0.737
31 leaves, learning rate 0.1, trees kept per fold: 11, 75, 166, 96, 58
settings tried: 8
best setting: 3 leaves, learning rate 0.1
lib.ranking.TREES holds it: yes

在 8 种设置中,最佳设置是 3 leaves, learning rate 0.1,得分为 0.798;LightGBM 默认的 31 个叶子、学习率 0.1(LGBMRanker)得分为 0.737。所有设置中的最佳者会高估自己,因此在留出的查询上应预期得分更低。

有一项选择取决于商店的评判。训练候选中有 0.286 的比例无人评判。模型可以把它们计为等级 0,也可以只从已评判的候选中学习;未评判并不等于不相关,但无论哪种做法,模型在查询时都要对它们排序。在选定的树大小下,用交叉验证比较两者,并在旁边放上对等级做逐点回归的结果:

lambdamart-choices-on-train.py Python 3.12.15 · bm25s 0.3.16 · lightgbm 4.7.0 · numpy 2.5.5 · pandas 3.1.1 · pystemmer 3.1.0 · scikit-learn 1.9.1 · scipy 1.18.1 通过
"""Compares three ways of training on the train queries, each by cross-validation: LambdaRank with unjudged candidates as 0,LambdaRank on judged candidates only, and pointwise regression. The prose's point rests on the differences and thepermutation p-values printed under each setup."""from lightgbm import LGBMRanker, LGBMRegressorfrom lib.metrics import evaluatefrom lib.ranking import DETERMINISTIC, FEATURES, LAMBDARANK, TREES, cross_validate, feature_tablefrom lib.semantic import SemanticSearchfrom lib.significance import comparefrom lib.store import load_products, load_queries, qrelsqueries, judgments = load_queries("train"), qrels("train")          # tuning: train queries onlycandidates = feature_table(load_products(), queries)candidates["grade or 0"] = candidates["grade"].fillna(0).astype(int)  # unjudged counts as irrelevant# the rows the model learns from when unjudged candidates are left out; it still ranks every candidate at query timejudged_only = candidates.dropna(subset=["grade"]).astype({"grade": int})setups = {                                                          # (model, rows it learns from, label)    "lambdarank, unjudged as 0": (lambda: LGBMRanker(**LAMBDARANK, **TREES, **DETERMINISTIC), candidates, "grade or 0"),    "lambdarank, unjudged left out": (lambda: LGBMRanker(**LAMBDARANK, **TREES, **DETERMINISTIC), judged_only, "grade"),    "pointwise regression, unjudged as 0": (lambda: LGBMRegressor(**TREES, **DETERMINISTIC), candidates, "grade or 0"),}semantic = evaluate(SemanticSearch().run(queries, k=100), judgments)["ndcg@10"]print(f"semantic, train ndcg@10: {semantic.mean():.3f}")scores = {name: cross_validate(model, rows, candidates, label, FEATURES, judgments)          for name, (model, rows, label) in setups.items()}chosen = scores["lambdarank, unjudged as 0"]for name, per_query in scores.items():    print(f"{name}, cross-validated ndcg@10: {per_query.mean():.3f}")    if name != "lambdarank, unjudged as 0":        # the mean difference is the first argument minus the second, tested by permutation over the queries        result = compare(per_query, chosen)        print(f"{name}, minus the first: {result['mean difference']:.3f}")        print(f"{name}, minus the first, permutation p: {result['permutation p']:.3f}")result = compare(chosen, semantic)print(f"first setup minus semantic: {result['mean difference']:.3f}")print(f"first setup minus semantic, permutation p: {result['permutation p']:.3f}")
输出python lambdamart-choices-on-train.py
semantic, train ndcg@10: 0.772
lambdarank, unjudged as 0, cross-validated ndcg@10: 0.798
lambdarank, unjudged left out, cross-validated ndcg@10: 0.733
lambdarank, unjudged left out, minus the first: -0.064
lambdarank, unjudged left out, minus the first, permutation p: 0.005
pointwise regression, unjudged as 0, cross-validated ndcg@10: 0.769
pointwise regression, unjudged as 0, minus the first: -0.029
pointwise regression, unjudged as 0, minus the first, permutation p: 0.007
first setup minus semantic: 0.026
first setup minus semantic, permutation p: 0.242

把未评判的候选计为 0 的得分高于将其排除,分别为 0.798 和 0.733,差值为 -0.064,置换检验 p 值为 0.005。一个在训练中从未见过未评判候选的模型,仍然要对它们排序,而且这些特征表明它们看起来像不相关的商品。

逐点回归的得分为 0.769,差值为 -0.029(p = 0.007),因此在商店的训练查询上,与逐点模型对照的调参检查更支持 LambdaMART。

与语义检索的 0.772 相比,所选方案的差值为 0.026,置换检验 p 值为 0.242:即使在选择了它的那些查询上,它也没有显示出训练查询能够与偶然区分开来的提升。

**工作准则:**按查询对行排序并传入 group,把 label_gain 设置为指标所用的增益,通过 eval_X 和 eval_y 在留出的训练查询上提前停止,并固定随机种子和线程数。按第 3 章的建议,把每个查询的候选按商品 ID 排序,并以同样的方式裁决得分平局。除非训练查询上的交叉验证另有结论,否则把未评判的候选计为等级 0。已评判的查询较少时,尝试小树。

这部分尚未翻译,因此以英文显示。

Report LambdaMART on the test queries

最近一次修改见于版次 2026.11.10

A model tuned on train queries is reported once on the test queries, by a comparison planned before looking. Feature importance shows what a model learned: LightGBM's "gain" importance is the "total gains of splits which use the feature" (Booster), so a feature with no gain plays no part in the score.

The first planned comparison uses the setup chosen on train: LambdaMART on all the features, with 3 leaves at a learning rate of 0.1, unjudged candidates as grade 0, and candidates in product_id order. It fits on 40 train queries, stops early on 10 others, and ranks each test query's candidates. A gain needs a permutation p-value below 0.01:

lambdamart-on-test.py Python 3.12.15 · bm25s 0.3.16 · lightgbm 4.7.0 · numpy 2.5.5 · pandas 3.1.1 · pystemmer 3.1.0 · scikit-learn 1.9.1 · scipy 1.18.1 通过
"""Fits LambdaMART on the train queries with the chosen setup, reports its early stopping and gain shares, then reports itsplanned comparison on the test queries against semantic retrieval. The prose's point rests on that comparison's permutationp-value, printed last, against the threshold printed first."""from lightgbm import LGBMRankerfrom lib.metrics import evaluatefrom lib.ranking import (DETERMINISTIC, FEATURES, LAMBDARANK, PLANNED_TEST_COMPARISONS, THRESHOLD, TREES, feature_table,                         fit, ranked, report_planned, stopping_split)from lib.semantic import SemanticSearchfrom lib.store import load_products, load_queries, qrelsproducts, train, test = load_products(), load_queries("train"), load_queries("test")print(f"planned test comparisons: {len(PLANNED_TEST_COMPARISONS)}")print(f"p-value threshold: {THRESHOLD:.2f}")table = feature_table(products, train)table["grade or 0"] = table["grade"].fillna(0).astype(int)            # the setup chosen on train# 10 train queries are held out, at random with a fixed seed, to stop onfit_ids, stop_ids = stopping_split(train.index)model = fit(LGBMRanker(**LAMBDARANK, **TREES, **DETERMINISTIC), table, "grade or 0", FEATURES, fit_ids, stop_ids)print(f"queries fitted: {len(fit_ids)}\nqueries held out for stopping: {len(stop_ids)}")curve = model.evals_result_["valid_0"]["ndcg@10"]                     # LightGBM's nDCG@10, one value per treeprint(f"trees tried: {len(curve)}\ntrees kept by early stopping: {model.best_iteration_}")print(f"stopping queries' ndcg@10, first tree: {curve[0]:.3f}")# best_iteration_ counts trees from 1 and the curve from 0, so the last tree kept is at index best_iteration_ - 1print(f"stopping queries' ndcg@10, trees kept: {curve[model.best_iteration_ - 1]:.3f}")gain = model.booster_.feature_importance(importance_type="gain")      # total gain of the splits on each feature# a feature with no gain was never split on, so it plays no part in the scoreprint(f"features the trees use: {(gain > 0).sum()} of {len(FEATURES)}")for name, share in sorted(zip(FEATURES, gain / gain.sum()), key=lambda pair: -pair[1]):    print(f"share of gain, {name}: {share:.3f}")candidates, judgments = feature_table(products, test), qrels("test")   # test queries: read once, to reportafter = evaluate(ranked(candidates, model.predict(candidates[FEATURES])), judgments)before = evaluate(SemanticSearch().run(test, k=100), judgments)for name, frame in [("semantic", before), ("lambdamart", after)]:    print(f"{name}, test ndcg@10: {frame['ndcg@10'].mean():.3f}")    print(f"{name}, test judged@10: {frame['judged@10'].mean():.3f}")    # each query's top 10 is 10 places, so (1 - judged@10) times 10 counts its unjudged places    print(f"{name}, unjudged places in the test top 10s: {round((1 - frame['judged@10']).sum() * 10)}")report_planned("lambdamart", after["ndcg@10"], before["ndcg@10"])
输出python lambdamart-on-test.py
planned test comparisons: 5
p-value threshold: 0.01
queries fitted: 40
queries held out for stopping: 10
trees tried: 243
trees kept by early stopping: 213
stopping queries' ndcg@10, first tree: 0.568
stopping queries' ndcg@10, trees kept: 0.794
features the trees use: 8 of 8
share of gain, cross_encoder: 0.641
share of gain, similarity: 0.172
share of gain, name_match: 0.097
share of gain, bm25: 0.043
share of gain, review_count: 0.021
share of gain, rating_count: 0.021
share of gain, attribute_match: 0.002
share of gain, average_rating: 0.001
semantic, test ndcg@10: 0.830
semantic, test judged@10: 0.990
semantic, unjudged places in the test top 10s: 3
lambdamart, test ndcg@10: 0.817
lambdamart, test judged@10: 0.990
lambdamart, unjudged places in the test top 10s: 3
lambdamart, mean difference: -0.012
lambdamart, wins, losses, ties: 14, 12, 4
lambdamart, permutation p: 0.260
lambdamart, established at 0.01: no
lambdamart, t-test p: 0.278
lambdamart, t-test agrees: yes
lambdamart, 99% interval: -0.044 to 0.011
lambdamart, interval agrees: yes

Of 243 trees tried, early stopping keeps 213. On the queries held out for stopping, LightGBM's own nDCG@10, whose ideal comes from their candidates alone, rises from 0.568 after the first tree to 0.794 at the trees kept.

The trees use 8 of 8 features, but they lean on the cross-encoder, which takes a share of 0.641 of the gain, ahead of similarity at 0.172, name_match at 0.097 and bm25 at 0.043. So serving the model means a cross-encoder call for every candidate.

LambdaMART doesn't establish a gain on the test queries: it scores 0.817 nDCG@10, against semantic retrieval's 0.830, at a permutation p-value of 0.260 (all five comparisons). Judged@10 is 0.990 for LambdaMART and 0.990 for semantic retrieval.

Working rule: Adopt a learned ranker only when its planned test comparison is established. Read its gain importance before you trust it, because the features it leans on are the ones you'll pay for at query time. Early stopping on a few held-out queries keeps very different numbers of trees from one split to the next, from 99, 20, 8, 43, 38 across the cross-validation folds to 213 here. A rule of thumb, not a measurement: with this few queries, judge more queries before you tune further, and treat any one stopping point as noise.

用交叉编码器重排语义检索结果

最近一次修改见于版次 2026.11.10

语义检索使用嵌入模型,也称为双编码器。它分别读取查询和每个商品,因此商品的嵌入可以提前计算。交叉编码器则把查询和一个商品放在一起读取,输出一个分数(Nogueira and Cho, 2019)。它可以把查询中的每个词与商品中的每个词联系起来,但这要付出代价。它通常比 Sentence Transformer 模型慢,因为它针对每个查询与商品的配对计算,而不是针对每段文本计算,所以“交叉编码器常用于对前 k 个结果重新排序”(Sentence Transformers)。本店的交叉编码器(见开篇)按语言模型的标准来看很小,有 149M 个参数(模型卡)。它是开放模型 Alibaba-NLP/gte-reranker-modernbert-base,其清单(随记录一起保存的文件)给出的许可证是 Apache-2.0。我们选择它,是因为任何人都可以下载它并重新运行这份记录。recordings/cross_encoder.py 为每个查询的 BM25 与语义检索前 100 名的并集打了分(有两个查询,其语义检索列表与 lib.semantic 的列表在第 100 位上有一处近乎并列的差异)。

第 1 章重排的是混合检索的结果。这里由交叉编码器重排需要超越的排序器,即语义检索的前 20 名:这些商品与 LLM 重排器读取的相同,因此本章的三个重排器从相同的列表出发。本章在看到任何测试结果之前就固定了这一深度,并且不对它调优。按记录的分数对前 20 名重新排序,其余部分保持原位。这是第二项计划内比较,要求 p 值低于 0.01:

cross-encoder-on-semantic.py Python 3.12.15 · bm25s 0.3.16 · lightgbm 4.7.0 · numpy 2.5.5 · pandas 3.1.1 · pystemmer 3.1.0 · scikit-learn 1.9.1 · scipy 1.18.1 通过
"""Reranks semantic retrieval's top 20 with the recorded cross-encoder scores, and compares the two on the test queries.The prose's point rests on the planned comparison's permutation p-value, and on the judged@10 lines, which show what thererank does to the judgments it can score."""from lib.metrics import evaluatefrom lib.ranking import DEPTH, rerank, report_plannedfrom lib.recorded import cross_encoder_scores, llm_listwise, manifestfrom lib.semantic import SemanticSearchfrom lib.store import load_queries, qrelsqueries, judgments = load_queries("test"), qrels("test")       # scored on the human judgmentssemantic = SemanticSearch().run(queries, k=100)recorded = cross_encoder_scores()# the recorded score of each (query, product) pairscore = dict(zip(zip(recorded["query_id"], recorded["product_id"]), recorded["score"]))print("cross-encoder model:", manifest("cross-encoder")["model"])print("licence in the manifest:", manifest("cross-encoder")["model_licence"])# the set of products the LLM was shown for each query, from the listwise recordingshown = llm_listwise().groupby("query_id")["product_id"].apply(set)  # what the LLM rerankers sawsame = sum(set(semantic[query_id][:DEPTH]) == shown[query_id] for query_id in queries.index)print(f"rerank depth: {DEPTH}")print(f"queries whose top {DEPTH} holds the products the llm rerankers saw: {same} of {len(queries)}")reranked = rerank(semantic, score, DEPTH)                       # top 20 reordered, the rest in placebefore, after = evaluate(semantic, judgments), evaluate(reranked, judgments)for name, frame in [("semantic", before), ("cross-encoder", after)]:    print(f"{name}, test ndcg@10: {frame['ndcg@10'].mean():.3f}")    print(f"{name}, test judged@10: {frame['judged@10'].mean():.3f}")    print(f"{name}, unjudged places in the test top 10s: {round((1 - frame['judged@10']).sum() * 10)}")# index 0 is the first placemoved = sum(reranked[query_id][0] != semantic[query_id][0] for query_id in queries.index)print(f"queries whose first product changes: {moved}")report_planned("cross-encoder", after["ndcg@10"], before["ndcg@10"])
输出python cross-encoder-on-semantic.py
cross-encoder model: Alibaba-NLP/gte-reranker-modernbert-base
licence in the manifest: Apache-2.0
rerank depth: 20
queries whose top 20 holds the products the llm rerankers saw: 30 of 30
semantic, test ndcg@10: 0.830
semantic, test judged@10: 0.990
semantic, unjudged places in the test top 10s: 3
cross-encoder, test ndcg@10: 0.813
cross-encoder, test judged@10: 0.983
cross-encoder, unjudged places in the test top 10s: 5
queries whose first product changes: 23
cross-encoder, mean difference: -0.016
cross-encoder, wins, losses, ties: 11, 12, 7
cross-encoder, permutation p: 0.178
cross-encoder, established at 0.01: no
cross-encoder, t-test p: 0.171
cross-encoder, t-test agrees: yes
cross-encoder, 99% interval: -0.058 to 0.006
cross-encoder, interval agrees: yes

重排前 20 名并不能确立提升:重排后的分数为 0.813,语义检索为 0.830,置换检验的 p 值为 0.178(见决策表)。对于 30 of 30 个测试查询,前 20 名包含 LLM 重排器所见的商品,而重排器改变了其中 23 个查询的第一个商品。

Judged@10从 0.990 降到 0.983,测试集前 10 名中未经评判的位置从 3 增加到 5。

在训练查询上,同样的重排得分为 0.802,语义检索为 0.772(深度扫描)。训练查询是另一份样本,其结论重复了第 3 章的发现:两个接近的排序器的先后顺序,可能在不同的查询集之间发生翻转。

工作准则: 对你最强的检索结果做重排,重排深度要在查看测试结果之前固定,并且只有在留出查询上确立了提升时,才采用该重排器。本店的通用交叉编码器不进入流水线。

用 LLM 为每个商品打分

最近一次修改见于版次 2026.11.10

LLM 也可以做重排序,方式有三种。逐点式(pointwise):LLM 单独判断每个商品,返回一个标签或分数。成对式(pairwise):LLM 指出两个商品中哪一个更符合查询。列表式(listwise):LLM 在一次回答中对整个列表排序(Qin et al., 2024)。在一项研究中,研究者使用 BEIR(一组用于检验模型在未训练过的数据上表现的检索基准)中的八个数据集,发现逐点式提示在采用分级回答时比采用是或否回答时效果更好,原因是“缺少中间相关性标签选项,可能导致 LLM 给出含噪声或有偏的回答”(Zhuang et al., 2024a)。

商店使用的 LLM 是本章开头提到的那个开源模型。它每次只看到查询和一个商品:名称、类(class)、类别、描述和属性(见LLM 看到的内容)。提示要求给出一个标签,即 exact、partial 或 irrelevant,并给出“一个 0 到 100 的分数,表示其匹配程度”。按该分数对语义检索的前 20 个结果重新排序;两个商品分数相同时,保持语义检索的顺序,这一规则在录制时已经固定。该重排序是第三项计划内比较,要求 p 值低于 0.01才算有提升:

llm-pointwise-rerank.py Python 3.12.15 · bm25s 0.3.16 · lightgbm 4.7.0 · numpy 2.5.5 · pandas 3.1.1 · pystemmer 3.1.0 · scikit-learn 1.9.1 · scipy 1.18.1 通过
"""Reads the pointwise LLM's answers for the test queries, one call per product, and reranks semantic retrieval's top 20 bytheir scores. The prose's point rests on the distinct-scores line, which shows how coarse the scores are, and on theplanned comparison's p-value."""from lib.metrics import evaluatefrom lib.ranking import DEPTH, rerank, report_plannedfrom lib.recorded import llm_pointwise, manifestfrom lib.semantic import SemanticSearchfrom lib.store import load_queries, qrelsqueries, judgments = load_queries("test"), qrels("test")       # scored on the human judgments, not the LLM'ssemantic = SemanticSearch().run(queries, k=100)answers = llm_pointwise()                                       # one call per product: a label and a scoreprint("llm:", manifest("llm-rerank")["model"])print("licence in the manifest:", manifest("llm-rerank")["model_licence"])print(f"calls per query: {len(answers) // answers['query_id'].nunique()}")print(f"answers without a score: {answers['score'].isna().sum()}")print(f"distinct scores per query, median: {answers.groupby('query_id')['score'].nunique().median():.0f}")print(f"share of products scored 100: {(answers['score'] == 100).mean():.3f}")# each answered product's human grade, or missing when nobody judged itpeople = answers.apply(lambda row: judgments[row["query_id"]].get(row["product_id"]), axis=1)# grades 2, 1 and 0 are exact, partial and irrelevant, the labels the LLM was asked forfor label, grade in [("exact", 2), ("partial", 1), ("irrelevant", 0)]:    print(f"products the llm labels {label}: {(answers['label'] == label).sum()}")    print(f"products people judged {label}: {(people == grade).sum()}")print(f"products nobody judged: {people.isna().sum()}")score = dict(zip(zip(answers["query_id"], answers["product_id"]), answers["score"]))reranked = rerank(semantic, score, DEPTH)                       # equal scores keep semantic retrieval's ordercounts = answers.groupby("query_id")["score"].value_counts()          # products per score, for each queryshared = {query_id: sum(counts[(query_id, score[(query_id, product_id)])] > 1 for product_id in reranked[query_id][:10])          for query_id in queries.index}                                # top-10 products that share their score# more than 5 of the top 10 share a score: most of the list is decided by the tie-breakprint(f"queries where most of the reranked top 10 shares its score: {sum(n > 5 for n in shared.values())}")before, after = evaluate(semantic, judgments), evaluate(reranked, judgments)for name, frame in [("semantic", before), ("llm pointwise", after)]:    print(f"{name}, test ndcg@10: {frame['ndcg@10'].mean():.3f}")    print(f"{name}, test judged@10: {frame['judged@10'].mean():.3f}")    print(f"{name}, unjudged places in the test top 10s: {round((1 - frame['judged@10']).sum() * 10)}")report_planned("llm pointwise", after["ndcg@10"], before["ndcg@10"])
输出python llm-pointwise-rerank.py
llm: gemma4:12b-it-q4_K_M
licence in the manifest: Apache-2.0
calls per query: 20
answers without a score: 0
distinct scores per query, median: 4
share of products scored 100: 0.390
products the llm labels exact: 251
products people judged exact: 310
products the llm labels partial: 345
products people judged partial: 265
products the llm labels irrelevant: 4
products people judged irrelevant: 7
products nobody judged: 18
queries where most of the reranked top 10 shares its score: 30
semantic, test ndcg@10: 0.830
semantic, test judged@10: 0.990
semantic, unjudged places in the test top 10s: 3
llm pointwise, test ndcg@10: 0.837
llm pointwise, test judged@10: 0.983
llm pointwise, unjudged places in the test top 10s: 5
llm pointwise, mean difference: 0.008
llm pointwise, wins, losses, ties: 11, 8, 11
llm pointwise, permutation p: 0.241
llm pointwise, established at 0.01: no
llm pointwise, t-test p: 0.244
llm pointwise, t-test agrees: yes
llm pointwise, 99% interval: -0.007 to 0.026
llm pointwise, interval agrees: yes

录制时每个查询调用了 20 次,有 0 个回答没有给出分数。分数的粒度很粗。对于 20 个商品,中位数查询只得到 4 个不同的分数,所有商品中有 0.390 得分为 100。在 30 个查询中,重排序后前 10 名里的大多数商品与另一个商品分数相同,因此用来打破平局的语义检索顺序在很大程度上决定了列表的排法。

在这个提示和这些列表上,LLM 判为 exact 的商品比人工判断的少:

标签 LLM 人工
exact 251 310
partial 345 265
irrelevant 4 7
无人判断 – 18

在这些列表上,这一结论成立,但第 4 章表明,LLM 与人工判断的分歧方向会随所判断的查询-商品对和提示而变化,因此不要把这一结论推广到其他场合。

逐点重排序同样没有证明带来了提升:它的 test ndcg@10 为 0.837,语义检索为 0.830,置换检验 p 值为 0.241(见决策表)。

LLM 的排序使前 10 名中出现了更多未被判断的商品:为 5 个位置,而语义检索为 3 个,judged@10 从 0.990 降到 0.983。

工作准则: 如果用 LLM 做逐点重排序,就要求它给出分级标签或分数,统计每个查询给出的不同分数的个数,并事先确定打破平局的方式:沿用检索的顺序是稳妥的默认做法。评估结果时使用人工判断,不要使用同一个 LLM 给出的等级。

用一条 LLM 提示词对 20 个商品排序

最近一次修改见于版次 2026.11.8

第三种方法是列表式(listwise)提示词:给候选项编号,并要求 LLM 按相关性顺序返回这些编号。RankGPT 将这种做法作为排列生成(permutation generation)提出(Sun et al., 2023)。一条提示词能容纳的商品数量有限,因此较长的列表要用滑动窗口重排:RankGPT 对 BM25 的前 100 个结果重排,采用“窗口大小为 20、步长为 10”,从列表底部向上进行(第 3.2 节和第 6.1 节)。答案必须是一个有效的排序,而这一点可能做不到:在 2023 年的中等规模 LLM 上,例如 200 亿参数的 Flan-UL2,列表式提示词几乎总是失败,原因是排序有缺失、重复或被拒绝(Qin et al., 2024)。

商店的 LLM 每条提示词看到 20 个商品,即一个窗口。提示词把每个商品显示为一短行:名称、类别、属性和描述的前 160 个字符。最后一条指令写道:“Order all 20 products from the best match for the query to the worst. Return the product numbers, each exactly once, in JSON.”JSON 模式要求恰好 20 个数字,但不要求这 20 个数字互不相同,因此记录环节会修复任何含有重复或缺失编号的答案,并加以标记。按 LLM 在看到按语义顺序排列的商品时给出的答案,对语义检索的前 20 个结果重新排序。这次重排是第四项计划内比较,p 值需低于 0.01:

llm-listwise-rerank.py Python 3.12.15 · bm25s 0.3.16 · lightgbm 4.7.0 · numpy 2.5.5 · pandas 3.1.1 · pystemmer 3.1.0 · scikit-learn 1.9.1 · scipy 1.18.1 通过
"""Reads the listwise LLM's answers for the test queries, in both orders, and reranks semantic retrieval's top 20 by theanswer given in semantic order. The prose's point rests on the planned comparison's p-value, and on the repair count,which shows how many answers needed a fix."""from lib.metrics import evaluatefrom lib.ranking import DEPTH, report_plannedfrom lib.recorded import llm_listwise, manifestfrom lib.semantic import SemanticSearchfrom lib.store import load_queries, qrelsqueries, judgments = load_queries("test"), qrels("test")       # scored on the human judgments, not the LLM'ssemantic = SemanticSearch().run(queries, k=100)answers = llm_listwise()                                        # one call per query and order shown# the answers hold one row per product, so keep one row per call: one call per query and ordercalls = answers.drop_duplicates(["query_id", "order"])print(f"listwise calls, both orders: {len(calls)}")# valid is false where the recording repaired a repeated or missing numberprint(f"answers that needed repair: {(~calls['valid']).sum()}")print(f"products per call: {answers.groupby(['query_id', 'order']).size().max()}")print("prompt asks for:", manifest("llm-rerank")["prompts"]["listwise"].splitlines()[-1].replace("{n}", str(DEPTH)))shown_in_semantic_order = answers[answers["order"] == "semantic"].sort_values(["query_id", "position"])order = shown_in_semantic_order.groupby("query_id")["product_id"].apply(list)# the reranked top 20, then the rest of semantic retrieval's list in place: ranking[DEPTH:] starts at the 21st productreranked = {query_id: order[query_id] + ranking[DEPTH:] for query_id, ranking in semantic.items()}before, after = evaluate(semantic, judgments), evaluate(reranked, judgments)for name, frame in [("semantic", before), ("llm listwise", after)]:    print(f"{name}, test ndcg@10: {frame['ndcg@10'].mean():.3f}")    print(f"{name}, test judged@10: {frame['judged@10'].mean():.3f}")    print(f"{name}, unjudged places in the test top 10s: {round((1 - frame['judged@10']).sum() * 10)}")moved = sum(reranked[query_id][0] != semantic[query_id][0] for query_id in queries.index)print(f"queries whose first product changes: {moved}")report_planned("llm listwise", after["ndcg@10"], before["ndcg@10"])
输出python llm-listwise-rerank.py
listwise calls, both orders: 60
answers that needed repair: 0
products per call: 20
prompt asks for: Order all 20 products from the best match for the query to the worst. Return the product numbers, each exactly once, in JSON.
semantic, test ndcg@10: 0.830
semantic, test judged@10: 0.990
semantic, unjudged places in the test top 10s: 3
llm listwise, test ndcg@10: 0.819
llm listwise, test judged@10: 0.983
llm listwise, unjudged places in the test top 10s: 5
queries whose first product changes: 19
llm listwise, mean difference: -0.010
llm listwise, wins, losses, ties: 12, 12, 6
llm listwise, permutation p: 0.580
llm listwise, established at 0.01: no
llm listwise, t-test p: 0.498
llm listwise, t-test agrees: yes
llm listwise, 99% interval: -0.070 to 0.016
llm listwise, interval agrees: yes

记录共进行了 60 次列表式调用,每个测试查询两次,其中 0 个答案需要修复。重排使 19 个查询的第一个商品发生了变化。

列表式重排的收益并未得到确认:它的得分为 0.819,对照为 0.830,置换检验的 p 值为 0.580(见全部五项比较)。

Judged@10 在这里同样下降,从 0.990 降到 0.983,未评判位置有 5 个,而语义检索为 3 个。

工作准则: 使用每个列表式答案之前都要检查:统计缺失、重复和未知的编号,并记录有多少答案需要修复。如果运行环境支持,就用模式约束答案的形状。测量之前先固定窗口、步长和展示商品的顺序,因为顺序会改变答案,下一项检查会说明这一点。

这部分尚未翻译,因此以英文显示。

Reverse the input to measure positional bias

最近一次修改见于版次 2026.11.10

A listwise answer can suffer from positional bias: an LLM's tendency to let a product's place in the prompt affect its rank, not the click position bias of shoppers. "Large language models (LLMs) exhibit positional bias in how they use context, which especially complicates listwise ranking" (Tang et al., 2024). On TREC DL19, the 2019 queries of TREC's Deep Learning track, a passage-ranking benchmark, gpt-3.5-turbo reranked BM25's top 100 to an nDCG@10 of 65.80 when shown BM25's order, 32.77 when shown it reversed and 25.17 when shown a random order (Sun et al., 2023, Table 5). The direct check is to show the same products in two orders and compare the answers. Kendall's tau measures how alike two orderings are, from 1 for the same order to −1 for the reverse, by counting the pairs they put the same way round (chapter 4).

The recording sent each test query's listwise prompt to the same LLM twice. The first prompt showed semantic retrieval's top 20 in semantic order, as the listwise rerank used; the second showed them reversed. Out of the 30 test queries, how often will the LLM put the same product first both times? Write your number down.

listwise-position-bias.py Python 3.12.15 · bm25s 0.3.16 · lightgbm 4.7.0 · numpy 2.5.5 · pandas 3.1.1 · pystemmer 3.1.0 · scikit-learn 1.9.1 · scipy 1.18.1 通过
"""Shows the listwise LLM the same test products in two orders, and measures how far its answers agree and how far eachanswer follows the order it was shown. The prose's point rests on the mean tau between the two answers, and on the twotaus between each answer and its own input order."""import numpy as npfrom scipy.stats import kendalltaufrom lib.metrics import evaluatefrom lib.ranking import DEPTHfrom lib.recorded import llm_listwisefrom lib.semantic import SemanticSearchfrom lib.store import load_queries, qrelsqueries, judgments = load_queries("test"), qrels("test")semantic = SemanticSearch().run(queries, k=DEPTH)               # the order the first prompt showedanswers = llm_listwise().sort_values(["query_id", "order", "position"])# the products in the order the LLM returned them, for each query and each order shownanswer = {key: list(group["product_id"]) for key, group in answers.groupby(["query_id", "order"])}def tau(first: list, second: list) -> float:    """Kendall's tau between two orderings of the same products."""    place = {product_id: rank for rank, product_id in enumerate(second)}    # each product's place in the second ordering    # the first ordering's positions, against where the second ordering puts each product    return kendalltau(range(len(first)), [place[product_id] for product_id in first]).statisticbetween, shown_semantic, shown_reversed, overlap, same_first = [], [], [], [], 0for query_id, shown in semantic.items():    forward, backward = answer[(query_id, "semantic")], answer[(query_id, "reversed")]    between.append(tau(forward, backward))    shown_semantic.append(tau(forward, shown))    shown_reversed.append(tau(backward, shown[::-1]))     # the reversed input, as the second prompt showed it    overlap.append(len(set(forward[:10]) & set(backward[:10])))    same_first += forward[0] == backward[0]               # index 0 is the first placeprint(f"test queries: {len(semantic)}")print(f"tau between the two answers, mean: {np.mean(between):.3f}")print(f"tau between the two answers, lowest: {min(between):.3f}\ntau between the two answers, highest: {max(between):.3f}")print(f"top-10 products in common, mean: {np.mean(overlap):.3f}")print(f"top-10 products in common, fewest: {min(overlap)}")print(f"queries with the same first product: {same_first}")print(f"share of queries with the same first product: {same_first / len(semantic):.3f}")print(f"tau between the order shown and the answer, semantic order: {np.mean(shown_semantic):.3f}")print(f"tau between the order shown and the answer, reversed order: {np.mean(shown_reversed):.3f}")# an answer that ignored the order would repeat itself, so its tau with the reversed input is minus its tau with the semantic orderprint(f"the same, if the order shown made no difference: {-np.mean(shown_semantic):.3f}")reversed_run = {query_id: answer[(query_id, "reversed")] for query_id in semantic}print(f"reversed answer, test ndcg@10, not a planned comparison: {evaluate(reversed_run, judgments)['ndcg@10'].mean():.3f}")
输出python listwise-position-bias.py
test queries: 30
tau between the two answers, mean: 0.311
tau between the two answers, lowest: -0.484
tau between the two answers, highest: 0.811
top-10 products in common, mean: 6.967
top-10 products in common, fewest: 3
queries with the same first product: 13
share of queries with the same first product: 0.433
tau between the order shown and the answer, semantic order: 0.414
tau between the order shown and the answer, reversed order: 0.045
the same, if the order shown made no difference: -0.414
reversed answer, test ndcg@10, not a planned comparison: 0.826

The same product comes first for 13 of the 30 queries, so the expectation that a capable model finds the same best match whichever way the list comes holds for only a share of 0.433 of the queries.

The two answers agree with a mean tau of 0.311, and per query the tau runs from -0.484 to 0.811. Their top 10s share 6.967 products on average, as few as 3.

The answers also lean toward the order shown. Each answer's tau against its own input order is 0.414 for the semantic order and 0.045 for the reversed one. An LLM that ignored the order would give the same answer both times, so its reversed answer would have a tau of -0.414 against the reversed input. The reversed answer sits far nearer its own input order than that: positional bias, for this model, prompt and these queries. The reversed answer scores 0.826 nDCG@10; it is not a planned comparison, so the chapter doesn't test it.

Three remedies exist. Pairwise ranking prompting asks the LLM about each pair twice, once in each order, and counts a disagreement as a tie (Qin et al., 2024). Permutation self-consistency reranks several shuffled orders and aggregates the answers, which raised scores by up to 7 to 18% for GPT-3.5 and 8 to 16% for LLaMA v2 (70B) (Tang et al., 2024). And rerankers fine-tuned for listwise ranking, such as the open RankZephyr, are reported to be "resilient against variations in initial document ordering" (Pradeep et al., 2023); the store didn't record one.

Working rule: Before you trust a listwise LLM reranker, run each query in at least two orders and report how much the answers agree. When agreement is low, aggregate several orders or ask each pair in both orders, and count those calls against your budget. In production, show the products in retrieval's order every time, so that the same list at least gets the same answer.

根据预算设定重排深度

最近一次修改见于版次 2026.11.8

搜索流水线分阶段运行,每个阶段都比上一阶段更慢,获得的候选项也更少,而本章中的每个重排器都会增加一个阶段。多阶段排序器提供了“通过控制候选项进入各流水线阶段的准入,在质量与延迟之间进行权衡的能力”(Nogueira et al., 2019)。因此,重排深度,即重排器对每个查询读取的商品数量,是一项预算决策。耗时会随硬件和运行时的选择而变化(Peng et al., 2025),所以本节转而统计决定耗时的因素:每个查询的模型调用次数和所读取的文本量,再乘以模型的规模。商店的交叉编码器有 149M 个参数(模型卡),其 LLM 约有 120 亿个参数(模型卡),约为前者的 80 倍。

在深度为 20 时,根据商品文本和已记录的提示词,统计每个重排器对每个测试查询的模型调用次数和读取的字符数,另外再统计商店未记录的两种设计所需的调用次数:

reranker-prompt-sizes.py Python 3.12.15 · bm25s 0.3.16 · lightgbm 4.7.0 · numpy 2.5.5 · pandas 3.1.1 · pystemmer 3.1.0 · scikit-learn 1.9.1 · scipy 1.18.1 通过
"""Counts the characters each reranker reads per test query at depth 20, from the products' text and the recorded prompts,and the model calls each design makes. The prose's point rests on the characters and calls lines, and on the two designsthe store didn't record."""import numpy as npfrom lib.agreement import as_the_llm_sawfrom lib.ranking import DEPTH, listwise_linefrom lib.recorded import manifestfrom lib.semantic import SemanticSearchfrom lib.store import load_products, load_queries, product_textproducts, queries = load_products(), load_queries("test")semantic = SemanticSearch().run(queries, k=DEPTH)                # the 20 products each reranker readsprompts, text = manifest("llm-rerank")["prompts"], product_text(products)cross_encoder, pointwise, listwise = [], [], []                  # characters read per queryfor query_id, query in queries["query"].items():    shown = products.loc[semantic[query_id]]    # one call per product, each reading the query and that product's text together    cross_encoder.append(sum(len(query) + len(text[product_id]) for product_id in semantic[query_id]))    # the pointwise prompt, filled in with the query and the product as the LLM saw it    pointwise.append(sum(len(prompts["pointwise"].format(query=query, product=as_the_llm_saw(product)))                         for _, product in shown.iterrows()))    # products numbered from 1, as the listwise prompt shows them    lines = "\n".join(f"[{number}] {listwise_line(product)}" for number, (_, product) in enumerate(shown.iterrows(), 1))    listwise.append(len(prompts["listwise"].format(query=query, products=lines, n=DEPTH)))# the cross-encoder and the pointwise LLM make one call per product; the listwise LLM makes one call per queryfor name, calls, characters in [("cross-encoder", DEPTH, cross_encoder), ("llm pointwise", DEPTH, pointwise),                                ("llm listwise", 1, listwise)]:    print(f"{name}, model calls per query: {calls}")    print(f"{name}, characters read per query, mean: {np.mean(characters):,.0f}")print(f"llm pairwise, every pair in both orders, calls per query: {DEPTH * (DEPTH - 1)}")# a sliding window of 20 products, moved 10 at a time: its starts are list positions 0, 10, ..., 80window, step, depth = 20, 10, 100                                # a sliding window over a deeper listprint(f"llm listwise over the top {depth}, window {window}, step {step}, calls per query: {(depth - window) // step + 1}")
输出python reranker-prompt-sizes.py
cross-encoder, model calls per query: 20
cross-encoder, characters read per query, mean: 9,911
llm pointwise, model calls per query: 20
llm pointwise, characters read per query, mean: 17,864
llm listwise, model calls per query: 1
llm listwise, characters read per query, mean: 5,973
llm pairwise, every pair in both orders, calls per query: 380
llm listwise over the top 100, window 20, step 10, calls per query: 9
重排器,前 20 每个查询的模型调用次数 每个查询读取的字符数
交叉编码器 20 9,911
LLM,逐点 20 17,864
LLM,列表式 1 5,973

逐点方式会重复指令,并完整展示每个商品,因此读取的文本最多,且调用相互独立。列表式读取的文本最少,只需一次调用,但必须按顺序写出 20 个数字。若采用成对提示,对所有商品对按两种顺序各比较一次,则每个查询需要 380 次调用;若采用 RankGPT 的滑动窗口处理前 100 个候选项,则需要 9 次。在 TREC DL19 上的一项已发表测量中,使用 Flan-T5-XL 在一块 RTX A6000 GPU 上对 BM25 的前 100 个结果重排,逐点的“是或否”提示得到 0.650 的 nDCG@10,每个查询耗时 1.5 秒;对所有商品对的成对提示得到 0.713,耗时 254.9 秒,共 9,900 次 LLM 调用(Zhuang et al., 2024b)。以交叉编码器作为特征的 LambdaMART 需要为每个候选项计算交叉编码器得分,每个查询的商品对数中位数为 131。

对于交叉编码器,成本随深度同步增长,所以问题在于质量在何处停止增长。在训练查询上扫描不同的深度:

rerank-depth-on-train.py Python 3.12.15 · bm25s 0.3.16 · lightgbm 4.7.0 · numpy 2.5.5 · pandas 3.1.1 · pystemmer 3.1.0 · scikit-learn 1.9.1 · scipy 1.18.1 通过
"""Reranks semantic retrieval's top 5 to 75 with the recorded cross-encoder scores, on the train queries, and prints nDCG@10at each depth. The prose's point rests on the change from the depth before, which shows where the gain flattens and thenfalls."""from lib.metrics import evaluatefrom lib.ranking import rerankfrom lib.recorded import cross_encoder_scoresfrom lib.semantic import SemanticSearchfrom lib.store import load_queries, qrelsqueries, judgments = load_queries("train"), qrels("train")      # choosing a depth is tuning: train queriessemantic = SemanticSearch().run(queries, k=100)recorded = cross_encoder_scores()# the recorded cross-encoder score of each (query, product) pairscore = dict(zip(zip(recorded["query_id"], recorded["product_id"]), recorded["score"]))train_pairs = recorded[recorded["query_id"].isin(queries.index)]print(f"depth 0 (semantic alone), train ndcg@10: {evaluate(semantic, judgments)['ndcg@10'].mean():.3f}")previous = round(evaluate(semantic, judgments)["ndcg@10"].mean(), 3)     # changes between the printed valuesfor depth in [5, 10, 20, 30, 50, 75]:    mean = round(evaluate(rerank(semantic, score, depth), judgments)["ndcg@10"].mean(), 3)    print(f"depth {depth}, train ndcg@10: {mean:.3f}")    print(f"depth {depth}, change from the depth before: {mean - previous:+.3f}")    previous = mean# every candidate needs its own cross-encoder score, so this counts the recorded pairs per queryprint(f"pairs per query to score every candidate, median: {train_pairs.groupby('query_id').size().median():.0f}")
输出python rerank-depth-on-train.py
depth 0 (semantic alone), train ndcg@10: 0.772
depth 5, train ndcg@10: 0.782
depth 5, change from the depth before: +0.010
depth 10, train ndcg@10: 0.784
depth 10, change from the depth before: +0.002
depth 20, train ndcg@10: 0.802
depth 20, change from the depth before: +0.018
depth 30, train ndcg@10: 0.804
depth 30, change from the depth before: +0.002
depth 50, train ndcg@10: 0.793
depth 50, change from the depth before: -0.011
depth 75, train ndcg@10: 0.788
depth 75, change from the depth before: -0.005
pairs per query to score every candidate, median: 131

交叉编码器的增益在 20 到 30 之间趋于平缓,接近为与 LLM 记录相匹配而确定的深度。从语义检索的 0.772 起,重排前 5 个得到 0.782,重排前 20 个得到 0.802。前 30 个再增加 +0.002,所需的商品对数则比之前多出一半。再往深处,结果下降:深度 50 时为 0.793,深度 75 时为 0.788。

在测试集上,深度为 20 的重排并未证实有增益,所以对这家商店而言,成本最低的重排器,也就是不使用重排器,同样是应当保留的方案。

工作准则: 对你考虑的每个重排器,统计每个深度下每个查询的调用次数和字符数,并在训练查询上设定深度。这是经验法则,而非测量结果:重排深度只取到训练集 nDCG@10 不再提升的位置,因为超过该位置的每个商品成本相同,却没有任何收益。然后在你自己的硬件和流量上为该配置计时,只有在它满足延迟预算时才保留这一阶段。

将交叉编码器蒸馏为小型排序器

最近一次修改见于版次 2026.11.10

对预算而言过慢的重排器,可以用来训练一个快速的排序器。蒸馏(distillation)训练一个小模型,即学生模型(student),去模仿一个大模型,即教师模型(teacher)。Hinton 等人将其描述为把一组集成模型的知识蒸馏进单个模型(Hinton et al., 2015)。在排序任务中,教师模型的分数成为学生模型的标签。当两个模型的分数量级不同时(不同架构的分数往往如此),学生模型可以改为学习两个商品分数之间的差值,所用的损失称为 Margin-MSE(Hofstätter et al., 2020)。蒸馏也可以从大语言模型向下进行。RankGPT 报告称,一个经蒸馏的 440M 模型在 BEIR 基准上优于一个 3B 的有监督模型(Sun et al., 2023;BEIR)。在一项针对含最高级表达的查询的商品搜索研究中,轻量模型接管了大语言模型的解读,而直接用大语言模型重排的延迟高到无法接受(Zhu et al., 2025)。教师模型不需要人工标签,并且为每个候选项打标签。

教师模型是交叉编码器。学生模型是一个 LightGBM 回归模型,使用7特征,这些特征除检索本身已经进行的那一次调用外,不需要再调用任何模型:即除 cross_encoder 之外的全部特征。它的标签是教师模型的原始分数,树回归可以在教师模型自身的尺度上拟合这一分数。它对同样的候选项按 product_id 的顺序排序,树的规模取为 LambdaMART 所选的大小。首先,通过对训练查询做交叉验证,将学生模型与一个使用相同特征、在人工评分上训练的 LambdaMART 模型进行比较:

distillation-on-train.py Python 3.12.15 · bm25s 0.3.16 · lightgbm 4.7.0 · numpy 2.5.5 · pandas 3.1.1 · pystemmer 3.1.0 · scikit-learn 1.9.1 · scipy 1.18.1 通过
"""Trains a student LightGBM regression on the cross-encoder's scores, and a LambdaMART model on human grades with the samefeatures, both by cross-validation on the train queries. The prose's point rests on the student's score, and on itsdifference from the human-graded ranker."""from lightgbm import LGBMRanker, LGBMRegressorfrom lib.metrics import evaluatefrom lib.ranking import CHEAP_FEATURES, DETERMINISTIC, LAMBDARANK, TREES, cross_validate, feature_table, rankedfrom lib.semantic import SemanticSearchfrom lib.significance import comparefrom lib.store import load_products, load_queries, qrelsqueries, judgments = load_queries("train"), qrels("train")          # design work: train queries onlycandidates = feature_table(load_products(), queries)# unjudged candidates count as grade 0, as in the metriccandidates["grade or 0"] = candidates["grade"].fillna(0).astype(int)student = cross_validate(lambda: LGBMRegressor(**TREES, **DETERMINISTIC), candidates, candidates,                         "cross_encoder", CHEAP_FEATURES, judgments)          # labels from the teacherhuman = cross_validate(lambda: LGBMRanker(**LAMBDARANK, **TREES, **DETERMINISTIC), candidates, candidates,                       "grade or 0", CHEAP_FEATURES, judgments)               # labels from people# the teacher's own order of the candidates, ties broken by product_idteacher = evaluate(ranked(candidates, candidates["cross_encoder"]), judgments)["ndcg@10"]semantic = evaluate(SemanticSearch().run(queries, k=100), judgments)["ndcg@10"]for name, scores in [("semantic", semantic), ("teacher", teacher), ("student, teacher's labels", student),                     ("lambdarank, human grades, same features", human)]:    print(f"{name}, train ndcg@10: {scores.mean():.3f}")# the student's score minus the human-graded ranker's, per query, tested by permutationresult = compare(student, human)print(f"student minus the human-graded ranker: {result['mean difference']:.3f}")print(f"student minus the human-graded ranker, permutation p: {result['permutation p']:.3f}")
输出python distillation-on-train.py
semantic, train ndcg@10: 0.772
teacher, train ndcg@10: 0.785
student, teacher's labels, train ndcg@10: 0.791
lambdarank, human grades, same features, train ndcg@10: 0.747
student minus the human-graded ranker: 0.044
student minus the human-graded ranker, permutation p: 0.002

在这些训练查询和这些特征上,教师模型对每个候选项给出的标签训练出的排序器,优于人工评分训练出的排序器,其中未经评判的候选项按 0 计。学生模型的得分为 0.791,教师模型自身对候选项的排序得分为 0.785。

在人工评分上训练的模型得分为 0.747,低于语义检索的 0.772,学生模型减去它所得的差为 0.044,置换检验的 p 值为 0.002。这是在训练查询上做的调参检查,不构成证据。

然后在训练查询上拟合学生模型,并使用与 LambdaMART 相同的留出训练查询做早停,再进行最后一项计划内比较,阈值同为 0.01:

distilled-ranker-on-test.py Python 3.12.15 · bm25s 0.3.16 · lightgbm 4.7.0 · numpy 2.5.5 · pandas 3.1.1 · pystemmer 3.1.0 · scikit-learn 1.9.1 · scipy 1.18.1 通过
"""Fits a student LightGBM regression on the teacher's scores for the train queries, then reports its gain shares, howclosely its order follows similarity and the teacher, and its planned comparison on the test queries. The prose's pointrests on that comparison and on the taus printed for each ranking."""import numpy as npfrom lightgbm import LGBMRegressorfrom scipy.stats import kendalltaufrom lib.metrics import evaluatefrom lib.ranking import CHEAP_FEATURES, DETERMINISTIC, TREES, feature_table, fit, ranked, report_planned, stopping_splitfrom lib.semantic import SemanticSearchfrom lib.store import load_products, load_queries, qrelsproducts, train, test = load_products(), load_queries("train"), load_queries("test")table = feature_table(products, train)                       # the teacher's score is the label: no human gradesprint(f"train candidates the teacher labels: {len(table)}")print(f"of them, judged by nobody: {table['grade'].isna().sum()}")print(f"student features: {len(CHEAP_FEATURES)}")# early stopping on the same 10 held-out train queries that LambdaMART stops onstudent = fit(LGBMRegressor(**TREES, **DETERMINISTIC), table, "cross_encoder", CHEAP_FEATURES, *stopping_split(train.index))print(f"trees kept by early stopping: {student.best_iteration_}")candidates, judgments = feature_table(products, test), qrels("test")       # test queries: read once, to reportpredicted = student.predict(candidates[CHEAP_FEATURES])gain = student.booster_.feature_importance(importance_type="gain")# the three features with the most gainfor name, share in sorted(zip(CHEAP_FEATURES, gain / gain.sum()), key=lambda pair: -pair[1])[:3]:    print(f"student's share of gain, {name}: {share:.3f}")for name in ["cross_encoder", "similarity"]:                   # whose order does the student follow?    # candidates has a default index, so group.index picks this query's rows out of predicted    tau = [kendalltau(group[name], predicted[group.index]).statistic for _, group in candidates.groupby("query_id")]    print(f"tau between student and {name}, mean over test queries: {np.mean(tau):.3f}")teacher = evaluate(ranked(candidates, candidates["cross_encoder"]), judgments)print(f"teacher's own order of the candidates, test ndcg@10, not a planned comparison: {teacher['ndcg@10'].mean():.3f}")after = evaluate(ranked(candidates, predicted), judgments)before = evaluate(SemanticSearch().run(test, k=100), judgments)for name, frame in [("semantic", before), ("distilled", after)]:    print(f"{name}, test ndcg@10: {frame['ndcg@10'].mean():.3f}")    print(f"{name}, test judged@10: {frame['judged@10'].mean():.3f}")    # each query's top 10 is 10 places, so (1 - judged@10) times 10 counts its unjudged places    print(f"{name}, unjudged places in the test top 10s: {round((1 - frame['judged@10']).sum() * 10)}")report_planned("distilled", after["ndcg@10"], before["ndcg@10"])
输出python distilled-ranker-on-test.py
train candidates the teacher labels: 6896
of them, judged by nobody: 1975
student features: 7
trees kept by early stopping: 202
student's share of gain, similarity: 0.812
student's share of gain, name_match: 0.126
student's share of gain, bm25: 0.055
tau between student and cross_encoder, mean over test queries: 0.466
tau between student and similarity, mean over test queries: 0.753
teacher's own order of the candidates, test ndcg@10, not a planned comparison: 0.782
semantic, test ndcg@10: 0.830
semantic, test judged@10: 0.990
semantic, unjudged places in the test top 10s: 3
distilled, test ndcg@10: 0.826
distilled, test judged@10: 0.987
distilled, unjudged places in the test top 10s: 4
distilled, mean difference: -0.004
distilled, wins, losses, ties: 10, 13, 7
distilled, permutation p: 0.720
distilled, established at 0.01: no
distilled, t-test p: 0.694
distilled, t-test agrees: yes
distilled, 99% interval: -0.034 to 0.017
distilled, interval agrees: yes

教师模型为全部 6896 个训练候选项打了标签,其中 1975 个无人评判,早停保留了 202 棵树。

学生模型主要依赖 similarity,它占增益的 0.812,其后是 name_match(0.126)和 bm25(0.055)。在测试候选项上,它的分数与语义相似度在各查询上的平均 tau 为 0.753,与教师模型的一致程度为 0.466。可见学生模型学到的主要是语义检索的排序,并在名称匹配和 BM25 匹配的推动下向教师模型的排序靠拢。

学生模型并未证明有所提升:蒸馏排序器的 nDCG@10 为 0.826,语义检索为 0.830,置换检验的 p 值为 0.720(见决策表)。教师模型对测试候选项的自身排序,本章并未计划检验,得分为 0.782。

学生模型的 Judged@10 为 0.987,语义检索为 0.990;学生模型的前 10 名中有 4 个未经评判的位置,语义检索则为 3。

工作准则: 当重排器带来了已确立的提升、但每次查询的成本过高时,就进行蒸馏:用教师模型为训练查询的每个候选项打标签,用无需教师模型即可计算的特征训练学生模型,并在留出查询上将学生模型与你的检索做对比检验。查看学生模型的增益重要性:当检索自身的分数在其中占主导时,学生模型基本上就是你的检索。当学生模型无法匹配教师模型的尺度时,使用成对商品之间的差值。商店目前还没有胜过语义检索的教师模型,因此保留代码,不上线学生模型。

生产环境中会出什么问题

最近一次修改见于版次 2026.11.8

线上排序器在购物者等待时计算特征并调用模型,而它所用的代码和模型版本,可能与训练和测试时所用的版本不一致。

特征在训练与服务之间发生漂移。 如果线上系统的分词方式不同、对 BM25 重新缩放,或升级了嵌入模型,已学到的权重面对的就是不同的数值,而且不会有任何报错。应通过同一条共享的代码路径计算特征,并将特征与每次返回的排序一并记录。

行顺序会泄漏到训练中。 LightGBM 的 lambdarank 从输入行的顺序出发,而排序总要以某种方式打破并列。按某个重排序器的得分排好的文件,会把该重排序器的排序交给模型。在每次拟合和每次打分之前,应按 ID 对候选排序,或用固定种子将其打乱。

判断来自一个候选池。 WANDS 只对其自身系统给出的商品做了判断(第 2 章)。把未经判断的候选视为不相关,会教会模型压低那些系统没有找到的商品。对你加入的每个排序器,都应检查 judged@10,并在相信某项提升之前,先让其未经判断的商品得到判断。

LLM 评判者偏向 LLM 重排序器。 在一项研究中,基于 LLM 评估者构建的排名,平均将基于 LLM 的重排序器高估了 9 至 17 位(Fröbe et al., 2025),而同时用 LLM 排序和判断是循环论证,“说明了对基于 LLM 的评估的依赖如何扭曲系统排名”(Clarke and Dietz, 2024)。应像本章这样,用人工判断为 LLM 重排序器打分。

LLM 重排序器对应唯一的模型版本和唯一的提示词。 不同的摘要值(标识确切权重的哈希)、量化方式(权重存储的精度)或提示词措辞,都可能改变每一个答案;再加上位置偏差,检索顺序的变化同样可能如此。LLM 记录的清单逐字记录了摘要值和两个提示词;两者中任何一个变化时,都应重新度量。

不熟悉的查询可能表现更差。 一项对 22 种重排序方法的比较发现,LLM 重排序器在熟悉的查询上表现更好,而“它们对新查询的泛化能力不一”(Abdallah et al., 2025)。应留意涉及比模型训练数据更新的商品和风格的查询。

每个重排序器都会在购物者的路径上增加一次调用。 应像预算中那样统计每次查询的调用次数和文本量,设置超时,并在重排序器未能及时响应时回退到检索的顺序。

工作准则: 随每次排序记录特征和模型版本,将候选按不携带任何信息的顺序排列,依据人工判断打分,检查 judged@10,并为每个重排序阶段设定时限和回退方案。

本章所做的决定

最近一次修改见于版次 2026.11.8

本章的每一项选择都是在训练查询上做出的,只有在测试查询上成立的计划内比较,才能改变商店的搜索。

5 项计划内比较均以 nDCG@10 与语义检索的 0.830 相比较,其成立所依据的 p 值低于 0.01:

指标 LambdaMART 交叉编码器 LLM 逐点式 LLM 列表式 蒸馏模型
nDCG@10 0.817 0.813 0.837 0.819 0.826
平均差值 -0.012 -0.016 0.008 -0.010 -0.004
胜 / 负 / 平 14, 12, 4 11, 12, 7 11, 8, 11 12, 12, 6 10, 13, 7
置换检验 p 值 0.260 0.178 0.241 0.580 0.720
是否成立 no no no no no
t 检验 p 值 0.278 0.171 0.244 0.498 0.694
99% 区间 -0.044 to 0.011 -0.058 to 0.006 -0.007 to 0.026 -0.070 to 0.016 -0.034 to 0.017
t 检验是否一致 yes yes yes yes yes
区间是否一致 yes yes yes yes yes
  • 5 项计划内比较:与语义检索相比,基于人工判断,成立所依据的 p 值低于 0.01。
  • 候选集:BM25 与语义检索各自前 100 名的并集,按 product_id 顺序排列,得分相同时按 product_id 决定先后。
  • LambdaMART:label_gain=[0, 1, 2],未标注判断的候选按等级 0 处理,3 个叶子,学习率 0.1,通过在训练集上的交叉验证选定,并借助 eval_X 与 eval_y,在留出的训练查询上提前停止。它的增益在测试集上未能成立,因此不予采用。
  • 重排深度:语义检索的前 20 名,为与 LLM 的记录保持一致而固定;在训练集上,交叉编码器的曲线在 20 到 30 之间趋于平缓。没有任何一个重排器的增益在测试集上成立,因此都不采用。
  • LLM 重排器只依据人工判断打分,列表式重排器还在两种输入顺序下做了检查。它的回答取决于所展示的顺序。
  • 成本只计数,不计时:统计每个查询的调用次数和字符数。
  • 蒸馏:以交叉编码器的得分训练 LightGBM 学生模型,除检索本身外不再调用模型。它学到的主要是语义检索的排序,没有显示出成立的增益,因此不予采用。
  • 需要超越的排序器仍然只是语义检索,以未做词干提取的 BM25 作为基线。
  • **本章的结论:**没有任何第二阶段证明了自己的价值;它们都没有显示出成立的增益。

本章,在本版次中

版次 2026.11.10示例于 2026年11月19日 通过本章 14个示例 个示例中有 14 个通过

译自英文版图书,版次 2026.11.10。

引用本节

格式

每种格式均引用英文原书,本译本以其为准。

每种格式都会注明该版次。其链接会打开当前版次,节标题处会说明该节文本最后一次修改的时间。访问日期和检索日期即为您打开此页的当天。

本页内容

第 9 章,共 11 章: Ranking

  1. 用特征描述每个候选商品
  2. Learn a ranking from pairs of products
  3. 用 LightGBM 训练 LambdaMART
  4. Report LambdaMART on the test queries
  5. 用交叉编码器重排语义检索结果
  6. 用 LLM 为每个商品打分
  7. 用一条 LLM 提示词对 20 个商品排序
  8. Reverse the input to measure positional bias
  9. 根据预算设定重排深度
  10. 将交叉编码器蒸馏为小型排序器
  11. 生产环境中会出什么问题
  12. 本章所做的决定

返回顶部全部章节