Ranking

閱讀約需 97 分鐘 分鐘,12 節 中共有 14 個範例。

版次 2026.11.10 範例於 2026年11 月 19 通過測試 132 個範例中有 132 個通過 變更內容

由 AI 翻譯。經人工以英文驗證。 我們如何製作書籍

譯自英文書,版次 2026.11.10。 閱讀原文

此部分尚未翻譯,因此以英文呈現。

By the end of this chapter you can describe each product that retrieval returns with features, train a learning-to-rank model with LightGBM on your own judgments, rerank the top of the list with a cross-encoder or an LLM, measure an LLM reranker's positional bias, set the rerank depth by counting its costs, and distil a slow reranker into a fast one. You can also decide, with a planned comparison on the test queries, whether any of them beats your retrieval.

Semantic retrieval alone is still the ranker to beat at the store, where every search is ordered by one embedding score and nothing else. It ranks products by embeddings from Snowflake/snowflake-arctic-embed-m-v1.5, and scores 0.830 nDCG@10 on the 30 test queries. No change in chapters 5 to 8 has beaten it by an established margin. This chapter asks whether a second stage can: a model trained on the store's own judgments, a reranker that reads the query and the product together, or an LLM. Chapter 1's cross-encoder reranked the hybrid run, not semantic retrieval, and never saw the store's judgments.

So this chapter tries all three on semantic retrieval's top results, each scored against the store's judgments. Reranking reorders the top of retrieval's list with a slower model. The store's rerankers are recorded: each ran once and saved its outputs, which the examples read.

The store's LLM is the open model gemma4:12b-it-q4_K_M, licensed Apache-2.0. It ran locally with chapter 4's settings, which make it take its likeliest next word every time, plus thinking off, so it answers without writing out reasoning, and a JSON schema, a machine-readable description of the answer's shape. recordings/llm_rerank.py reranked semantic retrieval's top 20 for the test queries only, so nothing about the LLM rerankers can be tuned. Every score uses the human judgments, because in one study LLM judges ranked LLM rerankers too high (What goes wrong in production). These results describe one model and two prompts, so treat them as the store's, not as a verdict on LLM rerankers.

The chapter fixed its 5 planned comparisons on the test queries before it read any test result. Each sets one method against semantic retrieval's run on nDCG@10:

  1. LambdaMART, a learned ranker trained on the train queries' judgments, ordering each query's candidate products
  2. the cross-encoder reranking semantic retrieval's top 20
  3. the LLM scoring each of the top 20 on its own
  4. the LLM ordering the top 20 in one prompt
  5. a small distilled ranker trained on the cross-encoder's scores

The bar is the book's rule: 5 planned comparisons split its 0.05, so each needs a paired permutation test p-value below 0.01. The paired t-test and a 99% bootstrap interval stand beside it.

用特徵描述每個候選商品

最近一次變動於第 2026.11.10 版

本章規劃的比較中,第一項是學習式排序器,它不讀文字。學習式排序器會根據一串數值(也就是它的特徵),為每一組查詢與商品配對打分數,並從評判結果中學習如何為這些特徵加權。查詢相依特徵同時描述查詢與商品,例如 BM25 分數。查詢無關特徵只描述商品本身,例如有多少購物者評價過它。網頁搜尋的特徵集通常兩者並用:LETOR 4.0 是學習排序的基準資料集,含有 46 個特徵,其中包括「BM25 of title」、「PageRank」與「Number of slash in URL」(Qin and Liu, 2013)。排序器只為候選商品打分數,也就是成本較低的檢索階段所回傳的商品,因此它的特徵可以比較昂貴。

商店每個查詢的候選商品,就是交叉編碼器(cross-encoder)錄製結果中已評分的商品:BM25 前 100 名與語意檢索前 100 名的聯集。lib.ranking.feature_table 為每個候選商品提供 8 個特徵:

  • bm25:來自 lib.lexical 的 BM25 分數,這是本書未做詞幹還原的基準線;若商品與查詢沒有任何共同詞彙,則為 0
  • similarity:查詢與商品嵌入向量的餘弦相似度
  • cross_encoder:已記錄的交叉編碼器分數
  • name_match 與 attribute_match:查詢詞彙(依第 5 章的方式斷詞)出現在商品名稱與屬性中的比例
  • rating_count、average_rating 與 review_count,由商店資料提供給每個商品

目錄中有 2632 的商品沒有評分,它們的三個評分特徵維持缺值。資料表也包含每個候選商品的人工等級;若無人評判,則留空。為訓練查詢建立資料表,先看單一查詢,再依等級比較各個特徵:

ranking-features-by-grade.py Python 3.12.15 · bm25s 0.3.16 · lightgbm 4.7.0 · numpy 2.5.5 · pandas 3.1.1 · pystemmer 3.1.0 · scikit-learn 1.9.1 · scipy 1.18.1 通過
"""Builds the feature table for the train queries, shows one query's candidates, and compares each feature's mean acrossthe human grades. The prose's point rests on the mean cross_encoder line for each grade and the mean of each feature by grade."""import pandas as pdfrom lib.ranking import FEATURES, feature_tablefrom lib.store import load_products, load_queriesproducts, queries = load_products(), load_queries("train")      # train queries: features are design worktable = feature_table(products, queries)print(f"features: {len(FEATURES)}")print(f"train candidates: {len(table)}")print(f"train candidates per query, median: {table.groupby('query_id').size().median():.0f}")print(f"share of train candidates nobody judged: {table['grade'].isna().mean():.3f}")query_id = queries.index[queries["query"] == "laundry basket with wheels"][0]one = table[table["query_id"] == query_id].sort_values("cross_encoder", ascending=False)one = one.assign(name=products.loc[one["product_id"], "name"].str[:38].to_numpy())pd.set_option("display.width", 200)print(f"\ncandidates for laundry basket with wheels: {len(one)}")print(one[["name", "bm25", "similarity", "cross_encoder", "name_match", "attribute_match", "rating_count", "grade"]]      .head(6).round(3).to_string(index=False))print(f"top 6 by cross-encoder, exact: {(one['grade'].head(6) == 2).sum()}")# grades 2, 1 and 0 are exact, partial and irrelevant; a missing grade means nobody judged the pairlabel = table["grade"].map({2: "exact", 1: "partial", 0: "irrelevant"}).fillna("unjudged")means = table[FEATURES].groupby(label).mean().loc[["exact", "partial", "irrelevant", "unjudged"]]print("\nmean of each feature by grade, train candidates:")print(means.round(3).to_string())for grade in means.index:    print(f"mean cross_encoder, {grade}: {means.at[grade, 'cross_encoder']:.3f}")relevance = ["bm25", "similarity", "cross_encoder", "name_match"]nearer = [name for name in relevance if abs(means.at["unjudged", name] - means.at["irrelevant", name])          < abs(means.at["unjudged", name] - means.at["partial", name])]print(f"relevance features where unjudged sits nearer irrelevant than partial: {len(nearer)} of {len(relevance)}")medians = table[["rating_count", "review_count"]].groupby(label).median().loc[means.index]  # counts are skewedfor grade in ["exact", "partial", "irrelevant"]:    print(f"rating_count, {grade}, mean: {means.at[grade, 'rating_count']:.3f}")    print(f"rating_count, {grade}, median: {medians.at[grade, 'rating_count']:.1f}")print(f"products with no rating: {products['rating_count'].isna().sum()}")
輸出python ranking-features-by-grade.py
features: 8
train candidates: 6896
train candidates per query, median: 131
share of train candidates nobody judged: 0.286

candidates for laundry basket with wheels: 125
                                  name  bm25  similarity  cross_encoder  name_match  attribute_match  rating_count  grade
       laundry sorter basket on wheels 8.229       0.597          3.433       1.000              0.0           NaN    2.0
                vintage laundry basket 6.270       0.572          2.995       0.667              0.0         638.0    2.0
         rolling wicker laundry basket 5.494       0.588          2.974       0.667              0.0         110.0    2.0
aluminum frame laundry basket can be f 4.739       0.524          2.941       0.667              0.0          14.0    2.0
    rebrilliant rolling laundry hamper 5.537       0.558          2.804       0.333              0.0          17.0    2.0
4-bag laundry hamper sorter cart baske 6.012       0.543          2.739       0.667              0.0           NaN    2.0
top 6 by cross-encoder, exact: 6

mean of each feature by grade, train candidates:
             bm25  similarity  cross_encoder  name_match  attribute_match  rating_count  average_rating  review_count
grade
exact       4.317       0.517          2.191       0.562            0.024       466.879           4.537       333.589
partial     3.936       0.454          1.513       0.461            0.030       385.454           4.539       288.779
irrelevant  2.760       0.360          1.159       0.404            0.006       246.781           4.552       159.052
unjudged    2.323       0.369          1.050       0.140            0.018       339.904           4.520       237.027
mean cross_encoder, exact: 2.191
mean cross_encoder, partial: 1.513
mean cross_encoder, irrelevant: 1.159
mean cross_encoder, unjudged: 1.050
relevance features where unjudged sits nearer irrelevant than partial: 4 of 4
rating_count, exact, mean: 466.879
rating_count, exact, median: 43.0
rating_count, partial, mean: 385.454
rating_count, partial, median: 31.0
rating_count, irrelevant, mean: 246.781
rating_count, irrelevant, median: 35.5
products with no rating: 2632

訓練查詢共有 6896 個候選商品,每個查詢的中位數為 131 個,其中有 0.286 的候選商品無人評判。

laundry basket with wheels 有 125 個候選商品。依交叉編碼器分數排名的前六名中,有 6 個是完全相關的商品。它們的 attribute_match 為 0,因為屬性記載的是顏色、材質與尺寸,而不是 wheels 這類詞彙。

在所有訓練候選商品中,交叉編碼器的平均分數由完全相關商品的 2.191,降到部分相關商品的 1.513,再降到不相關商品的 1.159;bm25、similarity 與 name_match 也呈現相同的下降趨勢。average_rating 與 attribute_match 則幾乎沒有變化。

rating_count 的平均數隨等級下降,中位數卻沒有:少數評價數很高的商品拉動了平均數。

等級 評分數平均 評分數中位數
exact 466.879 43.0
partial 385.454 31.0
irrelevant 246.781 35.5

未評判候選商品的交叉編碼器平均分數為 1.050;在 4 of 4 個隨等級下降的相關性特徵上,它們的平均值比較接近不相關商品,而非部分相關商品。這會影響我們該讓未評判的候選商品教給模型什麼。

實用原則: 從你已經在計算的特徵著手:每個檢索器的分數、若有重排器則加上它的分數、名稱等欄位的比對結果,以及熱門度訊號。訓練之前,先在訓練查詢上依等級比較每個特徵,分數用平均數,偏態的計數用中位數。不隨等級變動的特徵,能教給模型的很少。訓練與上線服務時,請用同一份程式碼計算特徵。

此部分尚未翻譯,因此以英文呈現。

Learn a ranking from pairs of products

最近一次變動於第 2026.11.10 版

Learning to rank trains a model on judged queries to order candidates by their features. Its methods fall into three families, pointwise, pairwise and listwise (Liu, 2009). A pointwise model predicts each product's grade on its own, as a regression. A pairwise model learns which of two products should rank higher. A listwise model scores a whole ranking. Liu's 2009 comparison on the LETOR benchmark favoured the listwise approach, though tentatively. The obstacle for all three is the metric: nDCG changes only when two products swap places, so ranking metrics "are everywhere either discontinuous or flat, so gradient descent appears to be problematic" (Burges, 2010).

RankNet, a pairwise method, works around it with a smooth loss on pairs. For two products where i should rank above j, its gradient is large when the model scores them the wrong way round or nearly level, and small when it already has them well apart. LambdaRank keeps that gradient and multiplies it by how much nDCG would change if the two products swapped places (Burges, 2010, section 4.1). Burges writes the gradient with a minus sign, as the slope of a cost; here a positive λ pushes i up:

λ_ij    = σ / (1 + exp(σ · (s_i − s_j))) · |ΔnDCG_ij|      s: the model's scores, σ = 1
ΔnDCG_ij = (g_i − g_j) · (1 / log2(1 + r_i) − 1 / log2(1 + r_j)) / ideal DCG      g: grades, r: positions

A product's total is its λ summed over the pairs where it should rank higher, minus those where it should rank lower. Burges (2010) also reads the λ's as forces, pushing products up and down the list.

LambdaMART uses LambdaRank's λ as the gradients for gradient-boosted trees, which add small decision trees one at a time, each fitted to the gradients of the trees before it: "LambdaMART is the boosted tree version of LambdaRank, which is based on RankNet". An ensemble of LambdaMART rankers won Track 1 of the Learning to Rank Challenge that Yahoo ran in 2010 (Burges, 2010).

One train query, outdoor waterproof chest, shows the arithmetic. Let semantic similarity be the model's scores so far, compute every pair's λ in its top 10 by hand, check each |ΔnDCG| by swapping the pair and recomputing nDCG with lib.metrics, and add up the forces:

lambdas-for-one-query.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 通過
"""Works out LambdaRank's λ for every pair in one train query's top 10 by hand, and checks each pair's |ΔnDCG| againstswapping the pair in the metric. The prose's point rests on the strongest push up and down, and on the count of pairswhose formula matches swapping."""import mathfrom itertools import combinationsfrom lib.metrics import dcg_at_k, ndcg_at_kfrom lib.semantic import SemanticSearchfrom lib.store import load_queries, qrelsqueries, judgments = load_queries("train"), qrels("train")query_id = queries.index[queries["query"] == "outdoor waterproof chest"][0]judged = judgments[query_id]scores = SemanticSearch().search(query_id, k=10)       # the model so far: semantic similarityranking, similarity = list(scores.index), scores.to_numpy()# an unjudged product counts as grade 0, as it does in the metricgrades = [judged.get(product_id, 0) for product_id in ranking]# the ideal DCG takes every judged product, as lib.metrics does; sigma is 1, LightGBM's default sigmoidideal, sigma = dcg_at_k(sorted(judged.values(), reverse=True), 10), 1.0print("grades from the top:", " ".join(map(str, grades)))force, checks, pulls = [0.0] * len(ranking), [], []for i, j in combinations(range(len(ranking)), 2):      # i is above j    if grades[i] == grades[j]:        continue    better, worse = (i, j) if grades[i] > grades[j] else (j, i)    ranknet = 1 / (1 + math.exp(sigma * (similarity[better] - similarity[worse])))      # RankNet's pull    # i and j count from 0, so position p is i + 1 and its discount is 1 / log2(p + 1), which is 1 / log2(i + 2)    delta = abs((grades[i] - grades[j]) * (1 / math.log2(i + 2) - 1 / math.log2(j + 2))) / ideal    swapped = ranking.copy()    swapped[i], swapped[j] = swapped[j], swapped[i]    checks.append(math.isclose(delta, abs(ndcg_at_k(swapped, judged, 10) - ndcg_at_k(ranking, judged, 10))))    pulls.append(ranknet)    pair_lambda = sigma * ranknet * delta                         # LambdaRank: scaled by |delta nDCG|    force[better], force[worse] = force[better] + pair_lambda, force[worse] - pair_lambdaprint(f"pairs with different grades: {len(checks)}")print(f"pairs where delta ndcg by formula matches swapping the pair: {sum(checks)}")for position, push in enumerate(force, start=1):    print(f"force at position {position}: {push:+.3f}")# list indexes count from 0 and positions from 1, hence the + 1print(f"strongest push up, position: {force.index(max(force)) + 1}")print(f"strongest push down, position: {force.index(min(force)) + 1}")print(f"ranknet's pull, lowest: {min(pulls):.3f}\nranknet's pull, highest: {max(pulls):.3f}")
輸出python lambdas-for-one-query.py
grades from the top: 1 2 2 0 1 2 0 2 0 0
pairs with different grades: 32
pairs where delta ndcg by formula matches swapping the pair: 32
force at position 1: +0.013
force at position 2: +0.159
force at position 3: +0.104
force at position 4: -0.083
force at position 5: -0.010
force at position 6: +0.063
force at position 7: -0.092
force at position 8: +0.063
force at position 9: -0.105
force at position 10: -0.111
strongest push up, position: 2
strongest push down, position: 10
ranknet's pull, lowest: 0.472
ranknet's pull, highest: 0.525

The formula's |ΔnDCG| matches the change from swapping the pair for 32 of the 32 pairs in the top 10 with different grades. The top 10's grades, from the top, are 1 2 2 0 1 2 0 2 0 0; an unjudged product counts as 0, as it does in the metric.

The strongest push up goes to position 2, an exact product, with +0.159; the strongest push down goes to position 10, an irrelevant product, with -0.111. The partial product at position 1 gets a net push of +0.013: the irrelevant products below push it up, and the exact ones push it down.

RankNet's factor in λ stays between 0.472 and 0.525, because the similarities lie close together, so |ΔnDCG| decides almost everything.

That example simplifies. It takes the ideal DCG from every judged product, as lib.metrics does, and looks at ten products. LightGBM takes the ideal from the query's training rows, and at 4.7.0 two defaults change the λ. lambdarank_truncation_level is 30, which limits training to the top 30 results: only pairs with a product in the current top 30 get a λ. lambdarank_norm is true, which normalizes the lambdas across queries; set it to false to get the original lambdarank algorithm (Parameters). Normalizing, as LightGBM's source does it, divides each pair's |ΔnDCG| by the gap between its scores plus 0.01, then rescales each query's λ (rank_objective.hpp). To check the formula against the library, write it as a custom LightGBM objective, grow five small trees on this query's candidates, and grow them again with LightGBM's lambdarank, with and without those defaults:

lambdas-match-lightgbm.py Python 3.12.15 · bm25s 0.3.16 · lightgbm 4.7.0 · numpy 2.5.5 · pandas 3.1.1 · pystemmer 3.1.0 · scikit-learn 1.9.1 · scipy 1.18.1 通過
"""Writes LambdaRank's gradients by hand as a custom LightGBM objective, then checks them against LightGBM's own lambdarankon one query, with and without its defaults. The prose's point rests on the two printed differences: a rounding-sized gapwithout the defaults, and a large one at them."""import numpy as npfrom lightgbm import LGBMRankerfrom lib.ranking import DETERMINISTIC, FEATURES, feature_tablefrom lib.store import load_products, load_queriesqueries = load_queries("train")one = queries[queries["query"] == "outdoor waterproof chest"]rows = feature_table(load_products(), one)                  # one query's candidates, in product_id order# unjudged candidates count as grade 0, as in the metricgrades = rows["grade"].fillna(0).astype(int).to_numpy()SIGMA = 1.0                                                   # LightGBM's default sigmoiddef lambdas(y_true, y_pred, weight, group):    """The chapter's lambda for every pair with different grades, as gradients and second derivatives."""    rank = np.empty(len(y_pred), dtype=int)    rank[np.argsort(-y_pred, kind="stable")] = np.arange(len(y_pred))   # LightGBM's stable sort on score    # rank counts from 0, so the product at position rank + 1 has the discount 1 / log2(rank + 2)    discount, ideal = 1 / np.log2(rank + 2), np.sum(np.sort(y_true)[::-1] / np.log2(np.arange(len(y_true)) + 2))    gradient, hessian = np.zeros(len(y_pred)), np.zeros(len(y_pred))    for i in range(len(y_true)):        for j in np.flatnonzero(y_true < y_true[i]):                      # i should rank above j            delta = (y_true[i] - y_true[j]) * abs(discount[i] - discount[j]) / ideal            pull = 1 / (1 + np.exp(SIGMA * (y_pred[i] - y_pred[j])))            gradient[i], gradient[j] = gradient[i] - SIGMA * pull * delta, gradient[j] + SIGMA * pull * delta            hessian[[i, j]] += SIGMA ** 2 * pull * (1 - pull) * delta    return gradient, hessian# five small trees; min_child_samples is 5 where LightGBM's default is 20, so the trees can split this query's candidatestree = {"n_estimators": 5, "num_leaves": 3, "min_child_samples": 5, **DETERMINISTIC}# turns off normalization and sets truncation to every candidate, so every pair gets a λ, as the formula has itplain = {"lambdarank_norm": False, "lambdarank_truncation_level": len(rows)}   # LightGBM's lambdas, untouchedmodels = {"by hand": LGBMRanker(objective=lambdas, **tree),          "lightgbm, no normalization or truncation": LGBMRanker(objective="lambdarank", label_gain=[0, 1, 2], **plain, **tree),          "lightgbm at its defaults": LGBMRanker(objective="lambdarank", label_gain=[0, 1, 2], **tree)}# raw scores: the values the gradients act on, before any sigmoidscores = {name: model.fit(rows[FEATURES], grades, group=[len(rows)]).predict(rows[FEATURES], raw_score=True)          for name, model in models.items()}print(f"candidates: {len(rows)}")for name in list(models)[1:]:    difference = np.abs(scores[name] - scores["by hand"]).max()    print(f"{name}, largest difference from the lambdas by hand after 5 trees: {difference:.6f}")
輸出python lambdas-match-lightgbm.py
candidates: 180
lightgbm, no normalization or truncation, largest difference from the lambdas by hand after 5 trees: 0.000006
lightgbm at its defaults, largest difference from the lambdas by hand after 5 trees: 0.595667

Over the query's 180 candidates, LightGBM without normalization or truncation scores every product within 0.000006 of the custom objective's scores. The gap is rounding, since LightGBM reads its sigmoid from a precomputed table (rank_objective.hpp). At its defaults, scores differ by up to 0.595667.

Working rule: Train rankers with a gradient that weighs each pair by the metric you report, as LambdaMART does, rather than regressing on grades. Know which defaults of your library change that gradient, and check on train queries that it beats a pointwise model on your data, because the families' order in a benchmark doesn't transfer by itself.

以 LightGBM 訓練 LambdaMART

最近一次變動於第 2026.11.8 版

LightGBM 以 LGBMRanker 搭配 objective="lambdarank" 來訓練 LambdaMART。有四項設定需要留意,每一項在 4.7.0 中都有文件說明:

  • 資料列依查詢分組。「資料應依查詢排序」,而 group 依該順序記錄每個查詢的資料列數,不是查詢 ID,因此各組大小的總和等於資料列總數(sum(group) = n_samples;Parameters;LGBMRanker)。
  • 增益預設為指數形式。label_gain 預設為 0,1,3,7,15,31,63,...,因此「標籤 2 的增益為 3」,而且每個標籤都必須小於增益的個數(Parameters)。這是 Burges 所定義的 DCG,增益為 2^grade − 1。本書的 nDCG 使用線性增益,所以要設定 label_gain=[0, 1, 2]。
  • **nDCG 預設在第 1 至第 5 個位置評估。**請設定 eval_at=[10],以 nDCG@10 作為停止依據,並且仍以 lib.metrics 報告最終分數。
  • **驗證資料放在 eval_X 與 eval_y。**在 4.7.0 中,eval_set 為「自 4.7.0 版起已棄用」,由 eval_X 與 eval_y 取代(LGBMRanker;版本說明)。這些範例傳入 eval_X、eval_y 與 eval_group。

提前停止會在每棵樹之後,對保留的訓練查詢評分;若再增加 30 棵樹仍沒有進步,就保留到最佳分數為止的那些樹。為了讓訓練能完全重現,這些範例使用單一執行緒、固定的隨機種子與 deterministic=True,後者「應能確保使用相同資料時結果穩定」(Parameters);同一條目也建議搭配 force_row_wise=True,以避免數值問題可能造成的不穩定。lib.ranking 保存這些設定,其 fit 負責套用分組與提前停止。

資料列的順序也很重要。第一棵樹之前,所有分數都是 0,因此 LightGBM 會把資料列的順序當作第一次計算 λ 時的起始排名,而分數相同時需要決定先後的規則。cross-encoder 的記錄把每個查詢的候選項目由最佳到最差排列,這會在模型學到任何東西之前,就把 cross-encoder 的排名交給模型。因此 feature_table 依 product_id 排序候選項目,而 lib.ranking.ranked 依 product_id 決定同分時的先後。

樹的大小以訓練查詢上的交叉驗證來選擇:把查詢分成五摺,用三摺擬合的模型替每一摺排名,並以第五摺進行提前停止,使每個查詢恰好被評分一次。嘗試八種設定,從 3 到 31 個葉節點,搭配兩種學習率;與指標的做法相同,未經判定的候選項目視為等級 0:

lambdamart-tree-size-on-train.py Python 3.12.15 · bm25s 0.3.16 · lightgbm 4.7.0 · numpy 2.5.5 · pandas 3.1.1 · pystemmer 3.1.0 · scikit-learn 1.9.1 · scipy 1.18.1 通過
"""Cross-validates LambdaMART on the train queries over eight tree sizes and learning rates, and names the best. The prose'spoint rests on the best setting's score, and on the score beside it for LightGBM's default of 31 leaves at 0.1."""from lightgbm import LGBMRankerfrom lib.metrics import evaluatefrom lib.ranking import DETERMINISTIC, FEATURES, LAMBDARANK, TREE_GRID, TREES, cross_validate, feature_tablefrom lib.semantic import SemanticSearchfrom lib.store import load_products, load_queries, qrelsqueries, judgments = load_queries("train"), qrels("train")          # tuning: train queries onlycandidates = feature_table(load_products(), queries)candidates["grade or 0"] = candidates["grade"].fillna(0).astype(int)  # unjudged counts as irrelevant, as in nDCGprint(f"train queries: {len(queries)}")print(f"semantic, train ndcg@10: {evaluate(SemanticSearch().run(queries, k=100), judgments)['ndcg@10'].mean():.3f}")results = {}for setting in TREE_GRID:                                            # LightGBM's default is 31 leaves at 0.1    name = f"{setting['num_leaves']} leaves, learning rate {setting['learning_rate']}"    # up to 500 trees; early stopping inside cross_validate keeps fewer    model = lambda: LGBMRanker(**LAMBDARANK, n_estimators=500, **setting, **DETERMINISTIC)    # one nDCG@10 per train query, each ranked by a model that never saw that query    results[name] = cross_validate(model, candidates, candidates, "grade or 0", FEATURES, judgments)    print(f"{name}, cross-validated ndcg@10: {results[name].mean():.3f}")    # the trees early stopping kept in each fold    print(f"{name}, trees kept per fold: {', '.join(map(str, results[name].attrs['trees']))}")best = max(results, key=lambda name: results[name].mean())print(f"settings tried: {len(results)}")print(f"best setting: {best}")print(f"lib.ranking.TREES holds it: {'yes' if best == f'{TREES['num_leaves']} leaves, learning rate {TREES['learning_rate']}' else 'no'}")
輸出python lambdamart-tree-size-on-train.py
train queries: 50
semantic, train ndcg@10: 0.772
3 leaves, learning rate 0.05, cross-validated ndcg@10: 0.780
3 leaves, learning rate 0.05, trees kept per fold: 18, 27, 52, 20, 85
3 leaves, learning rate 0.1, cross-validated ndcg@10: 0.798
3 leaves, learning rate 0.1, trees kept per fold: 99, 20, 8, 43, 38
7 leaves, learning rate 0.05, cross-validated ndcg@10: 0.744
7 leaves, learning rate 0.05, trees kept per fold: 2, 28, 30, 39, 55
7 leaves, learning rate 0.1, cross-validated ndcg@10: 0.772
7 leaves, learning rate 0.1, trees kept per fold: 80, 12, 13, 7, 20
15 leaves, learning rate 0.05, cross-validated ndcg@10: 0.720
15 leaves, learning rate 0.05, trees kept per fold: 4, 37, 175, 44, 120
15 leaves, learning rate 0.1, cross-validated ndcg@10: 0.728
15 leaves, learning rate 0.1, trees kept per fold: 19, 68, 136, 48, 53
31 leaves, learning rate 0.05, cross-validated ndcg@10: 0.679
31 leaves, learning rate 0.05, trees kept per fold: 15, 8, 91, 8, 22
31 leaves, learning rate 0.1, cross-validated ndcg@10: 0.737
31 leaves, learning rate 0.1, trees kept per fold: 11, 75, 166, 96, 58
settings tried: 8
best setting: 3 leaves, learning rate 0.1
lib.ranking.TREES holds it: yes

在 8 種設定中,最佳的是 3 leaves, learning rate 0.1,結果為 0.798;LightGBM 預設的 31 個葉節點、學習率 0.1(LGBMRanker)得分為 0.737。所有設定中的最佳者會高估自己,因此在保留的查詢上應預期較低的分數。

有一項選擇取決於商店的判定資料。訓練候選項目中有 0.286 的比例沒有人判定過。模型可以把它們視為等級 0,也可以只從已判定的候選項目學習;未判定並不等於不相關,但無論哪種做法,模型在查詢時都得替它們排名。在選定的樹大小下,以交叉驗證比較兩者,並附上以等級做逐點迴歸的結果:

lambdamart-choices-on-train.py Python 3.12.15 · bm25s 0.3.16 · lightgbm 4.7.0 · numpy 2.5.5 · pandas 3.1.1 · pystemmer 3.1.0 · scikit-learn 1.9.1 · scipy 1.18.1 通過
"""Compares three ways of training on the train queries, each by cross-validation: LambdaRank with unjudged candidates as 0,LambdaRank on judged candidates only, and pointwise regression. The prose's point rests on the differences and thepermutation p-values printed under each setup."""from lightgbm import LGBMRanker, LGBMRegressorfrom lib.metrics import evaluatefrom lib.ranking import DETERMINISTIC, FEATURES, LAMBDARANK, TREES, cross_validate, feature_tablefrom lib.semantic import SemanticSearchfrom lib.significance import comparefrom lib.store import load_products, load_queries, qrelsqueries, judgments = load_queries("train"), qrels("train")          # tuning: train queries onlycandidates = feature_table(load_products(), queries)candidates["grade or 0"] = candidates["grade"].fillna(0).astype(int)  # unjudged counts as irrelevant# the rows the model learns from when unjudged candidates are left out; it still ranks every candidate at query timejudged_only = candidates.dropna(subset=["grade"]).astype({"grade": int})setups = {                                                          # (model, rows it learns from, label)    "lambdarank, unjudged as 0": (lambda: LGBMRanker(**LAMBDARANK, **TREES, **DETERMINISTIC), candidates, "grade or 0"),    "lambdarank, unjudged left out": (lambda: LGBMRanker(**LAMBDARANK, **TREES, **DETERMINISTIC), judged_only, "grade"),    "pointwise regression, unjudged as 0": (lambda: LGBMRegressor(**TREES, **DETERMINISTIC), candidates, "grade or 0"),}semantic = evaluate(SemanticSearch().run(queries, k=100), judgments)["ndcg@10"]print(f"semantic, train ndcg@10: {semantic.mean():.3f}")scores = {name: cross_validate(model, rows, candidates, label, FEATURES, judgments)          for name, (model, rows, label) in setups.items()}chosen = scores["lambdarank, unjudged as 0"]for name, per_query in scores.items():    print(f"{name}, cross-validated ndcg@10: {per_query.mean():.3f}")    if name != "lambdarank, unjudged as 0":        # the mean difference is the first argument minus the second, tested by permutation over the queries        result = compare(per_query, chosen)        print(f"{name}, minus the first: {result['mean difference']:.3f}")        print(f"{name}, minus the first, permutation p: {result['permutation p']:.3f}")result = compare(chosen, semantic)print(f"first setup minus semantic: {result['mean difference']:.3f}")print(f"first setup minus semantic, permutation p: {result['permutation p']:.3f}")
輸出python lambdamart-choices-on-train.py
semantic, train ndcg@10: 0.772
lambdarank, unjudged as 0, cross-validated ndcg@10: 0.798
lambdarank, unjudged left out, cross-validated ndcg@10: 0.733
lambdarank, unjudged left out, minus the first: -0.064
lambdarank, unjudged left out, minus the first, permutation p: 0.005
pointwise regression, unjudged as 0, cross-validated ndcg@10: 0.769
pointwise regression, unjudged as 0, minus the first: -0.029
pointwise regression, unjudged as 0, minus the first, permutation p: 0.007
first setup minus semantic: 0.026
first setup minus semantic, permutation p: 0.242

把未判定候選項目視為 0 的得分高於將其排除,0.798 對 0.733,差距為 -0.064,排列檢定的 p 值為 0.005。訓練時從未見過未判定候選項目的模型,仍然得替它們排名,而這些特徵顯示它們看起來像不相關的商品。

逐點迴歸的得分為 0.769,差距為 -0.029(p = 0.007),因此在這家商店的訓練查詢上,與逐點模型對照的調校檢查偏向 LambdaMART。

相較於語意檢索的 0.772,所選設定的差距為 0.026,排列檢定的 p 值為 0.242:即使在選出它的那些查詢上,它也沒有顯示出訓練查詢能與隨機區分的進步。

**實用原則:**依查詢排序資料列並傳入 group,把 label_gain 設為指標所用的增益,透過 eval_X 與 eval_y 以保留的訓練查詢進行提前停止,並固定隨機種子與執行緒數。如第 3 章所建議,依商品 ID 排列每個查詢的候選項目,分數相同時也以同樣方式決定先後。除非訓練查詢上的交叉驗證另有結論,否則把未判定的候選項目視為等級 0。已判定的查詢很少時,請嘗試較小的樹。

此部分尚未翻譯,因此以英文呈現。

Report LambdaMART on the test queries

最近一次變動於第 2026.11.10 版

A model tuned on train queries is reported once on the test queries, by a comparison planned before looking. Feature importance shows what a model learned: LightGBM's "gain" importance is the "total gains of splits which use the feature" (Booster), so a feature with no gain plays no part in the score.

The first planned comparison uses the setup chosen on train: LambdaMART on all the features, with 3 leaves at a learning rate of 0.1, unjudged candidates as grade 0, and candidates in product_id order. It fits on 40 train queries, stops early on 10 others, and ranks each test query's candidates. A gain needs a permutation p-value below 0.01:

lambdamart-on-test.py Python 3.12.15 · bm25s 0.3.16 · lightgbm 4.7.0 · numpy 2.5.5 · pandas 3.1.1 · pystemmer 3.1.0 · scikit-learn 1.9.1 · scipy 1.18.1 通過
"""Fits LambdaMART on the train queries with the chosen setup, reports its early stopping and gain shares, then reports itsplanned comparison on the test queries against semantic retrieval. The prose's point rests on that comparison's permutationp-value, printed last, against the threshold printed first."""from lightgbm import LGBMRankerfrom lib.metrics import evaluatefrom lib.ranking import (DETERMINISTIC, FEATURES, LAMBDARANK, PLANNED_TEST_COMPARISONS, THRESHOLD, TREES, feature_table,                         fit, ranked, report_planned, stopping_split)from lib.semantic import SemanticSearchfrom lib.store import load_products, load_queries, qrelsproducts, train, test = load_products(), load_queries("train"), load_queries("test")print(f"planned test comparisons: {len(PLANNED_TEST_COMPARISONS)}")print(f"p-value threshold: {THRESHOLD:.2f}")table = feature_table(products, train)table["grade or 0"] = table["grade"].fillna(0).astype(int)            # the setup chosen on train# 10 train queries are held out, at random with a fixed seed, to stop onfit_ids, stop_ids = stopping_split(train.index)model = fit(LGBMRanker(**LAMBDARANK, **TREES, **DETERMINISTIC), table, "grade or 0", FEATURES, fit_ids, stop_ids)print(f"queries fitted: {len(fit_ids)}\nqueries held out for stopping: {len(stop_ids)}")curve = model.evals_result_["valid_0"]["ndcg@10"]                     # LightGBM's nDCG@10, one value per treeprint(f"trees tried: {len(curve)}\ntrees kept by early stopping: {model.best_iteration_}")print(f"stopping queries' ndcg@10, first tree: {curve[0]:.3f}")# best_iteration_ counts trees from 1 and the curve from 0, so the last tree kept is at index best_iteration_ - 1print(f"stopping queries' ndcg@10, trees kept: {curve[model.best_iteration_ - 1]:.3f}")gain = model.booster_.feature_importance(importance_type="gain")      # total gain of the splits on each feature# a feature with no gain was never split on, so it plays no part in the scoreprint(f"features the trees use: {(gain > 0).sum()} of {len(FEATURES)}")for name, share in sorted(zip(FEATURES, gain / gain.sum()), key=lambda pair: -pair[1]):    print(f"share of gain, {name}: {share:.3f}")candidates, judgments = feature_table(products, test), qrels("test")   # test queries: read once, to reportafter = evaluate(ranked(candidates, model.predict(candidates[FEATURES])), judgments)before = evaluate(SemanticSearch().run(test, k=100), judgments)for name, frame in [("semantic", before), ("lambdamart", after)]:    print(f"{name}, test ndcg@10: {frame['ndcg@10'].mean():.3f}")    print(f"{name}, test judged@10: {frame['judged@10'].mean():.3f}")    # each query's top 10 is 10 places, so (1 - judged@10) times 10 counts its unjudged places    print(f"{name}, unjudged places in the test top 10s: {round((1 - frame['judged@10']).sum() * 10)}")report_planned("lambdamart", after["ndcg@10"], before["ndcg@10"])
輸出python lambdamart-on-test.py
planned test comparisons: 5
p-value threshold: 0.01
queries fitted: 40
queries held out for stopping: 10
trees tried: 243
trees kept by early stopping: 213
stopping queries' ndcg@10, first tree: 0.568
stopping queries' ndcg@10, trees kept: 0.794
features the trees use: 8 of 8
share of gain, cross_encoder: 0.641
share of gain, similarity: 0.172
share of gain, name_match: 0.097
share of gain, bm25: 0.043
share of gain, review_count: 0.021
share of gain, rating_count: 0.021
share of gain, attribute_match: 0.002
share of gain, average_rating: 0.001
semantic, test ndcg@10: 0.830
semantic, test judged@10: 0.990
semantic, unjudged places in the test top 10s: 3
lambdamart, test ndcg@10: 0.817
lambdamart, test judged@10: 0.990
lambdamart, unjudged places in the test top 10s: 3
lambdamart, mean difference: -0.012
lambdamart, wins, losses, ties: 14, 12, 4
lambdamart, permutation p: 0.260
lambdamart, established at 0.01: no
lambdamart, t-test p: 0.278
lambdamart, t-test agrees: yes
lambdamart, 99% interval: -0.044 to 0.011
lambdamart, interval agrees: yes

Of 243 trees tried, early stopping keeps 213. On the queries held out for stopping, LightGBM's own nDCG@10, whose ideal comes from their candidates alone, rises from 0.568 after the first tree to 0.794 at the trees kept.

The trees use 8 of 8 features, but they lean on the cross-encoder, which takes a share of 0.641 of the gain, ahead of similarity at 0.172, name_match at 0.097 and bm25 at 0.043. So serving the model means a cross-encoder call for every candidate.

LambdaMART doesn't establish a gain on the test queries: it scores 0.817 nDCG@10, against semantic retrieval's 0.830, at a permutation p-value of 0.260 (all five comparisons). Judged@10 is 0.990 for LambdaMART and 0.990 for semantic retrieval.

Working rule: Adopt a learned ranker only when its planned test comparison is established. Read its gain importance before you trust it, because the features it leans on are the ones you'll pay for at query time. Early stopping on a few held-out queries keeps very different numbers of trees from one split to the next, from 99, 20, 8, 43, 38 across the cross-validation folds to 213 here. A rule of thumb, not a measurement: with this few queries, judge more queries before you tune further, and treat any one stopping point as noise.

以 cross-encoder 重新排序語意檢索結果

最近一次變動於第 2026.11.10 版

語意檢索使用嵌入模型,又稱為 bi-encoder,它分別讀取查詢與每項商品,因此商品嵌入可以事先計算。cross-encoder則是將查詢與一項商品一起讀取,並輸出一個分數(Nogueira and Cho, 2019)。它能讓每個查詢詞與每個商品詞互相關聯,但這要付出代價。它通常比 Sentence Transformer 模型慢,因為它是針對每個配對而非每段文字進行計算,所以「Cross Encoders 常用於對前 k 筆結果重新排序」(Sentence Transformers)。商店使用的 cross-encoder(見開頭)以語言模型而言規模很小,只有 149M 個參數(模型卡)。它是開放模型 Alibaba-NLP/gte-reranker-modernbert-base,其資訊清單(manifest,即與錄製結果一併儲存的檔案)將其授權標示為 Apache-2.0。我們選擇它,是因為任何人都可以下載並重新執行錄製。recordings/cross_encoder.py 為每個查詢的 BM25 與語意檢索前 100 名的聯集評分(有兩個查詢,其語意檢索清單與 lib.semantic 的清單在第 100 名上有近乎平手的差異)。

第 1 章重新排序的是混合檢索的結果。這裡由 cross-encoder 重新排序要超越的排序器,也就是語意檢索的前 20 名:這些商品與 LLM 重新排序器所讀取的相同,因此本章的三個重新排序器都從相同的清單出發。本章在看到任何測試結果之前就固定了這個深度,也不對它調校。依錄製的分數重新排列前 20 名,其餘維持原位。這是第二項預先規劃的比較,需要 p 值低於 0.01:

cross-encoder-on-semantic.py Python 3.12.15 · bm25s 0.3.16 · lightgbm 4.7.0 · numpy 2.5.5 · pandas 3.1.1 · pystemmer 3.1.0 · scikit-learn 1.9.1 · scipy 1.18.1 通過
"""Reranks semantic retrieval's top 20 with the recorded cross-encoder scores, and compares the two on the test queries.The prose's point rests on the planned comparison's permutation p-value, and on the judged@10 lines, which show what thererank does to the judgments it can score."""from lib.metrics import evaluatefrom lib.ranking import DEPTH, rerank, report_plannedfrom lib.recorded import cross_encoder_scores, llm_listwise, manifestfrom lib.semantic import SemanticSearchfrom lib.store import load_queries, qrelsqueries, judgments = load_queries("test"), qrels("test")       # scored on the human judgmentssemantic = SemanticSearch().run(queries, k=100)recorded = cross_encoder_scores()# the recorded score of each (query, product) pairscore = dict(zip(zip(recorded["query_id"], recorded["product_id"]), recorded["score"]))print("cross-encoder model:", manifest("cross-encoder")["model"])print("licence in the manifest:", manifest("cross-encoder")["model_licence"])# the set of products the LLM was shown for each query, from the listwise recordingshown = llm_listwise().groupby("query_id")["product_id"].apply(set)  # what the LLM rerankers sawsame = sum(set(semantic[query_id][:DEPTH]) == shown[query_id] for query_id in queries.index)print(f"rerank depth: {DEPTH}")print(f"queries whose top {DEPTH} holds the products the llm rerankers saw: {same} of {len(queries)}")reranked = rerank(semantic, score, DEPTH)                       # top 20 reordered, the rest in placebefore, after = evaluate(semantic, judgments), evaluate(reranked, judgments)for name, frame in [("semantic", before), ("cross-encoder", after)]:    print(f"{name}, test ndcg@10: {frame['ndcg@10'].mean():.3f}")    print(f"{name}, test judged@10: {frame['judged@10'].mean():.3f}")    print(f"{name}, unjudged places in the test top 10s: {round((1 - frame['judged@10']).sum() * 10)}")# index 0 is the first placemoved = sum(reranked[query_id][0] != semantic[query_id][0] for query_id in queries.index)print(f"queries whose first product changes: {moved}")report_planned("cross-encoder", after["ndcg@10"], before["ndcg@10"])
輸出python cross-encoder-on-semantic.py
cross-encoder model: Alibaba-NLP/gte-reranker-modernbert-base
licence in the manifest: Apache-2.0
rerank depth: 20
queries whose top 20 holds the products the llm rerankers saw: 30 of 30
semantic, test ndcg@10: 0.830
semantic, test judged@10: 0.990
semantic, unjudged places in the test top 10s: 3
cross-encoder, test ndcg@10: 0.813
cross-encoder, test judged@10: 0.983
cross-encoder, unjudged places in the test top 10s: 5
queries whose first product changes: 23
cross-encoder, mean difference: -0.016
cross-encoder, wins, losses, ties: 11, 12, 7
cross-encoder, permutation p: 0.178
cross-encoder, established at 0.01: no
cross-encoder, t-test p: 0.171
cross-encoder, t-test agrees: yes
cross-encoder, 99% interval: -0.058 to 0.006
cross-encoder, interval agrees: yes

重新排序前 20 名並未確立改善:重新排序的分數為 0.813,語意檢索為 0.830,排列檢定的 p 值為 0.178(見決策表)。有 30 of 30 個測試查詢的前 20 名包含 LLM 重新排序器看過的商品,而重新排序器改變了其中 23 個查詢的第一名商品。

Judged@10從 0.990 降到 0.983,測試用前 10 名中未判定的名次則從 3 增加到 5。

在訓練查詢上,相同的重新排序分數為 0.802,語意檢索則為 0.772(深度掃描)。訓練查詢是另一份樣本,其結論重複了第 3 章的發現:兩個相近排序器的順序,可能在不同的查詢集之間翻轉。

**實用原則:**對你最強的檢索結果進行重新排序,深度須在查看測試結果之前固定,並且只在保留查詢上確立了改善時,才採用該重新排序器。商店的通用 cross-encoder 不納入流程。

用 LLM 為每項商品評分

最近一次變動於第 2026.11.10 版

LLM 也能進行重新排序,方式有三種。逐點(pointwise):LLM 單獨判斷每項商品,並回傳標籤或分數。成對(pairwise):LLM 指出兩項商品中哪一項更符合查詢。列表式(listwise):LLM 一次回答就排出整份清單的順序(Qin et al., 2024)。在一項研究中,研究者使用 BEIR 的八個資料集(BEIR 是一組檢索基準,用來測試模型處理未經訓練資料的能力),結果顯示逐點提示在採用分級答案時,表現優於是非作答,原因是「缺少中間的相關性標籤選項,可能使 LLM 提供含雜訊或有偏誤的答案」(Zhuang et al., 2024a)。

商店使用的 LLM 是本章開頭提到的開放模型,它一次只看到查詢與一項商品:名稱、類別、分類、描述與屬性(LLM 看到的內容)。提示要求它給出一個標籤,即 exact、partial 或 irrelevant,以及「一個 0 到 100 的分數,表示商品符合查詢的程度」。接著依該分數重新排序語意檢索的前 20 名;若兩項商品同分,則維持語意檢索的順序,這項規則在錄製時即已固定。這次重新排序是第三項預先規劃的比較,其 p 值門檻為低於 0.01:

llm-pointwise-rerank.py Python 3.12.15 · bm25s 0.3.16 · lightgbm 4.7.0 · numpy 2.5.5 · pandas 3.1.1 · pystemmer 3.1.0 · scikit-learn 1.9.1 · scipy 1.18.1 通過
"""Reads the pointwise LLM's answers for the test queries, one call per product, and reranks semantic retrieval's top 20 bytheir scores. The prose's point rests on the distinct-scores line, which shows how coarse the scores are, and on theplanned comparison's p-value."""from lib.metrics import evaluatefrom lib.ranking import DEPTH, rerank, report_plannedfrom lib.recorded import llm_pointwise, manifestfrom lib.semantic import SemanticSearchfrom lib.store import load_queries, qrelsqueries, judgments = load_queries("test"), qrels("test")       # scored on the human judgments, not the LLM'ssemantic = SemanticSearch().run(queries, k=100)answers = llm_pointwise()                                       # one call per product: a label and a scoreprint("llm:", manifest("llm-rerank")["model"])print("licence in the manifest:", manifest("llm-rerank")["model_licence"])print(f"calls per query: {len(answers) // answers['query_id'].nunique()}")print(f"answers without a score: {answers['score'].isna().sum()}")print(f"distinct scores per query, median: {answers.groupby('query_id')['score'].nunique().median():.0f}")print(f"share of products scored 100: {(answers['score'] == 100).mean():.3f}")# each answered product's human grade, or missing when nobody judged itpeople = answers.apply(lambda row: judgments[row["query_id"]].get(row["product_id"]), axis=1)# grades 2, 1 and 0 are exact, partial and irrelevant, the labels the LLM was asked forfor label, grade in [("exact", 2), ("partial", 1), ("irrelevant", 0)]:    print(f"products the llm labels {label}: {(answers['label'] == label).sum()}")    print(f"products people judged {label}: {(people == grade).sum()}")print(f"products nobody judged: {people.isna().sum()}")score = dict(zip(zip(answers["query_id"], answers["product_id"]), answers["score"]))reranked = rerank(semantic, score, DEPTH)                       # equal scores keep semantic retrieval's ordercounts = answers.groupby("query_id")["score"].value_counts()          # products per score, for each queryshared = {query_id: sum(counts[(query_id, score[(query_id, product_id)])] > 1 for product_id in reranked[query_id][:10])          for query_id in queries.index}                                # top-10 products that share their score# more than 5 of the top 10 share a score: most of the list is decided by the tie-breakprint(f"queries where most of the reranked top 10 shares its score: {sum(n > 5 for n in shared.values())}")before, after = evaluate(semantic, judgments), evaluate(reranked, judgments)for name, frame in [("semantic", before), ("llm pointwise", after)]:    print(f"{name}, test ndcg@10: {frame['ndcg@10'].mean():.3f}")    print(f"{name}, test judged@10: {frame['judged@10'].mean():.3f}")    print(f"{name}, unjudged places in the test top 10s: {round((1 - frame['judged@10']).sum() * 10)}")report_planned("llm pointwise", after["ndcg@10"], before["ndcg@10"])
輸出python llm-pointwise-rerank.py
llm: gemma4:12b-it-q4_K_M
licence in the manifest: Apache-2.0
calls per query: 20
answers without a score: 0
distinct scores per query, median: 4
share of products scored 100: 0.390
products the llm labels exact: 251
products people judged exact: 310
products the llm labels partial: 345
products people judged partial: 265
products the llm labels irrelevant: 4
products people judged irrelevant: 7
products nobody judged: 18
queries where most of the reranked top 10 shares its score: 30
semantic, test ndcg@10: 0.830
semantic, test judged@10: 0.990
semantic, unjudged places in the test top 10s: 3
llm pointwise, test ndcg@10: 0.837
llm pointwise, test judged@10: 0.983
llm pointwise, unjudged places in the test top 10s: 5
llm pointwise, mean difference: 0.008
llm pointwise, wins, losses, ties: 11, 8, 11
llm pointwise, permutation p: 0.241
llm pointwise, established at 0.01: no
llm pointwise, t-test p: 0.244
llm pointwise, t-test agrees: yes
llm pointwise, 99% interval: -0.007 to 0.026
llm pointwise, interval agrees: yes

錄製時每次查詢呼叫了 20 次,其中有 0 個回答沒有附上分數。分數相當粗略。中位數的查詢,其 20 項商品只得到 4 種不同的分數,而且所有商品中有 0.390 的分數為 100。在 30 個查詢中,重新排序後的前 10 名裡大多數商品與其他商品同分,因此決勝規則(也就是語意檢索的順序)決定了清單的大部分內容。

在這份提示與這些清單上,LLM 判定為 exact 的商品比人工判定的少:

標籤 LLM 人工
exact 251 310
partial 345 265
irrelevant 4 7
無人判定 – 18

在這些清單上確實如此,但第 4 章顯示,LLM 與人工意見分歧的方向會隨所判斷的配對與提示而改變,因此不要把這個結果套用到其他情境。

逐點重新排序同樣無法證明有所增益:逐點排序的 test ndcg@10 為 0.837,語意檢索為 0.830,排列檢定 p 值為 0.241(見決策表)。

LLM 的排序使前 10 名中出現更多未經判定的商品:有 5 個位置,而語意檢索為 3 個,judged@10 因而從 0.990 降到 0.983。

實用原則: 若以 LLM 進行逐點重新排序,請要求它給出分級標籤或分數,統計每個查詢所給出的不同分數數量,並事先決定決勝規則:沿用檢索的順序是安全的預設做法。評估結果時,請使用人工判定,切勿使用同一個 LLM 的評分。

用一個 LLM 提示詞排序 20 個商品

最近一次變動於第 2026.11.8 版

第三種方式是 listwise 提示詞:它替候選商品編號,並要求 LLM 依相關性高低回傳這些編號。RankGPT 將這種做法稱為排列生成(permutation generation)(Sun et al., 2023)。一個提示詞能容納的商品數量有限,因此較長的清單會以滑動視窗(sliding window)重新排序:RankGPT 對 BM25 的前 100 名重新排序,採用「視窗大小 20、步長 10」,由清單底部往上進行(第 3.2 節與第 6.1 節)。回答必須是有效的排序,但這可能失敗:在 2023 年的中等規模 LLM 上,例如 200 億參數的 Flan-UL2,listwise 提示詞幾乎總是失敗,原因包括排名缺漏、重複或遭到拒答(Qin et al., 2024)。

商店的 LLM 每個提示詞看到 20 個商品,也就是一個視窗。提示詞以一行簡短文字呈現每個商品:名稱、類別、屬性,以及商品描述的前 160 個字元。最後一項指示為:「Order all 20 products from the best match for the query to the worst. Return the product numbers, each exactly once, in JSON.」JSON 結構描述(schema)要求恰好 20 個數字,但並未要求這 20 個數字互不相同,因此錄製時會修復任何含有重複或缺漏編號的回答,並加以標記。我們將 語意檢索 的前 20 名,依 LLM 在以語意順序看到這些商品時給出的回答重新排序。這項重新排序是第四個預先規劃的比較,顯著性門檻為 p 值低於 0.01:

llm-listwise-rerank.py Python 3.12.15 · bm25s 0.3.16 · lightgbm 4.7.0 · numpy 2.5.5 · pandas 3.1.1 · pystemmer 3.1.0 · scikit-learn 1.9.1 · scipy 1.18.1 通過
"""Reads the listwise LLM's answers for the test queries, in both orders, and reranks semantic retrieval's top 20 by theanswer given in semantic order. The prose's point rests on the planned comparison's p-value, and on the repair count,which shows how many answers needed a fix."""from lib.metrics import evaluatefrom lib.ranking import DEPTH, report_plannedfrom lib.recorded import llm_listwise, manifestfrom lib.semantic import SemanticSearchfrom lib.store import load_queries, qrelsqueries, judgments = load_queries("test"), qrels("test")       # scored on the human judgments, not the LLM'ssemantic = SemanticSearch().run(queries, k=100)answers = llm_listwise()                                        # one call per query and order shown# the answers hold one row per product, so keep one row per call: one call per query and ordercalls = answers.drop_duplicates(["query_id", "order"])print(f"listwise calls, both orders: {len(calls)}")# valid is false where the recording repaired a repeated or missing numberprint(f"answers that needed repair: {(~calls['valid']).sum()}")print(f"products per call: {answers.groupby(['query_id', 'order']).size().max()}")print("prompt asks for:", manifest("llm-rerank")["prompts"]["listwise"].splitlines()[-1].replace("{n}", str(DEPTH)))shown_in_semantic_order = answers[answers["order"] == "semantic"].sort_values(["query_id", "position"])order = shown_in_semantic_order.groupby("query_id")["product_id"].apply(list)# the reranked top 20, then the rest of semantic retrieval's list in place: ranking[DEPTH:] starts at the 21st productreranked = {query_id: order[query_id] + ranking[DEPTH:] for query_id, ranking in semantic.items()}before, after = evaluate(semantic, judgments), evaluate(reranked, judgments)for name, frame in [("semantic", before), ("llm listwise", after)]:    print(f"{name}, test ndcg@10: {frame['ndcg@10'].mean():.3f}")    print(f"{name}, test judged@10: {frame['judged@10'].mean():.3f}")    print(f"{name}, unjudged places in the test top 10s: {round((1 - frame['judged@10']).sum() * 10)}")moved = sum(reranked[query_id][0] != semantic[query_id][0] for query_id in queries.index)print(f"queries whose first product changes: {moved}")report_planned("llm listwise", after["ndcg@10"], before["ndcg@10"])
輸出python llm-listwise-rerank.py
listwise calls, both orders: 60
answers that needed repair: 0
products per call: 20
prompt asks for: Order all 20 products from the best match for the query to the worst. Return the product numbers, each exactly once, in JSON.
semantic, test ndcg@10: 0.830
semantic, test judged@10: 0.990
semantic, unjudged places in the test top 10s: 3
llm listwise, test ndcg@10: 0.819
llm listwise, test judged@10: 0.983
llm listwise, unjudged places in the test top 10s: 5
queries whose first product changes: 19
llm listwise, mean difference: -0.010
llm listwise, wins, losses, ties: 12, 12, 6
llm listwise, permutation p: 0.580
llm listwise, established at 0.01: no
llm listwise, t-test p: 0.498
llm listwise, t-test agrees: yes
llm listwise, 99% interval: -0.070 to 0.016
llm listwise, interval agrees: yes

錄製共進行了 60 次 listwise 呼叫,每個測試查詢兩次,其中 0 個回答需要修復。重新排序使 19 個查詢的第一個商品改變。

listwise 重新排序的增益並未得到確立:它的得分為 0.819,語意檢索為 0.830,排列檢定 p 值為 0.580(全部五項比較)。

Judged@10 在此也下降,從 0.990 降至 0.983,未判定的位置有 5 個,而語意檢索為 3 個。

**實用原則:**使用每個 listwise 回答之前都要先檢查:計算缺漏、重複與未知的編號,並記錄有多少回答需要修復。若執行環境支援,請以結構描述限制回答的形式。在測量之前,先固定視窗、步長,以及呈現商品的順序,因為順序會改變回答,如下一項檢查所示。

此部分尚未翻譯,因此以英文呈現。

Reverse the input to measure positional bias

最近一次變動於第 2026.11.10 版

A listwise answer can suffer from positional bias: an LLM's tendency to let a product's place in the prompt affect its rank, not the click position bias of shoppers. "Large language models (LLMs) exhibit positional bias in how they use context, which especially complicates listwise ranking" (Tang et al., 2024). On TREC DL19, the 2019 queries of TREC's Deep Learning track, a passage-ranking benchmark, gpt-3.5-turbo reranked BM25's top 100 to an nDCG@10 of 65.80 when shown BM25's order, 32.77 when shown it reversed and 25.17 when shown a random order (Sun et al., 2023, Table 5). The direct check is to show the same products in two orders and compare the answers. Kendall's tau measures how alike two orderings are, from 1 for the same order to −1 for the reverse, by counting the pairs they put the same way round (chapter 4).

The recording sent each test query's listwise prompt to the same LLM twice. The first prompt showed semantic retrieval's top 20 in semantic order, as the listwise rerank used; the second showed them reversed. Out of the 30 test queries, how often will the LLM put the same product first both times? Write your number down.

listwise-position-bias.py Python 3.12.15 · bm25s 0.3.16 · lightgbm 4.7.0 · numpy 2.5.5 · pandas 3.1.1 · pystemmer 3.1.0 · scikit-learn 1.9.1 · scipy 1.18.1 通過
"""Shows the listwise LLM the same test products in two orders, and measures how far its answers agree and how far eachanswer follows the order it was shown. The prose's point rests on the mean tau between the two answers, and on the twotaus between each answer and its own input order."""import numpy as npfrom scipy.stats import kendalltaufrom lib.metrics import evaluatefrom lib.ranking import DEPTHfrom lib.recorded import llm_listwisefrom lib.semantic import SemanticSearchfrom lib.store import load_queries, qrelsqueries, judgments = load_queries("test"), qrels("test")semantic = SemanticSearch().run(queries, k=DEPTH)               # the order the first prompt showedanswers = llm_listwise().sort_values(["query_id", "order", "position"])# the products in the order the LLM returned them, for each query and each order shownanswer = {key: list(group["product_id"]) for key, group in answers.groupby(["query_id", "order"])}def tau(first: list, second: list) -> float:    """Kendall's tau between two orderings of the same products."""    place = {product_id: rank for rank, product_id in enumerate(second)}    # each product's place in the second ordering    # the first ordering's positions, against where the second ordering puts each product    return kendalltau(range(len(first)), [place[product_id] for product_id in first]).statisticbetween, shown_semantic, shown_reversed, overlap, same_first = [], [], [], [], 0for query_id, shown in semantic.items():    forward, backward = answer[(query_id, "semantic")], answer[(query_id, "reversed")]    between.append(tau(forward, backward))    shown_semantic.append(tau(forward, shown))    shown_reversed.append(tau(backward, shown[::-1]))     # the reversed input, as the second prompt showed it    overlap.append(len(set(forward[:10]) & set(backward[:10])))    same_first += forward[0] == backward[0]               # index 0 is the first placeprint(f"test queries: {len(semantic)}")print(f"tau between the two answers, mean: {np.mean(between):.3f}")print(f"tau between the two answers, lowest: {min(between):.3f}\ntau between the two answers, highest: {max(between):.3f}")print(f"top-10 products in common, mean: {np.mean(overlap):.3f}")print(f"top-10 products in common, fewest: {min(overlap)}")print(f"queries with the same first product: {same_first}")print(f"share of queries with the same first product: {same_first / len(semantic):.3f}")print(f"tau between the order shown and the answer, semantic order: {np.mean(shown_semantic):.3f}")print(f"tau between the order shown and the answer, reversed order: {np.mean(shown_reversed):.3f}")# an answer that ignored the order would repeat itself, so its tau with the reversed input is minus its tau with the semantic orderprint(f"the same, if the order shown made no difference: {-np.mean(shown_semantic):.3f}")reversed_run = {query_id: answer[(query_id, "reversed")] for query_id in semantic}print(f"reversed answer, test ndcg@10, not a planned comparison: {evaluate(reversed_run, judgments)['ndcg@10'].mean():.3f}")
輸出python listwise-position-bias.py
test queries: 30
tau between the two answers, mean: 0.311
tau between the two answers, lowest: -0.484
tau between the two answers, highest: 0.811
top-10 products in common, mean: 6.967
top-10 products in common, fewest: 3
queries with the same first product: 13
share of queries with the same first product: 0.433
tau between the order shown and the answer, semantic order: 0.414
tau between the order shown and the answer, reversed order: 0.045
the same, if the order shown made no difference: -0.414
reversed answer, test ndcg@10, not a planned comparison: 0.826

The same product comes first for 13 of the 30 queries, so the expectation that a capable model finds the same best match whichever way the list comes holds for only a share of 0.433 of the queries.

The two answers agree with a mean tau of 0.311, and per query the tau runs from -0.484 to 0.811. Their top 10s share 6.967 products on average, as few as 3.

The answers also lean toward the order shown. Each answer's tau against its own input order is 0.414 for the semantic order and 0.045 for the reversed one. An LLM that ignored the order would give the same answer both times, so its reversed answer would have a tau of -0.414 against the reversed input. The reversed answer sits far nearer its own input order than that: positional bias, for this model, prompt and these queries. The reversed answer scores 0.826 nDCG@10; it is not a planned comparison, so the chapter doesn't test it.

Three remedies exist. Pairwise ranking prompting asks the LLM about each pair twice, once in each order, and counts a disagreement as a tie (Qin et al., 2024). Permutation self-consistency reranks several shuffled orders and aggregates the answers, which raised scores by up to 7 to 18% for GPT-3.5 and 8 to 16% for LLaMA v2 (70B) (Tang et al., 2024). And rerankers fine-tuned for listwise ranking, such as the open RankZephyr, are reported to be "resilient against variations in initial document ordering" (Pradeep et al., 2023); the store didn't record one.

Working rule: Before you trust a listwise LLM reranker, run each query in at least two orders and report how much the answers agree. When agreement is low, aggregate several orders or ask each pair in both orders, and count those calls against your budget. In production, show the products in retrieval's order every time, so that the same list at least gets the same answer.

依預算設定重排深度

最近一次變動於第 2026.11.8 版

搜尋管線分成多個階段,每個階段都比前一階段更慢,分到的候選項目也更少,而本章的每一種重排器都會多加一個階段。多階段排序器提供「透過控制候選項目進入各管線階段的條件,在品質與延遲之間取捨的能力」(Nogueira et al., 2019)。因此,重排深度,也就是重排器每次查詢要讀取的商品數量,是一項預算決策。計時結果會隨硬體與執行時期的選擇而變動(Peng et al., 2025),所以本節改為計算影響計時的因素:每次查詢的模型呼叫次數與所讀取的文字量,再乘上模型的規模。商店使用的 cross-encoder 有 149M 個參數(模型卡),其 LLM 則約有 120 億個參數(模型卡),約為前者的 80 倍。

在深度為 20 時,依商品文字與已記錄的提示詞,計算各重排器對每個測試查詢的模型呼叫次數與讀取的字元數,另外加上兩種商店未記錄的設計所需的呼叫次數:

reranker-prompt-sizes.py Python 3.12.15 · bm25s 0.3.16 · lightgbm 4.7.0 · numpy 2.5.5 · pandas 3.1.1 · pystemmer 3.1.0 · scikit-learn 1.9.1 · scipy 1.18.1 通過
"""Counts the characters each reranker reads per test query at depth 20, from the products' text and the recorded prompts,and the model calls each design makes. The prose's point rests on the characters and calls lines, and on the two designsthe store didn't record."""import numpy as npfrom lib.agreement import as_the_llm_sawfrom lib.ranking import DEPTH, listwise_linefrom lib.recorded import manifestfrom lib.semantic import SemanticSearchfrom lib.store import load_products, load_queries, product_textproducts, queries = load_products(), load_queries("test")semantic = SemanticSearch().run(queries, k=DEPTH)                # the 20 products each reranker readsprompts, text = manifest("llm-rerank")["prompts"], product_text(products)cross_encoder, pointwise, listwise = [], [], []                  # characters read per queryfor query_id, query in queries["query"].items():    shown = products.loc[semantic[query_id]]    # one call per product, each reading the query and that product's text together    cross_encoder.append(sum(len(query) + len(text[product_id]) for product_id in semantic[query_id]))    # the pointwise prompt, filled in with the query and the product as the LLM saw it    pointwise.append(sum(len(prompts["pointwise"].format(query=query, product=as_the_llm_saw(product)))                         for _, product in shown.iterrows()))    # products numbered from 1, as the listwise prompt shows them    lines = "\n".join(f"[{number}] {listwise_line(product)}" for number, (_, product) in enumerate(shown.iterrows(), 1))    listwise.append(len(prompts["listwise"].format(query=query, products=lines, n=DEPTH)))# the cross-encoder and the pointwise LLM make one call per product; the listwise LLM makes one call per queryfor name, calls, characters in [("cross-encoder", DEPTH, cross_encoder), ("llm pointwise", DEPTH, pointwise),                                ("llm listwise", 1, listwise)]:    print(f"{name}, model calls per query: {calls}")    print(f"{name}, characters read per query, mean: {np.mean(characters):,.0f}")print(f"llm pairwise, every pair in both orders, calls per query: {DEPTH * (DEPTH - 1)}")# a sliding window of 20 products, moved 10 at a time: its starts are list positions 0, 10, ..., 80window, step, depth = 20, 10, 100                                # a sliding window over a deeper listprint(f"llm listwise over the top {depth}, window {window}, step {step}, calls per query: {(depth - window) // step + 1}")
輸出python reranker-prompt-sizes.py
cross-encoder, model calls per query: 20
cross-encoder, characters read per query, mean: 9,911
llm pointwise, model calls per query: 20
llm pointwise, characters read per query, mean: 17,864
llm listwise, model calls per query: 1
llm listwise, characters read per query, mean: 5,973
llm pairwise, every pair in both orders, calls per query: 380
llm listwise over the top 100, window 20, step 10, calls per query: 9
重排器,前 20 名 每次查詢的模型呼叫次數 每次查詢讀取的字元數
Cross-encoder 20 9,911
LLM,pointwise 20 17,864
LLM,listwise 1 5,973

Pointwise 會重複指示內容,並完整呈現每項商品,因此讀取的文字最多,且是在彼此獨立的呼叫中進行。Listwise 讀取的文字最少,只需一次呼叫,但必須依序寫出 20 個編號。若以 pairwise 提示對所有配對進行雙向比較,每次查詢需要 380 次呼叫;若以 RankGPT 的滑動視窗處理前 100 名,則需要 9 次。在一項已發表的量測中,於 TREC DL19 上以單張 RTX A6000 GPU,使用 Flan-T5-XL 重排 BM25 的前 100 名,pointwise 的是非判斷提示在每次查詢 1.5 秒下得到 0.650 的 nDCG@10,而對所有配對進行的 pairwise 提示則在 254.9 秒下得到 0.713,並使用了 9,900 次 LLM 呼叫(Zhuang et al., 2024b)。以 cross-encoder 作為特徵的 LambdaMART 需要為每個候選項目計算 cross-encoder 分數,每次查詢的配對數中位數為 131。

對 cross-encoder 而言,成本會隨深度同步增加,所以問題在於品質在何處停止成長。在訓練查詢上掃描不同的深度:

rerank-depth-on-train.py Python 3.12.15 · bm25s 0.3.16 · lightgbm 4.7.0 · numpy 2.5.5 · pandas 3.1.1 · pystemmer 3.1.0 · scikit-learn 1.9.1 · scipy 1.18.1 通過
"""Reranks semantic retrieval's top 5 to 75 with the recorded cross-encoder scores, on the train queries, and prints nDCG@10at each depth. The prose's point rests on the change from the depth before, which shows where the gain flattens and thenfalls."""from lib.metrics import evaluatefrom lib.ranking import rerankfrom lib.recorded import cross_encoder_scoresfrom lib.semantic import SemanticSearchfrom lib.store import load_queries, qrelsqueries, judgments = load_queries("train"), qrels("train")      # choosing a depth is tuning: train queriessemantic = SemanticSearch().run(queries, k=100)recorded = cross_encoder_scores()# the recorded cross-encoder score of each (query, product) pairscore = dict(zip(zip(recorded["query_id"], recorded["product_id"]), recorded["score"]))train_pairs = recorded[recorded["query_id"].isin(queries.index)]print(f"depth 0 (semantic alone), train ndcg@10: {evaluate(semantic, judgments)['ndcg@10'].mean():.3f}")previous = round(evaluate(semantic, judgments)["ndcg@10"].mean(), 3)     # changes between the printed valuesfor depth in [5, 10, 20, 30, 50, 75]:    mean = round(evaluate(rerank(semantic, score, depth), judgments)["ndcg@10"].mean(), 3)    print(f"depth {depth}, train ndcg@10: {mean:.3f}")    print(f"depth {depth}, change from the depth before: {mean - previous:+.3f}")    previous = mean# every candidate needs its own cross-encoder score, so this counts the recorded pairs per queryprint(f"pairs per query to score every candidate, median: {train_pairs.groupby('query_id').size().median():.0f}")
輸出python rerank-depth-on-train.py
depth 0 (semantic alone), train ndcg@10: 0.772
depth 5, train ndcg@10: 0.782
depth 5, change from the depth before: +0.010
depth 10, train ndcg@10: 0.784
depth 10, change from the depth before: +0.002
depth 20, train ndcg@10: 0.802
depth 20, change from the depth before: +0.018
depth 30, train ndcg@10: 0.804
depth 30, change from the depth before: +0.002
depth 50, train ndcg@10: 0.793
depth 50, change from the depth before: -0.011
depth 75, train ndcg@10: 0.788
depth 75, change from the depth before: -0.005
pairs per query to score every candidate, median: 131

Cross-encoder 的增益在 20 到 30 之間趨於平緩,接近為了配合 LLM 紀錄而固定的深度。從語意檢索的 0.772 起算,重排前 5 名得到 0.782,重排前 20 名得到 0.802。前 30 名增加 +0.002,代價是再多出一半的配對數。更深時反而下降:50 時為 0.793,75 時為 0.788。

在測試集上,深度為 20 的重排並未證明有任何增益,因此對這家商店而言,成本最低的重排器,也就是不使用重排器,同時也是應該保留的選擇。

實用原則: 對你考慮的每一種重排器,計算各深度下每次查詢的呼叫次數與字元數,並在訓練查詢上設定深度。這是經驗法則,而非量測結果:重排的深度只要到訓練集 nDCG@10 不再提升的深度即可,因為超過該深度的每項商品都要付出相同的成本,卻沒有任何助益。接著在你自己的硬體與流量下為該組態計時,只有在符合延遲預算時才保留這個階段。

將交叉編碼器蒸餾成小型排序器

最近一次變動於第 2026.11.10 版

對預算而言太慢的重排序器,可以用來教導一個快速的排序器。蒸餾(distillation)是訓練一個小模型,即學生模型(student),去模仿一個大模型,即教師模型(teacher)。Hinton 等人將它描述為把整個模型集成的知識蒸餾到單一模型中(Hinton et al., 2015)。在排序任務中,教師模型的分數會成為學生模型的標籤。當兩個模型的分數尺度不同時(來自不同架構的分數通常如此),學生模型可以改為學習兩個商品分數之間的差距(margin),所用的損失函數稱為 Margin-MSE(Hofstätter et al., 2020)。蒸餾也可以從 LLM 向下進行。RankGPT 報告指出,一個經蒸餾的 440M 模型在 BEIR 基準上勝過一個 3B 的監督式模型(Sun et al., 2023;BEIR)。在一項針對含最高級用語查詢的商品搜尋研究中,輕量模型承接了 LLM 的詮釋,而直接以 LLM 重排序的延遲高得無法接受(Zhu et al., 2025)。教師模型不需要人工標籤,並且會為每個候選項目標註。

教師模型是交叉編碼器。學生模型是一個 LightGBM 迴歸模型,使用7個特徵,這些特徵除了檢索本來就要做的那一次之外,不需要任何額外的模型呼叫:也就是除了cross_encoder以外的所有特徵。它的標籤是教師模型的原始分數,樹迴歸可以在教師模型自己的尺度上與之吻合。它對相同的候選項目排序,依product_id的順序,樹的規模則沿用為 LambdaMART 選定的規模。首先,以訓練查詢做交叉驗證,將學生模型與以人工評分和相同特徵訓練的 LambdaMART 模型相比較:

distillation-on-train.py Python 3.12.15 · bm25s 0.3.16 · lightgbm 4.7.0 · numpy 2.5.5 · pandas 3.1.1 · pystemmer 3.1.0 · scikit-learn 1.9.1 · scipy 1.18.1 通過
"""Trains a student LightGBM regression on the cross-encoder's scores, and a LambdaMART model on human grades with the samefeatures, both by cross-validation on the train queries. The prose's point rests on the student's score, and on itsdifference from the human-graded ranker."""from lightgbm import LGBMRanker, LGBMRegressorfrom lib.metrics import evaluatefrom lib.ranking import CHEAP_FEATURES, DETERMINISTIC, LAMBDARANK, TREES, cross_validate, feature_table, rankedfrom lib.semantic import SemanticSearchfrom lib.significance import comparefrom lib.store import load_products, load_queries, qrelsqueries, judgments = load_queries("train"), qrels("train")          # design work: train queries onlycandidates = feature_table(load_products(), queries)# unjudged candidates count as grade 0, as in the metriccandidates["grade or 0"] = candidates["grade"].fillna(0).astype(int)student = cross_validate(lambda: LGBMRegressor(**TREES, **DETERMINISTIC), candidates, candidates,                         "cross_encoder", CHEAP_FEATURES, judgments)          # labels from the teacherhuman = cross_validate(lambda: LGBMRanker(**LAMBDARANK, **TREES, **DETERMINISTIC), candidates, candidates,                       "grade or 0", CHEAP_FEATURES, judgments)               # labels from people# the teacher's own order of the candidates, ties broken by product_idteacher = evaluate(ranked(candidates, candidates["cross_encoder"]), judgments)["ndcg@10"]semantic = evaluate(SemanticSearch().run(queries, k=100), judgments)["ndcg@10"]for name, scores in [("semantic", semantic), ("teacher", teacher), ("student, teacher's labels", student),                     ("lambdarank, human grades, same features", human)]:    print(f"{name}, train ndcg@10: {scores.mean():.3f}")# the student's score minus the human-graded ranker's, per query, tested by permutationresult = compare(student, human)print(f"student minus the human-graded ranker: {result['mean difference']:.3f}")print(f"student minus the human-graded ranker, permutation p: {result['permutation p']:.3f}")
輸出python distillation-on-train.py
semantic, train ndcg@10: 0.772
teacher, train ndcg@10: 0.785
student, teacher's labels, train ndcg@10: 0.791
lambdarank, human grades, same features, train ndcg@10: 0.747
student minus the human-graded ranker: 0.044
student minus the human-graded ranker, permutation p: 0.002

在這些訓練查詢與這些特徵上,教師模型對每個候選項目的標籤,訓練出的排序器比人工評分更好,其中未經評判的候選項目以 0 計。學生模型的分數是0.791,而教師模型自己對候選項目的排序則是0.785。

以人工評分訓練的模型得分為0.747,低於語意檢索的0.772;學生模型減去該模型的差為0.044,排列檢定的 p 值為0.002。這是在訓練查詢上的調校檢查,不能當作證據。

接著在訓練查詢上擬合學生模型,並以與 LambdaMART 相同的保留訓練查詢做提前停止,然後執行最後一項預先規劃的比較,門檻相同,為0.01:

distilled-ranker-on-test.py Python 3.12.15 · bm25s 0.3.16 · lightgbm 4.7.0 · numpy 2.5.5 · pandas 3.1.1 · pystemmer 3.1.0 · scikit-learn 1.9.1 · scipy 1.18.1 通過
"""Fits a student LightGBM regression on the teacher's scores for the train queries, then reports its gain shares, howclosely its order follows similarity and the teacher, and its planned comparison on the test queries. The prose's pointrests on that comparison and on the taus printed for each ranking."""import numpy as npfrom lightgbm import LGBMRegressorfrom scipy.stats import kendalltaufrom lib.metrics import evaluatefrom lib.ranking import CHEAP_FEATURES, DETERMINISTIC, TREES, feature_table, fit, ranked, report_planned, stopping_splitfrom lib.semantic import SemanticSearchfrom lib.store import load_products, load_queries, qrelsproducts, train, test = load_products(), load_queries("train"), load_queries("test")table = feature_table(products, train)                       # the teacher's score is the label: no human gradesprint(f"train candidates the teacher labels: {len(table)}")print(f"of them, judged by nobody: {table['grade'].isna().sum()}")print(f"student features: {len(CHEAP_FEATURES)}")# early stopping on the same 10 held-out train queries that LambdaMART stops onstudent = fit(LGBMRegressor(**TREES, **DETERMINISTIC), table, "cross_encoder", CHEAP_FEATURES, *stopping_split(train.index))print(f"trees kept by early stopping: {student.best_iteration_}")candidates, judgments = feature_table(products, test), qrels("test")       # test queries: read once, to reportpredicted = student.predict(candidates[CHEAP_FEATURES])gain = student.booster_.feature_importance(importance_type="gain")# the three features with the most gainfor name, share in sorted(zip(CHEAP_FEATURES, gain / gain.sum()), key=lambda pair: -pair[1])[:3]:    print(f"student's share of gain, {name}: {share:.3f}")for name in ["cross_encoder", "similarity"]:                   # whose order does the student follow?    # candidates has a default index, so group.index picks this query's rows out of predicted    tau = [kendalltau(group[name], predicted[group.index]).statistic for _, group in candidates.groupby("query_id")]    print(f"tau between student and {name}, mean over test queries: {np.mean(tau):.3f}")teacher = evaluate(ranked(candidates, candidates["cross_encoder"]), judgments)print(f"teacher's own order of the candidates, test ndcg@10, not a planned comparison: {teacher['ndcg@10'].mean():.3f}")after = evaluate(ranked(candidates, predicted), judgments)before = evaluate(SemanticSearch().run(test, k=100), judgments)for name, frame in [("semantic", before), ("distilled", after)]:    print(f"{name}, test ndcg@10: {frame['ndcg@10'].mean():.3f}")    print(f"{name}, test judged@10: {frame['judged@10'].mean():.3f}")    # each query's top 10 is 10 places, so (1 - judged@10) times 10 counts its unjudged places    print(f"{name}, unjudged places in the test top 10s: {round((1 - frame['judged@10']).sum() * 10)}")report_planned("distilled", after["ndcg@10"], before["ndcg@10"])
輸出python distilled-ranker-on-test.py
train candidates the teacher labels: 6896
of them, judged by nobody: 1975
student features: 7
trees kept by early stopping: 202
student's share of gain, similarity: 0.812
student's share of gain, name_match: 0.126
student's share of gain, bm25: 0.055
tau between student and cross_encoder, mean over test queries: 0.466
tau between student and similarity, mean over test queries: 0.753
teacher's own order of the candidates, test ndcg@10, not a planned comparison: 0.782
semantic, test ndcg@10: 0.830
semantic, test judged@10: 0.990
semantic, unjudged places in the test top 10s: 3
distilled, test ndcg@10: 0.826
distilled, test judged@10: 0.987
distilled, unjudged places in the test top 10s: 4
distilled, mean difference: -0.004
distilled, wins, losses, ties: 10, 13, 7
distilled, permutation p: 0.720
distilled, established at 0.01: no
distilled, t-test p: 0.694
distilled, t-test agrees: yes
distilled, 99% interval: -0.034 to 0.017
distilled, interval agrees: yes

教師模型為全部6896個訓練候選項目標註,其中1975個無人評判,而提前停止保留了202棵樹。

學生模型倚賴similarity,它佔增益的0.812,領先佔0.126的name_match,以及佔0.055的bm25。在測試候選項目上,它的分數與語意相似度一致,每個查詢的平均 tau 為0.753,與教師模型的一致程度為0.466。因此,學生模型學到的主要是語意檢索的順序,再受名稱與 BM25 匹配的影響而向教師模型的順序微調。

學生模型並未證實有所增益:蒸餾排序器的 nDCG@10 為0.826,語意檢索為0.830,排列檢定的 p 值為0.720(見決策表)。教師模型自身對測試候選項目的排序,本章原本沒有計畫檢定,其得分為0.782。

Judged@10方面,學生模型為0.987,語意檢索為0.990;學生模型的前 10 名中有4個未經評判的名次,語意檢索則有3個。

實用原則: 當重排序器已有確立的增益,但每次查詢的成本太高時,就進行蒸餾:以教師模型為訓練查詢的每個候選項目標註,用不需要教師模型即可計算的特徵訓練學生模型,並在保留查詢上以你的檢索為對照檢定學生模型。查看學生模型的增益重要性:當檢索本身的分數佔主導時,學生模型大致就是你的檢索。當學生模型無法吻合教師模型的尺度時,改用成對之間的差距。商店目前還沒有勝過語意檢索的教師模型,因此保留程式碼,不部署學生模型。

正式環境中會出什麼問題

最近一次變動於第 2026.11.8 版

線上排序器會在購物者等待時計算特徵並呼叫模型,而它所用的程式碼與模型版本,可能與訓練和測試時所用的版本不一致。

特徵會在訓練與服務之間漂移。 如果線上系統的斷詞方式不同、對 BM25 重新縮放,或升級了嵌入模型,已學得的權重面對的就是不同的數值,而且不會有任何錯誤訊息。請用同一條共用的程式碼路徑計算特徵,並將特徵與每一次提供的排序一併記錄下來。

資料列順序會洩漏到訓練中。 LightGBM 的 lambdarank 從輸入資料列的順序開始處理,而任何排序都得以某種方式處理同分。依某個重排序器分數排序的檔案,等於把該重排序器的排序交給了模型。在每次擬合與每次評分之前,請依 ID 排序候選項目,或以固定的種子將它們隨機打亂。

判斷來自一個資料池。 WANDS 只判斷了它自己的系統所提出的商品(第 2 章)。若將未經判斷的候選項目視為不相關,模型就會學著降低那些系統沒有找到的商品的排名。請為你加入的每個排序器檢查 judged@10,並在相信任何增益之前,先讓其中未經判斷的商品完成判斷。

LLM 評審偏好 LLM 重排序器。 在一項研究中,以 LLM 評估者的判斷建立的排名,平均將以 LLM 為基礎的重排序器高估了 9 到 17 個名次(Fröbe et al., 2025),而同時以 LLM 進行排序與判斷會形成循環,「說明了依賴以 LLM 為基礎的評估,可能扭曲系統排名」(Clarke and Dietz, 2024)。請如本章所做,以人工判斷為 LLM 重排序器評分。

LLM 重排序器屬於單一模型版本與單一提示詞。 不同的摘要值(用來識別確切權重的雜湊)、量化方式(權重儲存的精度)或提示詞用字,都可能改變每一個答案;再加上位置偏誤,檢索順序的變動也可能如此。LLM 錄製檔的清單檔會記錄摘要值與兩個提示詞的原文;只要其中任一項改變,就要重新量測。

不熟悉的查詢可能表現較差。 一項比較 22 種重排序方法的研究發現,LLM 重排序器在熟悉的查詢上表現較好,而「它們對新穎查詢的泛化能力則不一」(Abdallah et al., 2025)。請留意與商品和風格有關、且比模型訓練資料更新的查詢。

每個重排序器都會在購物者的路徑上增加一次呼叫。 如同預算所示,請計算每個查詢的呼叫次數與文字量、設定逾時,並在重排序器未能及時回應時,退回使用檢索的順序。

實用原則: 隨每次排序記錄特徵與模型版本,將候選項目放入不帶任何資訊的順序,依人工判斷評分,檢查 judged@10,並為每個重排序階段設定時間上限與後備方案。

此部分尚未翻譯,因此以英文呈現。

Decisions this chapter made

最近一次變動於第 2026.11.8 版

The chapter made each choice in the list below on the train queries, and only an established planned comparison on the test queries could change the store's search.

The 5 planned comparisons, each on nDCG@10 against semantic retrieval's 0.830, established below 0.01:

Measure LambdaMART Cross-encoder LLM pointwise LLM listwise Distilled
nDCG@10 0.817 0.813 0.837 0.819 0.826
Mean difference -0.012 -0.016 0.008 -0.010 -0.004
Wins / losses / ties 14, 12, 4 11, 12, 7 11, 8, 11 12, 12, 6 10, 13, 7
Permutation p 0.260 0.178 0.241 0.580 0.720
Established no no no no no
t-test p 0.278 0.171 0.244 0.498 0.694
99% interval -0.044 to 0.011 -0.058 to 0.006 -0.007 to 0.026 -0.070 to 0.016 -0.034 to 0.017
t-test agrees yes yes yes yes yes
Interval agrees yes yes yes yes yes
  • 5 planned comparisons against semantic retrieval, on human judgments, established below 0.01.
  • Candidates: the union of BM25's and semantic retrieval's top 100, in product_id order, with score ties broken by product_id.
  • LambdaMART: label_gain=[0, 1, 2], unjudged candidates as grade 0, 3 leaves at a learning rate of 0.1, chosen by cross-validation on train, with early stopping on held-out train queries through eval_X and eval_y. Its gain isn't established on test, so it isn't used.
  • Rerank depth: semantic retrieval's top 20, fixed to match the LLM recording; on train, the cross-encoder's curve flattens between 20 and 30. No reranker's gain is established on test, so none is used.
  • LLM rerankers are scored on human judgments only, and the listwise one is checked in two input orders. Its answer depends on the order shown.
  • Costs are counted, not timed: calls and characters per query.
  • Distillation: a LightGBM student on the cross-encoder's scores, with no model call beyond retrieval's. It learns mostly semantic retrieval's order, shows no established gain, and isn't used.
  • The ranker to beat stays semantic retrieval alone, with BM25, unstemmed, as the baseline.
  • What the chapter showed: no second stage earned its place; none shows an established gain.

本章,於本版次中

版次 2026.11.10範例於 2026年11 月 19 通過測試本章 14 個範例 個範例中有 14 個通過

譯自英文書,版次 2026.11.10。

引用本節

格式

各種格式皆引用英文原書,本譯本即依循該原書。

每種格式都會標示本版。其連結會開啟目前版次,節標題會說明該節內文最後一次變動的時間。存取日期與擷取日期即為你開啟此頁的當日。

本頁內容

第 9 章,共 11 章: Ranking

  1. 用特徵描述每個候選商品
  2. Learn a ranking from pairs of products
  3. 以 LightGBM 訓練 LambdaMART
  4. Report LambdaMART on the test queries
  5. 以 cross-encoder 重新排序語意檢索結果
  6. 用 LLM 為每項商品評分
  7. 用一個 LLM 提示詞排序 20 個商品
  8. Reverse the input to measure positional bias
  9. 依預算設定重排深度
  10. 將交叉編碼器蒸餾成小型排序器
  11. 正式環境中會出什麼問題
  12. Decisions this chapter made

返回頂端所有章節