Ranking
読了まで約80 分分、12 節 節の 14 例 例。
このページの内容
- 各候補を特徴量で記述する
- Learn a ranking from pairs of products
- LightGBM で LambdaMART を学習する
- Report LambdaMART on the test queries
- クロスエンコーダでセマンティック検索を再ランキングする
- Score each product with an LLM
- 1回のLLMプロンプトで20件の商品を並べ替える
- Reverse the input to measure positional bias
- 予算からリランク深度を決める
- クロスエンコーダを小さなランカーに蒸留する
- 本番環境で何が問題になるか
- Decisions this chapter made
この部分はまだ翻訳されていないため、英語で表示している。
By the end of this chapter you can describe each product that retrieval returns with features, train a learning-to-rank model with LightGBM on your own judgments, rerank the top of the list with a cross-encoder or an LLM, measure an LLM reranker's positional bias, set the rerank depth by counting its costs, and distil a slow reranker into a fast one. You can also decide, with a planned comparison on the test queries, whether any of them beats your retrieval.
Semantic retrieval alone is still the ranker to beat at the store, where every search is ordered by one embedding score and nothing else. It ranks products by embeddings from Snowflake/snowflake-arctic-embed-m-v1.5, and scores 0.830 nDCG@10 on the 30 test queries. No change in chapters 5 to 8 has beaten it by an established margin. This chapter asks whether a second stage can: a model trained on the store's own judgments, a reranker that reads the query and the product together, or an LLM. Chapter 1's cross-encoder reranked the hybrid run, not semantic retrieval, and never saw the store's judgments.
So this chapter tries all three on semantic retrieval's top results, each scored against the store's judgments. Reranking reorders the top of retrieval's list with a slower model. The store's rerankers are recorded: each ran once and saved its outputs, which the examples read.
The store's LLM is the open model gemma4:12b-it-q4_K_M, licensed Apache-2.0. It ran locally with chapter 4's settings, which make it take its likeliest next word every time, plus thinking off, so it answers without writing out reasoning, and a JSON schema, a machine-readable description of the answer's shape. recordings/llm_rerank.py reranked semantic retrieval's top 20 for the test queries only, so nothing about the LLM rerankers can be tuned. Every score uses the human judgments, because in one study LLM judges ranked LLM rerankers too high (What goes wrong in production). These results describe one model and two prompts, so treat them as the store's, not as a verdict on LLM rerankers.
The chapter fixed its 5 planned comparisons on the test queries before it read any test result. Each sets one method against semantic retrieval's run on nDCG@10:
- LambdaMART, a learned ranker trained on the train queries' judgments, ordering each query's candidate products
- the cross-encoder reranking semantic retrieval's top 20
- the LLM scoring each of the top 20 on its own
- the LLM ordering the top 20 in one prompt
- a small distilled ranker trained on the cross-encoder's scores
The bar is the book's rule: 5 planned comparisons split its 0.05, so each needs a paired permutation test p-value below 0.01. The paired t-test and a 99% bootstrap interval stand beside it.
各候補を特徴量で記述する
本章で計画した比較の最初のものは、学習済みランカーである。学習済みランカーはテキストを読まない。クエリと商品の各組を、特徴量と呼ぶ数値の並びから採点し、各特徴量をどう重み付けるかを判定から学習する。クエリ依存の特徴量は、クエリと商品の両方を記述するもので、BM25 スコアがその例である。クエリ非依存の特徴量は、商品だけを記述するもので、何人の買い物客が評価したかがその例である。ウェブ検索の特徴量セットは両方を含む。ランキング学習のベンチマークである LETOR 4.0 は 46 個の特徴量を持ち、その中には「タイトルの BM25」「PageRank」「URL 中のスラッシュの数」がある(Qin and Liu, 2013)。ランカーが採点するのは候補、すなわちより安価な検索段階が返す商品だけであるため、特徴量の計算コストは高くてもよい。
ストアの各クエリに対する候補は、クロスエンコーダの記録が採点した商品であり、BM25 の上位 100 件と意味検索の上位 100 件の和集合である。lib.ranking.feature_table は各候補に 8 個の特徴量を与える。
bm25:lib.lexicalによる BM25 スコアで、本書のステミングなしのベースラインである。クエリと語を共有しない商品では 0 になるsimilarity:クエリの埋め込みと商品の埋め込みのコサイン類似度cross_encoder:記録済みのクロスエンコーダのスコアname_matchとattribute_match:第 5 章と同じ方法でトークン化したクエリの語のうち、商品名と商品の属性にそれぞれ見つかった語の割合rating_count、average_rating、review_count:ストアのデータが各商品に与える値
カタログの商品のうち 2632 には評価がなく、その 3 つの評価特徴量は欠損のままとなる。テーブルには各候補の人手による評点も入っており、誰も判定しなかった候補は空欄になる。学習用クエリについてテーブルを作り、1 つのクエリを見たうえで、評点ごとに各特徴量を比較する。
1"""Builds the feature table for the train queries, shows one query's candidates, and compares each feature's mean across2the human grades. The prose's point rests on the mean cross_encoder line for each grade and the mean of each feature by grade."""3import pandas as pd45from lib.ranking import FEATURES, feature_table6from lib.store import load_products, load_queries78products, queries = load_products(), load_queries("train") # train queries: features are design work9table = feature_table(products, queries)10print(f"features: {len(FEATURES)}")11print(f"train candidates: {len(table)}")12print(f"train candidates per query, median: {table.groupby('query_id').size().median():.0f}")13print(f"share of train candidates nobody judged: {table['grade'].isna().mean():.3f}")1415query_id = queries.index[queries["query"] == "laundry basket with wheels"][0]16one = table[table["query_id"] == query_id].sort_values("cross_encoder", ascending=False)17one = one.assign(name=products.loc[one["product_id"], "name"].str[:38].to_numpy())18pd.set_option("display.width", 200)19print(f"\ncandidates for laundry basket with wheels: {len(one)}")20print(one[["name", "bm25", "similarity", "cross_encoder", "name_match", "attribute_match", "rating_count", "grade"]]21 .head(6).round(3).to_string(index=False))22print(f"top 6 by cross-encoder, exact: {(one['grade'].head(6) == 2).sum()}")2324# grades 2, 1 and 0 are exact, partial and irrelevant; a missing grade means nobody judged the pair25label = table["grade"].map({2: "exact", 1: "partial", 0: "irrelevant"}).fillna("unjudged")26means = table[FEATURES].groupby(label).mean().loc[["exact", "partial", "irrelevant", "unjudged"]]27print("\nmean of each feature by grade, train candidates:")28print(means.round(3).to_string())29for grade in means.index:30 print(f"mean cross_encoder, {grade}: {means.at[grade, 'cross_encoder']:.3f}")31relevance = ["bm25", "similarity", "cross_encoder", "name_match"]32nearer = [name for name in relevance if abs(means.at["unjudged", name] - means.at["irrelevant", name])33 < abs(means.at["unjudged", name] - means.at["partial", name])]34print(f"relevance features where unjudged sits nearer irrelevant than partial: {len(nearer)} of {len(relevance)}")35medians = table[["rating_count", "review_count"]].groupby(label).median().loc[means.index] # counts are skewed36for grade in ["exact", "partial", "irrelevant"]:37 print(f"rating_count, {grade}, mean: {means.at[grade, 'rating_count']:.3f}")38 print(f"rating_count, {grade}, median: {medians.at[grade, 'rating_count']:.1f}")39print(f"products with no rating: {products['rating_count'].isna().sum()}")
python ranking-features-by-grade.pyfeatures: 8
train candidates: 6896
train candidates per query, median: 131
share of train candidates nobody judged: 0.286
candidates for laundry basket with wheels: 125
name bm25 similarity cross_encoder name_match attribute_match rating_count grade
laundry sorter basket on wheels 8.229 0.597 3.433 1.000 0.0 NaN 2.0
vintage laundry basket 6.270 0.572 2.995 0.667 0.0 638.0 2.0
rolling wicker laundry basket 5.494 0.588 2.974 0.667 0.0 110.0 2.0
aluminum frame laundry basket can be f 4.739 0.524 2.941 0.667 0.0 14.0 2.0
rebrilliant rolling laundry hamper 5.537 0.558 2.804 0.333 0.0 17.0 2.0
4-bag laundry hamper sorter cart baske 6.012 0.543 2.739 0.667 0.0 NaN 2.0
top 6 by cross-encoder, exact: 6
mean of each feature by grade, train candidates:
bm25 similarity cross_encoder name_match attribute_match rating_count average_rating review_count
grade
exact 4.317 0.517 2.191 0.562 0.024 466.879 4.537 333.589
partial 3.936 0.454 1.513 0.461 0.030 385.454 4.539 288.779
irrelevant 2.760 0.360 1.159 0.404 0.006 246.781 4.552 159.052
unjudged 2.323 0.369 1.050 0.140 0.018 339.904 4.520 237.027
mean cross_encoder, exact: 2.191
mean cross_encoder, partial: 1.513
mean cross_encoder, irrelevant: 1.159
mean cross_encoder, unjudged: 1.050
relevance features where unjudged sits nearer irrelevant than partial: 4 of 4
rating_count, exact, mean: 466.879
rating_count, exact, median: 43.0
rating_count, partial, mean: 385.454
rating_count, partial, median: 31.0
rating_count, irrelevant, mean: 246.781
rating_count, irrelevant, median: 35.5
products with no rating: 2632
学習用クエリの候補は 6896 件で、1 クエリあたりの中央値は 131 件であり、その 0.286 を誰も判定していない。
laundry basket with wheels には 125 件の候補がある。クロスエンコーダのスコア上位 6 件には、6 件の完全一致の商品が含まれる。それらの attribute_match は 0 である。属性が保持するのは色、素材、サイズであり、wheels のような語ではないからである。
学習用の候補全体で見ると、クロスエンコーダの平均スコアは、完全一致の商品の 2.191 から、部分一致の 1.513、無関係の 1.159 へと下がり、bm25、similarity、name_match も同じように下がる。average_rating と attribute_match はほとんど動かない。
rating_count の平均は評点とともに下がるが、中央値は下がらない。評価数の多い少数の商品が平均を決めているのである。
| 評点 | 評価数の平均 | 評価数の中央値 |
|---|---|---|
| 完全一致 | 466.879 | 43.0 |
| partial | 385.454 | 31.0 |
| irrelevant | 246.781 | 35.5 |
未判定の候補のクロスエンコーダの平均スコアは 1.050 であり、評点とともに下がる 4 of 4 個の特徴量では、その平均は部分一致の商品よりも無関係な商品の平均に近い。このことは、未判定の候補がモデルに何を教えるべきかを考えるうえで重要である。
実践ルール: まず、すでに計算している特徴量から始める。各検索器のスコア、リランカーがあればそのスコア、商品名のようなフィールドでの一致、人気を示すシグナルである。学習の前に、学習用クエリで各特徴量を評点別に比較する。スコアには平均を、偏りのある件数には中央値を使う。評点とともに動かない特徴量は、モデルに教えられることがほとんどない。特徴量は、学習時にも運用時にも同じコードで計算する。
この部分はまだ翻訳されていないため、英語で表示している。
Learn a ranking from pairs of products
Learning to rank trains a model on judged queries to order candidates by their features. Its methods fall into three families, pointwise, pairwise and listwise (Liu, 2009). A pointwise model predicts each product's grade on its own, as a regression. A pairwise model learns which of two products should rank higher. A listwise model scores a whole ranking. Liu's 2009 comparison on the LETOR benchmark favoured the listwise approach, though tentatively. The obstacle for all three is the metric: nDCG changes only when two products swap places, so ranking metrics "are everywhere either discontinuous or flat, so gradient descent appears to be problematic" (Burges, 2010).
RankNet, a pairwise method, works around it with a smooth loss on pairs. For two products where i should rank above j, its gradient is large when the model scores them the wrong way round or nearly level, and small when it already has them well apart. LambdaRank keeps that gradient and multiplies it by how much nDCG would change if the two products swapped places (Burges, 2010, section 4.1). Burges writes the gradient with a minus sign, as the slope of a cost; here a positive λ pushes i up:
λ_ij = σ / (1 + exp(σ · (s_i − s_j))) · |ΔnDCG_ij| s: the model's scores, σ = 1
ΔnDCG_ij = (g_i − g_j) · (1 / log2(1 + r_i) − 1 / log2(1 + r_j)) / ideal DCG g: grades, r: positions
A product's total is its λ summed over the pairs where it should rank higher, minus those where it should rank lower. Burges (2010) also reads the λ's as forces, pushing products up and down the list.
LambdaMART uses LambdaRank's λ as the gradients for gradient-boosted trees, which add small decision trees one at a time, each fitted to the gradients of the trees before it: "LambdaMART is the boosted tree version of LambdaRank, which is based on RankNet". An ensemble of LambdaMART rankers won Track 1 of the Learning to Rank Challenge that Yahoo ran in 2010 (Burges, 2010).
One train query, outdoor waterproof chest, shows the arithmetic. Let semantic similarity be the model's scores so far, compute every pair's λ in its top 10 by hand, check each |ΔnDCG| by swapping the pair and recomputing nDCG with lib.metrics, and add up the forces:
1"""Works out LambdaRank's λ for every pair in one train query's top 10 by hand, and checks each pair's |ΔnDCG| against2swapping the pair in the metric. The prose's point rests on the strongest push up and down, and on the count of pairs3whose formula matches swapping."""4import math5from itertools import combinations67from lib.metrics import dcg_at_k, ndcg_at_k8from lib.semantic import SemanticSearch9from lib.store import load_queries, qrels1011queries, judgments = load_queries("train"), qrels("train")12query_id = queries.index[queries["query"] == "outdoor waterproof chest"][0]13judged = judgments[query_id]14scores = SemanticSearch().search(query_id, k=10) # the model so far: semantic similarity15ranking, similarity = list(scores.index), scores.to_numpy()16# an unjudged product counts as grade 0, as it does in the metric17grades = [judged.get(product_id, 0) for product_id in ranking]18# the ideal DCG takes every judged product, as lib.metrics does; sigma is 1, LightGBM's default sigmoid19ideal, sigma = dcg_at_k(sorted(judged.values(), reverse=True), 10), 1.020print("grades from the top:", " ".join(map(str, grades)))2122force, checks, pulls = [0.0] * len(ranking), [], []23for i, j in combinations(range(len(ranking)), 2): # i is above j24 if grades[i] == grades[j]:25 continue26 better, worse = (i, j) if grades[i] > grades[j] else (j, i)27 ranknet = 1 / (1 + math.exp(sigma * (similarity[better] - similarity[worse]))) # RankNet's pull28 # i and j count from 0, so position p is i + 1 and its discount is 1 / log2(p + 1), which is 1 / log2(i + 2)29 delta = abs((grades[i] - grades[j]) * (1 / math.log2(i + 2) - 1 / math.log2(j + 2))) / ideal30 swapped = ranking.copy()31 swapped[i], swapped[j] = swapped[j], swapped[i]32 checks.append(math.isclose(delta, abs(ndcg_at_k(swapped, judged, 10) - ndcg_at_k(ranking, judged, 10))))33 pulls.append(ranknet)34 pair_lambda = sigma * ranknet * delta # LambdaRank: scaled by |delta nDCG|35 force[better], force[worse] = force[better] + pair_lambda, force[worse] - pair_lambda3637print(f"pairs with different grades: {len(checks)}")38print(f"pairs where delta ndcg by formula matches swapping the pair: {sum(checks)}")39for position, push in enumerate(force, start=1):40 print(f"force at position {position}: {push:+.3f}")41# list indexes count from 0 and positions from 1, hence the + 142print(f"strongest push up, position: {force.index(max(force)) + 1}")43print(f"strongest push down, position: {force.index(min(force)) + 1}")44print(f"ranknet's pull, lowest: {min(pulls):.3f}\nranknet's pull, highest: {max(pulls):.3f}")
python lambdas-for-one-query.pygrades from the top: 1 2 2 0 1 2 0 2 0 0
pairs with different grades: 32
pairs where delta ndcg by formula matches swapping the pair: 32
force at position 1: +0.013
force at position 2: +0.159
force at position 3: +0.104
force at position 4: -0.083
force at position 5: -0.010
force at position 6: +0.063
force at position 7: -0.092
force at position 8: +0.063
force at position 9: -0.105
force at position 10: -0.111
strongest push up, position: 2
strongest push down, position: 10
ranknet's pull, lowest: 0.472
ranknet's pull, highest: 0.525
The formula's |ΔnDCG| matches the change from swapping the pair for 32 of the 32 pairs in the top 10 with different grades. The top 10's grades, from the top, are 1 2 2 0 1 2 0 2 0 0; an unjudged product counts as 0, as it does in the metric.
The strongest push up goes to position 2, an exact product, with +0.159; the strongest push down goes to position 10, an irrelevant product, with -0.111. The partial product at position 1 gets a net push of +0.013: the irrelevant products below push it up, and the exact ones push it down.
RankNet's factor in λ stays between 0.472 and 0.525, because the similarities lie close together, so |ΔnDCG| decides almost everything.
That example simplifies. It takes the ideal DCG from every judged product, as lib.metrics does, and looks at ten products. LightGBM takes the ideal from the query's training rows, and at 4.7.0 two defaults change the λ. lambdarank_truncation_level is 30, which limits training to the top 30 results: only pairs with a product in the current top 30 get a λ. lambdarank_norm is true, which normalizes the lambdas across queries; set it to false to get the original lambdarank algorithm (Parameters). Normalizing, as LightGBM's source does it, divides each pair's |ΔnDCG| by the gap between its scores plus 0.01, then rescales each query's λ (rank_objective.hpp). To check the formula against the library, write it as a custom LightGBM objective, grow five small trees on this query's candidates, and grow them again with LightGBM's lambdarank, with and without those defaults:
1"""Writes LambdaRank's gradients by hand as a custom LightGBM objective, then checks them against LightGBM's own lambdarank2on one query, with and without its defaults. The prose's point rests on the two printed differences: a rounding-sized gap3without the defaults, and a large one at them."""4import numpy as np5from lightgbm import LGBMRanker67from lib.ranking import DETERMINISTIC, FEATURES, feature_table8from lib.store import load_products, load_queries910queries = load_queries("train")11one = queries[queries["query"] == "outdoor waterproof chest"]12rows = feature_table(load_products(), one) # one query's candidates, in product_id order13# unjudged candidates count as grade 0, as in the metric14grades = rows["grade"].fillna(0).astype(int).to_numpy()15SIGMA = 1.0 # LightGBM's default sigmoid161718def lambdas(y_true, y_pred, weight, group):19 """The chapter's lambda for every pair with different grades, as gradients and second derivatives."""20 rank = np.empty(len(y_pred), dtype=int)21 rank[np.argsort(-y_pred, kind="stable")] = np.arange(len(y_pred)) # LightGBM's stable sort on score22 # rank counts from 0, so the product at position rank + 1 has the discount 1 / log2(rank + 2)23 discount, ideal = 1 / np.log2(rank + 2), np.sum(np.sort(y_true)[::-1] / np.log2(np.arange(len(y_true)) + 2))24 gradient, hessian = np.zeros(len(y_pred)), np.zeros(len(y_pred))25 for i in range(len(y_true)):26 for j in np.flatnonzero(y_true < y_true[i]): # i should rank above j27 delta = (y_true[i] - y_true[j]) * abs(discount[i] - discount[j]) / ideal28 pull = 1 / (1 + np.exp(SIGMA * (y_pred[i] - y_pred[j])))29 gradient[i], gradient[j] = gradient[i] - SIGMA * pull * delta, gradient[j] + SIGMA * pull * delta30 hessian[[i, j]] += SIGMA ** 2 * pull * (1 - pull) * delta31 return gradient, hessian323334# five small trees; min_child_samples is 5 where LightGBM's default is 20, so the trees can split this query's candidates35tree = {"n_estimators": 5, "num_leaves": 3, "min_child_samples": 5, **DETERMINISTIC}36# turns off normalization and sets truncation to every candidate, so every pair gets a λ, as the formula has it37plain = {"lambdarank_norm": False, "lambdarank_truncation_level": len(rows)} # LightGBM's lambdas, untouched38models = {"by hand": LGBMRanker(objective=lambdas, **tree),39 "lightgbm, no normalization or truncation": LGBMRanker(objective="lambdarank", label_gain=[0, 1, 2], **plain, **tree),40 "lightgbm at its defaults": LGBMRanker(objective="lambdarank", label_gain=[0, 1, 2], **tree)}41# raw scores: the values the gradients act on, before any sigmoid42scores = {name: model.fit(rows[FEATURES], grades, group=[len(rows)]).predict(rows[FEATURES], raw_score=True)43 for name, model in models.items()}44print(f"candidates: {len(rows)}")45for name in list(models)[1:]:46 difference = np.abs(scores[name] - scores["by hand"]).max()47 print(f"{name}, largest difference from the lambdas by hand after 5 trees: {difference:.6f}")
python lambdas-match-lightgbm.pycandidates: 180
lightgbm, no normalization or truncation, largest difference from the lambdas by hand after 5 trees: 0.000006
lightgbm at its defaults, largest difference from the lambdas by hand after 5 trees: 0.595667
Over the query's 180 candidates, LightGBM without normalization or truncation scores every product within 0.000006 of the custom objective's scores. The gap is rounding, since LightGBM reads its sigmoid from a precomputed table (rank_objective.hpp). At its defaults, scores differ by up to 0.595667.
Working rule: Train rankers with a gradient that weighs each pair by the metric you report, as LambdaMART does, rather than regressing on grades. Know which defaults of your library change that gradient, and check on train queries that it beats a pointwise model on your data, because the families' order in a benchmark doesn't transfer by itself.
LightGBM で LambdaMART を学習する
LightGBM は LGBMRanker と objective="lambdarank" で LambdaMART を学習する。注意を要する設定が四つあり、いずれも 4.7.0 向けに文書化されている。
- 行はクエリごとにまとめる。 「data should be ordered by the query」とあり、
groupには、クエリ ID ではなく、その順序での各クエリの行数を並べる。したがって各サイズの合計は行数に等しくなる(sum(group) = n_samples、Parameters、LGBMRanker)。 - ゲインは既定で指数型である。
label_gainの既定値は0,1,3,7,15,31,63,...であり、「the gain of label2is3」となる。また、すべてのラベルはゲインの個数より小さくなければならない(Parameters)。これは、ゲインを 2^grade − 1 とする Burges の DCG である。本書の nDCG は線形ゲインを用いるため、label_gain=[0, 1, 2]と設定する。 - nDCG は既定で 1 位から 5 位までで評価される。 nDCG@10 で停止させるには
eval_at=[10]を設定し、最終スコアはいずれにせよlib.metricsで報告する。 - 検証データは
eval_Xとeval_yに渡す。 4.7.0 では、eval_setは「Deprecated since version 4.7.0」であり、eval_Xとeval_yがこれに代わる(LGBMRanker、リリースノート)。例ではeval_X、eval_y、eval_groupを渡す。
早期停止は、木を一本追加するたびに、取り置いた train クエリを採点し、さらに 30 本追加しても改善が得られなくなった時点で、最良スコアまでの木を残す。学習を正確に再現できるよう、例ではスレッド数を 1 に、シードを固定値に設定し、deterministic=True を用いる。これは「should ensure the stable results when using the same data」とされている(Parameters)。あわせて force_row_wise=True も指定する。同じ項目が推奨しているもので、数値計算上の問題による不安定さが生じる可能性を避けるためである。lib.ranking はこれらの設定を保持しており、その fit がグループ化と早期停止を適用する。
行の順序も重要である。最初の木の前ではすべてのスコアが 0 であるため、LightGBM は行の並びを最初の λ 計算のための初期ランキングとして扱い、スコアが等しい場合には同順位の解消規則が必要になる。クロスエンコーダの記録は各クエリの候補を上位から順に並べているため、そのままではモデルが何も学習しないうちにクロスエンコーダのランキングを与えてしまう。そこで feature_table は候補を product_id で並べ替え、lib.ranking.ranked は同順位を product_id で解消する。
木の大きさは、train クエリに対する交差検証で選ぶ。クエリを五つの分割に分け、各分割を、他の四つのうち三つで学習し残る一つで早期停止させたモデルでランキングし、すべてのクエリをちょうど一度ずつ採点する。葉の数を 3 から 31 まで、学習率を二通りとして、八つの設定を試す。指標と同様に、判定のない候補は等級 0 として数える。
1"""Cross-validates LambdaMART on the train queries over eight tree sizes and learning rates, and names the best. The prose's2point rests on the best setting's score, and on the score beside it for LightGBM's default of 31 leaves at 0.1."""3from lightgbm import LGBMRanker45from lib.metrics import evaluate6from lib.ranking import DETERMINISTIC, FEATURES, LAMBDARANK, TREE_GRID, TREES, cross_validate, feature_table7from lib.semantic import SemanticSearch8from lib.store import load_products, load_queries, qrels910queries, judgments = load_queries("train"), qrels("train") # tuning: train queries only11candidates = feature_table(load_products(), queries)12candidates["grade or 0"] = candidates["grade"].fillna(0).astype(int) # unjudged counts as irrelevant, as in nDCG13print(f"train queries: {len(queries)}")14print(f"semantic, train ndcg@10: {evaluate(SemanticSearch().run(queries, k=100), judgments)['ndcg@10'].mean():.3f}")1516results = {}17for setting in TREE_GRID: # LightGBM's default is 31 leaves at 0.118 name = f"{setting['num_leaves']} leaves, learning rate {setting['learning_rate']}"19 # up to 500 trees; early stopping inside cross_validate keeps fewer20 model = lambda: LGBMRanker(**LAMBDARANK, n_estimators=500, **setting, **DETERMINISTIC)21 # one nDCG@10 per train query, each ranked by a model that never saw that query22 results[name] = cross_validate(model, candidates, candidates, "grade or 0", FEATURES, judgments)23 print(f"{name}, cross-validated ndcg@10: {results[name].mean():.3f}")24 # the trees early stopping kept in each fold25 print(f"{name}, trees kept per fold: {', '.join(map(str, results[name].attrs['trees']))}")26best = max(results, key=lambda name: results[name].mean())27print(f"settings tried: {len(results)}")28print(f"best setting: {best}")29print(f"lib.ranking.TREES holds it: {'yes' if best == f'{TREES['num_leaves']} leaves, learning rate {TREES['learning_rate']}' else 'no'}")
python lambdamart-tree-size-on-train.pytrain queries: 50
semantic, train ndcg@10: 0.772
3 leaves, learning rate 0.05, cross-validated ndcg@10: 0.780
3 leaves, learning rate 0.05, trees kept per fold: 18, 27, 52, 20, 85
3 leaves, learning rate 0.1, cross-validated ndcg@10: 0.798
3 leaves, learning rate 0.1, trees kept per fold: 99, 20, 8, 43, 38
7 leaves, learning rate 0.05, cross-validated ndcg@10: 0.744
7 leaves, learning rate 0.05, trees kept per fold: 2, 28, 30, 39, 55
7 leaves, learning rate 0.1, cross-validated ndcg@10: 0.772
7 leaves, learning rate 0.1, trees kept per fold: 80, 12, 13, 7, 20
15 leaves, learning rate 0.05, cross-validated ndcg@10: 0.720
15 leaves, learning rate 0.05, trees kept per fold: 4, 37, 175, 44, 120
15 leaves, learning rate 0.1, cross-validated ndcg@10: 0.728
15 leaves, learning rate 0.1, trees kept per fold: 19, 68, 136, 48, 53
31 leaves, learning rate 0.05, cross-validated ndcg@10: 0.679
31 leaves, learning rate 0.05, trees kept per fold: 15, 8, 91, 8, 22
31 leaves, learning rate 0.1, cross-validated ndcg@10: 0.737
31 leaves, learning rate 0.1, trees kept per fold: 11, 75, 166, 96, 58
settings tried: 8
best setting: 3 leaves, learning rate 0.1
lib.ranking.TREES holds it: yes
8 の設定のうち最良は 3 leaves, learning rate 0.1 で、0.798 である。LightGBM の既定である葉 31 個、学習率 0.1(LGBMRanker)は 0.737 となる。全設定の中の最良値は自らを実際より良く見せるため、取り置いたクエリではこれより低い値を見込むべきである。
ひとつの選択は、ストアの判定に依存する。train 候補の 0.286 は、誰も判定していない。モデルはこれらを等級 0 として数えることも、判定のある候補のみから学習することもできる。未判定は無関連ではないが、どちらの場合でも、モデルはクエリ時にそれらをランキングすることになる。選んだ木の大きさで両者を交差検証により比較し、等級に対する pointwise 回帰も並べる。
1"""Compares three ways of training on the train queries, each by cross-validation: LambdaRank with unjudged candidates as 0,2LambdaRank on judged candidates only, and pointwise regression. The prose's point rests on the differences and the3permutation p-values printed under each setup."""4from lightgbm import LGBMRanker, LGBMRegressor56from lib.metrics import evaluate7from lib.ranking import DETERMINISTIC, FEATURES, LAMBDARANK, TREES, cross_validate, feature_table8from lib.semantic import SemanticSearch9from lib.significance import compare10from lib.store import load_products, load_queries, qrels1112queries, judgments = load_queries("train"), qrels("train") # tuning: train queries only13candidates = feature_table(load_products(), queries)14candidates["grade or 0"] = candidates["grade"].fillna(0).astype(int) # unjudged counts as irrelevant15# the rows the model learns from when unjudged candidates are left out; it still ranks every candidate at query time16judged_only = candidates.dropna(subset=["grade"]).astype({"grade": int})1718setups = { # (model, rows it learns from, label)19 "lambdarank, unjudged as 0": (lambda: LGBMRanker(**LAMBDARANK, **TREES, **DETERMINISTIC), candidates, "grade or 0"),20 "lambdarank, unjudged left out": (lambda: LGBMRanker(**LAMBDARANK, **TREES, **DETERMINISTIC), judged_only, "grade"),21 "pointwise regression, unjudged as 0": (lambda: LGBMRegressor(**TREES, **DETERMINISTIC), candidates, "grade or 0"),22}23semantic = evaluate(SemanticSearch().run(queries, k=100), judgments)["ndcg@10"]24print(f"semantic, train ndcg@10: {semantic.mean():.3f}")25scores = {name: cross_validate(model, rows, candidates, label, FEATURES, judgments)26 for name, (model, rows, label) in setups.items()}27chosen = scores["lambdarank, unjudged as 0"]28for name, per_query in scores.items():29 print(f"{name}, cross-validated ndcg@10: {per_query.mean():.3f}")30 if name != "lambdarank, unjudged as 0":31 # the mean difference is the first argument minus the second, tested by permutation over the queries32 result = compare(per_query, chosen)33 print(f"{name}, minus the first: {result['mean difference']:.3f}")34 print(f"{name}, minus the first, permutation p: {result['permutation p']:.3f}")35result = compare(chosen, semantic)36print(f"first setup minus semantic: {result['mean difference']:.3f}")37print(f"first setup minus semantic, permutation p: {result['permutation p']:.3f}")
python lambdamart-choices-on-train.pysemantic, train ndcg@10: 0.772
lambdarank, unjudged as 0, cross-validated ndcg@10: 0.798
lambdarank, unjudged left out, cross-validated ndcg@10: 0.733
lambdarank, unjudged left out, minus the first: -0.064
lambdarank, unjudged left out, minus the first, permutation p: 0.005
pointwise regression, unjudged as 0, cross-validated ndcg@10: 0.769
pointwise regression, unjudged as 0, minus the first: -0.029
pointwise regression, unjudged as 0, minus the first, permutation p: 0.007
first setup minus semantic: 0.026
first setup minus semantic, permutation p: 0.242
未判定の候補を 0 として数えるほうが、除外するよりスコアが高く、0.798 対 0.733、差は -0.064、順列検定の p 値は 0.005 である。学習で未判定の候補を一度も見ないモデルも、それらをランキングしなければならず、特徴量はそれらが無関連な製品に似ていることを示している。
pointwise 回帰のスコアは 0.769 で、差は -0.029(p = 0.007)である。したがって、ストアの train クエリでは、pointwise モデルとの調整チェックは LambdaMART に軍配を上げる。
セマンティック検索の 0.772 に対して、選んだ設定との差は 0.026 (選んだ設定からセマンティック検索を引いた値)で、順列検定の p 値は 0.242 である。その設定を選んだクエリ自体においてすら、train クエリで偶然と区別できるほどの改善は見られない。
実践ルール: 行をクエリごとに並べ替えて group を渡し、label_gain には指標が用いるゲインを設定し、eval_X と eval_y を通じて取り置いた train クエリで早期停止させ、シードとスレッド数を固定する。各クエリの候補は、3 章で勧めたとおり製品 ID で並べ、スコアが同順位の場合も同じ方法で解消する。train クエリでの交差検証が別の結果を示さない限り、未判定の候補は等級 0 として数える。判定済みクエリが少ない場合は、小さな木を試す。
この部分はまだ翻訳されていないため、英語で表示している。
Report LambdaMART on the test queries
A model tuned on train queries is reported once on the test queries, by a comparison planned before looking. Feature importance shows what a model learned: LightGBM's "gain" importance is the "total gains of splits which use the feature" (Booster), so a feature with no gain plays no part in the score.
The first planned comparison uses the setup chosen on train: LambdaMART on all the features, with 3 leaves at a learning rate of 0.1, unjudged candidates as grade 0, and candidates in product_id order. It fits on 40 train queries, stops early on 10 others, and ranks each test query's candidates. A gain needs a permutation p-value below 0.01:
1"""Fits LambdaMART on the train queries with the chosen setup, reports its early stopping and gain shares, then reports its2planned comparison on the test queries against semantic retrieval. The prose's point rests on that comparison's permutation3p-value, printed last, against the threshold printed first."""4from lightgbm import LGBMRanker56from lib.metrics import evaluate7from lib.ranking import (DETERMINISTIC, FEATURES, LAMBDARANK, PLANNED_TEST_COMPARISONS, THRESHOLD, TREES, feature_table,8 fit, ranked, report_planned, stopping_split)9from lib.semantic import SemanticSearch10from lib.store import load_products, load_queries, qrels1112products, train, test = load_products(), load_queries("train"), load_queries("test")13print(f"planned test comparisons: {len(PLANNED_TEST_COMPARISONS)}")14print(f"p-value threshold: {THRESHOLD:.2f}")1516table = feature_table(products, train)17table["grade or 0"] = table["grade"].fillna(0).astype(int) # the setup chosen on train18# 10 train queries are held out, at random with a fixed seed, to stop on19fit_ids, stop_ids = stopping_split(train.index)20model = fit(LGBMRanker(**LAMBDARANK, **TREES, **DETERMINISTIC), table, "grade or 0", FEATURES, fit_ids, stop_ids)21print(f"queries fitted: {len(fit_ids)}\nqueries held out for stopping: {len(stop_ids)}")22curve = model.evals_result_["valid_0"]["ndcg@10"] # LightGBM's nDCG@10, one value per tree23print(f"trees tried: {len(curve)}\ntrees kept by early stopping: {model.best_iteration_}")24print(f"stopping queries' ndcg@10, first tree: {curve[0]:.3f}")25# best_iteration_ counts trees from 1 and the curve from 0, so the last tree kept is at index best_iteration_ - 126print(f"stopping queries' ndcg@10, trees kept: {curve[model.best_iteration_ - 1]:.3f}")27gain = model.booster_.feature_importance(importance_type="gain") # total gain of the splits on each feature28# a feature with no gain was never split on, so it plays no part in the score29print(f"features the trees use: {(gain > 0).sum()} of {len(FEATURES)}")30for name, share in sorted(zip(FEATURES, gain / gain.sum()), key=lambda pair: -pair[1]):31 print(f"share of gain, {name}: {share:.3f}")3233candidates, judgments = feature_table(products, test), qrels("test") # test queries: read once, to report34after = evaluate(ranked(candidates, model.predict(candidates[FEATURES])), judgments)35before = evaluate(SemanticSearch().run(test, k=100), judgments)36for name, frame in [("semantic", before), ("lambdamart", after)]:37 print(f"{name}, test ndcg@10: {frame['ndcg@10'].mean():.3f}")38 print(f"{name}, test judged@10: {frame['judged@10'].mean():.3f}")39 # each query's top 10 is 10 places, so (1 - judged@10) times 10 counts its unjudged places40 print(f"{name}, unjudged places in the test top 10s: {round((1 - frame['judged@10']).sum() * 10)}")41report_planned("lambdamart", after["ndcg@10"], before["ndcg@10"])
python lambdamart-on-test.pyplanned test comparisons: 5
p-value threshold: 0.01
queries fitted: 40
queries held out for stopping: 10
trees tried: 243
trees kept by early stopping: 213
stopping queries' ndcg@10, first tree: 0.568
stopping queries' ndcg@10, trees kept: 0.794
features the trees use: 8 of 8
share of gain, cross_encoder: 0.641
share of gain, similarity: 0.172
share of gain, name_match: 0.097
share of gain, bm25: 0.043
share of gain, review_count: 0.021
share of gain, rating_count: 0.021
share of gain, attribute_match: 0.002
share of gain, average_rating: 0.001
semantic, test ndcg@10: 0.830
semantic, test judged@10: 0.990
semantic, unjudged places in the test top 10s: 3
lambdamart, test ndcg@10: 0.817
lambdamart, test judged@10: 0.990
lambdamart, unjudged places in the test top 10s: 3
lambdamart, mean difference: -0.012
lambdamart, wins, losses, ties: 14, 12, 4
lambdamart, permutation p: 0.260
lambdamart, established at 0.01: no
lambdamart, t-test p: 0.278
lambdamart, t-test agrees: yes
lambdamart, 99% interval: -0.044 to 0.011
lambdamart, interval agrees: yes
Of 243 trees tried, early stopping keeps 213. On the queries held out for stopping, LightGBM's own nDCG@10, whose ideal comes from their candidates alone, rises from 0.568 after the first tree to 0.794 at the trees kept.
The trees use 8 of 8 features, but they lean on the cross-encoder, which takes a share of 0.641 of the gain, ahead of similarity at 0.172, name_match at 0.097 and bm25 at 0.043. So serving the model means a cross-encoder call for every candidate.
LambdaMART doesn't establish a gain on the test queries: it scores 0.817 nDCG@10, against semantic retrieval's 0.830, at a permutation p-value of 0.260 (all five comparisons). Judged@10 is 0.990 for LambdaMART and 0.990 for semantic retrieval.
Working rule: Adopt a learned ranker only when its planned test comparison is established. Read its gain importance before you trust it, because the features it leans on are the ones you'll pay for at query time. Early stopping on a few held-out queries keeps very different numbers of trees from one split to the next, from 99, 20, 8, 43, 38 across the cross-validation folds to 213 here. A rule of thumb, not a measurement: with this few queries, judge more queries before you tune further, and treat any one stopping point as noise.
クロスエンコーダでセマンティック検索を再ランキングする
セマンティック検索は、バイエンコーダとも呼ばれる埋め込みモデルを用いる。このモデルはクエリと各商品を別々に読み込むため、商品の埋め込みはあらかじめ計算しておける。クロスエンコーダは、クエリと1つの商品を一緒に読み込み、1つのスコアを出力する(Nogueira and Cho, 2019)。クエリの各単語と商品の各単語を結び付けられるが、その分だけ計算コストがかかる。テキストごとではなく組ごとに計算するため、Sentence Transformer モデルより遅いことが多い。そのため「クロスエンコーダは上位 k 件の結果を再ランキングするのによく使われる」(Sentence Transformers)。ストアのクロスエンコーダ(冒頭を参照)は、言語モデルとしては小さく、パラメータ数は149Mである(モデルカード)。これはオープンモデルの Alibaba-NLP/gte-reranker-modernbert-base であり、記録とともに保存されるマニフェストには、そのライセンスが Apache-2.0 と記されている。このモデルを選んだのは、誰でもダウンロードして記録を再実行できるからである。recordings/cross_encoder.py は、すべてのクエリについて、BM25 とセマンティック検索の上位100件の和集合にスコアを付けた(2つのクエリでは、100位での僅差のため、そのセマンティックのリストが lib.semantic のものと異なる)。
第1章ではハイブリッドの実行結果を再ランキングした。ここでは、クロスエンコーダが、打ち負かすべきランカーであるセマンティック検索の上位 20 件を再ランキングする。これはLLM リランカーが読んだのと同じ商品であり、この章の3つのリランカーは同一のリストから出発する。この章は、テスト結果を見る前にこの深さを固定しており、調整はしない。上位20件を記録済みのスコアで並べ替え、残りはそのままにする。これは2番目の計画済みの比較であり、p 値が 0.01 を下回る必要がある。
1"""Reranks semantic retrieval's top 20 with the recorded cross-encoder scores, and compares the two on the test queries.2The prose's point rests on the planned comparison's permutation p-value, and on the judged@10 lines, which show what the3rerank does to the judgments it can score."""4from lib.metrics import evaluate5from lib.ranking import DEPTH, rerank, report_planned6from lib.recorded import cross_encoder_scores, llm_listwise, manifest7from lib.semantic import SemanticSearch8from lib.store import load_queries, qrels910queries, judgments = load_queries("test"), qrels("test") # scored on the human judgments11semantic = SemanticSearch().run(queries, k=100)12recorded = cross_encoder_scores()13# the recorded score of each (query, product) pair14score = dict(zip(zip(recorded["query_id"], recorded["product_id"]), recorded["score"]))15print("cross-encoder model:", manifest("cross-encoder")["model"])16print("licence in the manifest:", manifest("cross-encoder")["model_licence"])1718# the set of products the LLM was shown for each query, from the listwise recording19shown = llm_listwise().groupby("query_id")["product_id"].apply(set) # what the LLM rerankers saw20same = sum(set(semantic[query_id][:DEPTH]) == shown[query_id] for query_id in queries.index)21print(f"rerank depth: {DEPTH}")22print(f"queries whose top {DEPTH} holds the products the llm rerankers saw: {same} of {len(queries)}")2324reranked = rerank(semantic, score, DEPTH) # top 20 reordered, the rest in place25before, after = evaluate(semantic, judgments), evaluate(reranked, judgments)26for name, frame in [("semantic", before), ("cross-encoder", after)]:27 print(f"{name}, test ndcg@10: {frame['ndcg@10'].mean():.3f}")28 print(f"{name}, test judged@10: {frame['judged@10'].mean():.3f}")29 print(f"{name}, unjudged places in the test top 10s: {round((1 - frame['judged@10']).sum() * 10)}")30# index 0 is the first place31moved = sum(reranked[query_id][0] != semantic[query_id][0] for query_id in queries.index)32print(f"queries whose first product changes: {moved}")33report_planned("cross-encoder", after["ndcg@10"], before["ndcg@10"])
python cross-encoder-on-semantic.pycross-encoder model: Alibaba-NLP/gte-reranker-modernbert-base
licence in the manifest: Apache-2.0
rerank depth: 20
queries whose top 20 holds the products the llm rerankers saw: 30 of 30
semantic, test ndcg@10: 0.830
semantic, test judged@10: 0.990
semantic, unjudged places in the test top 10s: 3
cross-encoder, test ndcg@10: 0.813
cross-encoder, test judged@10: 0.983
cross-encoder, unjudged places in the test top 10s: 5
queries whose first product changes: 23
cross-encoder, mean difference: -0.016
cross-encoder, wins, losses, ties: 11, 12, 7
cross-encoder, permutation p: 0.178
cross-encoder, established at 0.01: no
cross-encoder, t-test p: 0.171
cross-encoder, t-test agrees: yes
cross-encoder, 99% interval: -0.058 to 0.006
cross-encoder, interval agrees: yes
上位20件を再ランキングしても、改善が確立されたとはいえない。再ランキングのスコアは 0.813 で、セマンティックの 0.830 に対し、順列検定の p 値は 0.178 である(判定の表を参照)。上位20件には、テストクエリ 30 of 30 件で LLM リランカーが見た商品が含まれており、リランカーは、そのうち 23 件で先頭の商品を変える。
Judged@10は 0.990 から 0.983 へ下がり、テストの上位10件に含まれる未判定の位置は 3 から 5 になる。
訓練クエリでは、同じ再ランキングのスコアは 0.802 で、セマンティック検索の 0.772 に対するものとなる(深さの掃引)。訓練クエリは別の標本であり、その判定は第3章で分かったことを繰り返している。すなわち、近い2つのランカーの順位は、クエリの集合が変わると入れ替わりうる。
実践ルール: 最も強力な検索を、テスト結果を見る前に固定した深さで再ランキングし、保留したクエリで改善が確立された場合にのみそのリランカーを採用する。ストアの汎用クロスエンコーダは、パイプラインに入れない。
この部分はまだ翻訳されていないため、英語で表示している。
Score each product with an LLM
An LLM can rerank too, in three ways. Pointwise, an LLM judges each product alone and returns a label or a score. Pairwise, an LLM says which of two products fits the query better. Listwise, an LLM orders a whole list in one answer (Qin et al., 2024). In one study, on eight datasets from BEIR, a collection of retrieval benchmarks used to test models on data they weren't trained for, pointwise prompts did better with graded answers than with yes or no, because "the lack of intermediate relevance label options may cause the LLM to provide noisy or biased answers" (Zhuang et al., 2024a).
The store's LLM, the open model named in the chapter's opening, saw the query and one product at a time: name, class, category, description and attributes (what the LLM saw). The prompt asks for a label, exact, partial or irrelevant, and "a score from 0 to 100 for how well it matches". Reorder semantic retrieval's top 20 by that score; when two products tie, they keep semantic retrieval's order, a rule fixed with the recording. That reorder is the third planned comparison, held to a p-value below 0.01:
1"""Reads the pointwise LLM's answers for the test queries, one call per product, and reranks semantic retrieval's top 20 by2their scores. The prose's point rests on the distinct-scores line, which shows how coarse the scores are, and on the3planned comparison's p-value."""4from lib.metrics import evaluate5from lib.ranking import DEPTH, rerank, report_planned6from lib.recorded import llm_pointwise, manifest7from lib.semantic import SemanticSearch8from lib.store import load_queries, qrels910queries, judgments = load_queries("test"), qrels("test") # scored on the human judgments, not the LLM's11semantic = SemanticSearch().run(queries, k=100)12answers = llm_pointwise() # one call per product: a label and a score13print("llm:", manifest("llm-rerank")["model"])14print("licence in the manifest:", manifest("llm-rerank")["model_licence"])15print(f"calls per query: {len(answers) // answers['query_id'].nunique()}")16print(f"answers without a score: {answers['score'].isna().sum()}")17print(f"distinct scores per query, median: {answers.groupby('query_id')['score'].nunique().median():.0f}")18print(f"share of products scored 100: {(answers['score'] == 100).mean():.3f}")19# each answered product's human grade, or missing when nobody judged it20people = answers.apply(lambda row: judgments[row["query_id"]].get(row["product_id"]), axis=1)21# grades 2, 1 and 0 are exact, partial and irrelevant, the labels the LLM was asked for22for label, grade in [("exact", 2), ("partial", 1), ("irrelevant", 0)]:23 print(f"products the llm labels {label}: {(answers['label'] == label).sum()}")24 print(f"products people judged {label}: {(people == grade).sum()}")25print(f"products nobody judged: {people.isna().sum()}")2627score = dict(zip(zip(answers["query_id"], answers["product_id"]), answers["score"]))28reranked = rerank(semantic, score, DEPTH) # equal scores keep semantic retrieval's order29counts = answers.groupby("query_id")["score"].value_counts() # products per score, for each query30shared = {query_id: sum(counts[(query_id, score[(query_id, product_id)])] > 1 for product_id in reranked[query_id][:10])31 for query_id in queries.index} # top-10 products that share their score32# more than 5 of the top 10 share a score: most of the list is decided by the tie-break33print(f"queries where most of the reranked top 10 shares its score: {sum(n > 5 for n in shared.values())}")34before, after = evaluate(semantic, judgments), evaluate(reranked, judgments)35for name, frame in [("semantic", before), ("llm pointwise", after)]:36 print(f"{name}, test ndcg@10: {frame['ndcg@10'].mean():.3f}")37 print(f"{name}, test judged@10: {frame['judged@10'].mean():.3f}")38 print(f"{name}, unjudged places in the test top 10s: {round((1 - frame['judged@10']).sum() * 10)}")39report_planned("llm pointwise", after["ndcg@10"], before["ndcg@10"])
python llm-pointwise-rerank.pyllm: gemma4:12b-it-q4_K_M
licence in the manifest: Apache-2.0
calls per query: 20
answers without a score: 0
distinct scores per query, median: 4
share of products scored 100: 0.390
products the llm labels exact: 251
products people judged exact: 310
products the llm labels partial: 345
products people judged partial: 265
products the llm labels irrelevant: 4
products people judged irrelevant: 7
products nobody judged: 18
queries where most of the reranked top 10 shares its score: 30
semantic, test ndcg@10: 0.830
semantic, test judged@10: 0.990
semantic, unjudged places in the test top 10s: 3
llm pointwise, test ndcg@10: 0.837
llm pointwise, test judged@10: 0.983
llm pointwise, unjudged places in the test top 10s: 5
llm pointwise, mean difference: 0.008
llm pointwise, wins, losses, ties: 11, 8, 11
llm pointwise, permutation p: 0.241
llm pointwise, established at 0.01: no
llm pointwise, t-test p: 0.244
llm pointwise, t-test agrees: yes
llm pointwise, 99% interval: -0.007 to 0.026
llm pointwise, interval agrees: yes
The recording made 20 calls per query, and 0 answers came back without a score. The scores are coarse. The median query gets 4 distinct scores for its 20 products, and a share of 0.390 of all products score 100. In 30 queries, most of the reranked top 10 shares its score with another product, so the tie-break, semantic retrieval's order, decides much of the list.
With this prompt, on these lists, the LLM calls fewer products exact than people did:
| Label | LLM | People |
|---|---|---|
| exact | 251 | 310 |
| partial | 345 | 265 |
| irrelevant | 4 | 7 |
| nobody judged | – | 18 |
On these lists that holds, but chapter 4 shows that the direction of an LLM's disagreement with people changes with the pairs judged and the prompt, so don't carry this one elsewhere.
The pointwise rerank doesn't establish a gain either: it scores 0.837 against 0.830, at a permutation p-value of 0.241 (see the decisions table).
The LLM's order puts more unjudged products into the top 10s: 5 places instead of 3, and judged@10 falls from 0.990 to 0.983.
Working rule: If you rerank pointwise with an LLM, ask for a graded label or a score, count the distinct scores it gives per query, and decide the tie-break in advance: retrieval's order is a safe default. Score the result with human judgments, never with the same LLM's grades.
1回のLLMプロンプトで20件の商品を並べ替える
3つ目の方法であるリストワイズプロンプトは、候補に番号を振り、関連性の高い順に番号を返すようLLMに求める。RankGPTはこれを順列生成として導入した(Sun et al., 2023)。1つのプロンプトに収められる商品数には限りがあるため、長いリストはスライディングウィンドウで再ランキングする。RankGPTはBM25の上位100件を、リストの下から上へ向かって「ウィンドウサイズ20、ステップサイズ10」で再ランキングしている(3.2節および6.1節)。回答は有効な順序でなければならず、これは失敗しうる。2023年当時の中程度のLLM、たとえば200億パラメータのFlan-UL2では、リストワイズプロンプトはほぼ常に失敗した。順位の欠落、重複、拒否のいずれかが起きたのである(Qin et al., 2024)。
このストアのLLMは、1回のプロンプトで20件の商品、すなわち1つのウィンドウを見た。プロンプトは各商品を短い1行で示す。商品名、クラス、属性、そして説明文の先頭160文字である。最後の指示は次のとおりである。「Order all 20 products from the best match for the query to the worst. Return the product numbers, each exactly once, in JSON.」JSONスキーマが要求するのはちょうど20個の数値であり、20個の異なる数値ではない。そのため、記録の段階で、番号が重複または欠落している回答を修復し、その旨を記録する。セマンティック検索の上位20件を、その20件をセマンティック順で見せたときにLLMが返した回答の順に並べ替える。この再ランキングは4つ目の計画済みの比較であり、p値が0.01を下回ることを基準とする。
1"""Reads the listwise LLM's answers for the test queries, in both orders, and reranks semantic retrieval's top 20 by the2answer given in semantic order. The prose's point rests on the planned comparison's p-value, and on the repair count,3which shows how many answers needed a fix."""4from lib.metrics import evaluate5from lib.ranking import DEPTH, report_planned6from lib.recorded import llm_listwise, manifest7from lib.semantic import SemanticSearch8from lib.store import load_queries, qrels910queries, judgments = load_queries("test"), qrels("test") # scored on the human judgments, not the LLM's11semantic = SemanticSearch().run(queries, k=100)12answers = llm_listwise() # one call per query and order shown13# the answers hold one row per product, so keep one row per call: one call per query and order14calls = answers.drop_duplicates(["query_id", "order"])15print(f"listwise calls, both orders: {len(calls)}")16# valid is false where the recording repaired a repeated or missing number17print(f"answers that needed repair: {(~calls['valid']).sum()}")18print(f"products per call: {answers.groupby(['query_id', 'order']).size().max()}")19print("prompt asks for:", manifest("llm-rerank")["prompts"]["listwise"].splitlines()[-1].replace("{n}", str(DEPTH)))2021shown_in_semantic_order = answers[answers["order"] == "semantic"].sort_values(["query_id", "position"])22order = shown_in_semantic_order.groupby("query_id")["product_id"].apply(list)23# the reranked top 20, then the rest of semantic retrieval's list in place: ranking[DEPTH:] starts at the 21st product24reranked = {query_id: order[query_id] + ranking[DEPTH:] for query_id, ranking in semantic.items()}25before, after = evaluate(semantic, judgments), evaluate(reranked, judgments)26for name, frame in [("semantic", before), ("llm listwise", after)]:27 print(f"{name}, test ndcg@10: {frame['ndcg@10'].mean():.3f}")28 print(f"{name}, test judged@10: {frame['judged@10'].mean():.3f}")29 print(f"{name}, unjudged places in the test top 10s: {round((1 - frame['judged@10']).sum() * 10)}")30moved = sum(reranked[query_id][0] != semantic[query_id][0] for query_id in queries.index)31print(f"queries whose first product changes: {moved}")32report_planned("llm listwise", after["ndcg@10"], before["ndcg@10"])
python llm-listwise-rerank.pylistwise calls, both orders: 60
answers that needed repair: 0
products per call: 20
prompt asks for: Order all 20 products from the best match for the query to the worst. Return the product numbers, each exactly once, in JSON.
semantic, test ndcg@10: 0.830
semantic, test judged@10: 0.990
semantic, unjudged places in the test top 10s: 3
llm listwise, test ndcg@10: 0.819
llm listwise, test judged@10: 0.983
llm listwise, unjudged places in the test top 10s: 5
queries whose first product changes: 19
llm listwise, mean difference: -0.010
llm listwise, wins, losses, ties: 12, 12, 6
llm listwise, permutation p: 0.580
llm listwise, established at 0.01: no
llm listwise, t-test p: 0.498
llm listwise, t-test agrees: yes
llm listwise, 99% interval: -0.070 to 0.016
llm listwise, interval agrees: yes
記録では、テストクエリ1件につき2回、合計60回のリストワイズ呼び出しを行い、0件の回答に修復が必要であった。この再ランキングにより、19件のクエリで先頭の商品が変わる。
リストワイズ再ランキングによる改善は確立されていない。スコアは0.819で、セマンティック検索の0.830に対して、並べ替え検定のp値は0.580である(5つの比較すべて)。
Judged@10もここで低下し、0.990から0.983になる。未判定の順位は5か所で、セマンティック検索の3か所より多い。
実践ルール: リストワイズの回答は、使う前に必ず検査する。欠落、重複、未知の番号を数え、修復が必要だった回答の件数を記録する。実行環境がスキーマに対応しているなら、スキーマで回答の形式を制約する。ウィンドウ、ステップ、商品を見せる順序は、測定する前に固定する。次の検証が示すとおり、順序によって回答が変わるためである。
この部分はまだ翻訳されていないため、英語で表示している。
Reverse the input to measure positional bias
A listwise answer can suffer from positional bias: an LLM's tendency to let a product's place in the prompt affect its rank, not the click position bias of shoppers. "Large language models (LLMs) exhibit positional bias in how they use context, which especially complicates listwise ranking" (Tang et al., 2024). On TREC DL19, the 2019 queries of TREC's Deep Learning track, a passage-ranking benchmark, gpt-3.5-turbo reranked BM25's top 100 to an nDCG@10 of 65.80 when shown BM25's order, 32.77 when shown it reversed and 25.17 when shown a random order (Sun et al., 2023, Table 5). The direct check is to show the same products in two orders and compare the answers. Kendall's tau measures how alike two orderings are, from 1 for the same order to −1 for the reverse, by counting the pairs they put the same way round (chapter 4).
The recording sent each test query's listwise prompt to the same LLM twice. The first prompt showed semantic retrieval's top 20 in semantic order, as the listwise rerank used; the second showed them reversed. Out of the 30 test queries, how often will the LLM put the same product first both times? Write your number down.
1"""Shows the listwise LLM the same test products in two orders, and measures how far its answers agree and how far each2answer follows the order it was shown. The prose's point rests on the mean tau between the two answers, and on the two3taus between each answer and its own input order."""4import numpy as np5from scipy.stats import kendalltau67from lib.metrics import evaluate8from lib.ranking import DEPTH9from lib.recorded import llm_listwise10from lib.semantic import SemanticSearch11from lib.store import load_queries, qrels1213queries, judgments = load_queries("test"), qrels("test")14semantic = SemanticSearch().run(queries, k=DEPTH) # the order the first prompt showed15answers = llm_listwise().sort_values(["query_id", "order", "position"])16# the products in the order the LLM returned them, for each query and each order shown17answer = {key: list(group["product_id"]) for key, group in answers.groupby(["query_id", "order"])}1819def tau(first: list, second: list) -> float:20 """Kendall's tau between two orderings of the same products."""21 place = {product_id: rank for rank, product_id in enumerate(second)} # each product's place in the second ordering22 # the first ordering's positions, against where the second ordering puts each product23 return kendalltau(range(len(first)), [place[product_id] for product_id in first]).statistic2425between, shown_semantic, shown_reversed, overlap, same_first = [], [], [], [], 026for query_id, shown in semantic.items():27 forward, backward = answer[(query_id, "semantic")], answer[(query_id, "reversed")]28 between.append(tau(forward, backward))29 shown_semantic.append(tau(forward, shown))30 shown_reversed.append(tau(backward, shown[::-1])) # the reversed input, as the second prompt showed it31 overlap.append(len(set(forward[:10]) & set(backward[:10])))32 same_first += forward[0] == backward[0] # index 0 is the first place33print(f"test queries: {len(semantic)}")34print(f"tau between the two answers, mean: {np.mean(between):.3f}")35print(f"tau between the two answers, lowest: {min(between):.3f}\ntau between the two answers, highest: {max(between):.3f}")36print(f"top-10 products in common, mean: {np.mean(overlap):.3f}")37print(f"top-10 products in common, fewest: {min(overlap)}")38print(f"queries with the same first product: {same_first}")39print(f"share of queries with the same first product: {same_first / len(semantic):.3f}")40print(f"tau between the order shown and the answer, semantic order: {np.mean(shown_semantic):.3f}")41print(f"tau between the order shown and the answer, reversed order: {np.mean(shown_reversed):.3f}")42# an answer that ignored the order would repeat itself, so its tau with the reversed input is minus its tau with the semantic order43print(f"the same, if the order shown made no difference: {-np.mean(shown_semantic):.3f}")44reversed_run = {query_id: answer[(query_id, "reversed")] for query_id in semantic}45print(f"reversed answer, test ndcg@10, not a planned comparison: {evaluate(reversed_run, judgments)['ndcg@10'].mean():.3f}")
python listwise-position-bias.pytest queries: 30
tau between the two answers, mean: 0.311
tau between the two answers, lowest: -0.484
tau between the two answers, highest: 0.811
top-10 products in common, mean: 6.967
top-10 products in common, fewest: 3
queries with the same first product: 13
share of queries with the same first product: 0.433
tau between the order shown and the answer, semantic order: 0.414
tau between the order shown and the answer, reversed order: 0.045
the same, if the order shown made no difference: -0.414
reversed answer, test ndcg@10, not a planned comparison: 0.826
The same product comes first for 13 of the 30 queries, so the expectation that a capable model finds the same best match whichever way the list comes holds for only a share of 0.433 of the queries.
The two answers agree with a mean tau of 0.311, and per query the tau runs from -0.484 to 0.811. Their top 10s share 6.967 products on average, as few as 3.
The answers also lean toward the order shown. Each answer's tau against its own input order is 0.414 for the semantic order and 0.045 for the reversed one. An LLM that ignored the order would give the same answer both times, so its reversed answer would have a tau of -0.414 against the reversed input. The reversed answer sits far nearer its own input order than that: positional bias, for this model, prompt and these queries. The reversed answer scores 0.826 nDCG@10; it is not a planned comparison, so the chapter doesn't test it.
Three remedies exist. Pairwise ranking prompting asks the LLM about each pair twice, once in each order, and counts a disagreement as a tie (Qin et al., 2024). Permutation self-consistency reranks several shuffled orders and aggregates the answers, which raised scores by up to 7 to 18% for GPT-3.5 and 8 to 16% for LLaMA v2 (70B) (Tang et al., 2024). And rerankers fine-tuned for listwise ranking, such as the open RankZephyr, are reported to be "resilient against variations in initial document ordering" (Pradeep et al., 2023); the store didn't record one.
Working rule: Before you trust a listwise LLM reranker, run each query in at least two orders and report how much the answers agree. When agreement is low, aggregate several orders or ask each pair in both orders, and count those calls against your budget. In production, show the products in retrieval's order every time, so that the same list at least gets the same answer.
予算からリランク深度を決める
検索パイプラインは複数の段階で動き、後の段階ほど処理が遅く、渡される候補も少なくなる。本章のリランカーはどれも、その段階を一つ増やす。多段階ランカーは「各パイプライン段階への候補の受け入れを制御することで、品質とレイテンシのトレードオフを調整できる」(Nogueira et al., 2019)。したがって、リランカーがクエリごとに読む商品の数であるリランク深度は、予算に関わる判断である。実行時間はハードウェアや実行時の選択によって変わるため(Peng et al., 2025)、本節では時間そのものではなく、それを左右する量を数える。すなわち、クエリごとのモデル呼び出し回数と、モデルが読むテキストの量であり、これらにモデルの大きさを掛け合わせたものがコストになる。ストアのクロスエンコーダーは 149M パラメータ(モデルカード)、LLM は約 120 億パラメータ(モデルカード)で、後者はおよそ 80 倍である。
深度 20 で、各リランカーについて、テストクエリ 1 件あたりのモデル呼び出し回数と読み取る文字数を、商品のテキストと記録済みのプロンプトから数える。あわせて、ストアが記録していない二つの設計に必要な呼び出し回数も数える。
1"""Counts the characters each reranker reads per test query at depth 20, from the products' text and the recorded prompts,2and the model calls each design makes. The prose's point rests on the characters and calls lines, and on the two designs3the store didn't record."""4import numpy as np56from lib.agreement import as_the_llm_saw7from lib.ranking import DEPTH, listwise_line8from lib.recorded import manifest9from lib.semantic import SemanticSearch10from lib.store import load_products, load_queries, product_text1112products, queries = load_products(), load_queries("test")13semantic = SemanticSearch().run(queries, k=DEPTH) # the 20 products each reranker reads14prompts, text = manifest("llm-rerank")["prompts"], product_text(products)1516cross_encoder, pointwise, listwise = [], [], [] # characters read per query17for query_id, query in queries["query"].items():18 shown = products.loc[semantic[query_id]]19 # one call per product, each reading the query and that product's text together20 cross_encoder.append(sum(len(query) + len(text[product_id]) for product_id in semantic[query_id]))21 # the pointwise prompt, filled in with the query and the product as the LLM saw it22 pointwise.append(sum(len(prompts["pointwise"].format(query=query, product=as_the_llm_saw(product)))23 for _, product in shown.iterrows()))24 # products numbered from 1, as the listwise prompt shows them25 lines = "\n".join(f"[{number}] {listwise_line(product)}" for number, (_, product) in enumerate(shown.iterrows(), 1))26 listwise.append(len(prompts["listwise"].format(query=query, products=lines, n=DEPTH)))2728# the cross-encoder and the pointwise LLM make one call per product; the listwise LLM makes one call per query29for name, calls, characters in [("cross-encoder", DEPTH, cross_encoder), ("llm pointwise", DEPTH, pointwise),30 ("llm listwise", 1, listwise)]:31 print(f"{name}, model calls per query: {calls}")32 print(f"{name}, characters read per query, mean: {np.mean(characters):,.0f}")33print(f"llm pairwise, every pair in both orders, calls per query: {DEPTH * (DEPTH - 1)}")34# a sliding window of 20 products, moved 10 at a time: its starts are list positions 0, 10, ..., 8035window, step, depth = 20, 10, 100 # a sliding window over a deeper list36print(f"llm listwise over the top {depth}, window {window}, step {step}, calls per query: {(depth - window) // step + 1}")
python reranker-prompt-sizes.pycross-encoder, model calls per query: 20
cross-encoder, characters read per query, mean: 9,911
llm pointwise, model calls per query: 20
llm pointwise, characters read per query, mean: 17,864
llm listwise, model calls per query: 1
llm listwise, characters read per query, mean: 5,973
llm pairwise, every pair in both orders, calls per query: 380
llm listwise over the top 100, window 20, step 10, calls per query: 9
| リランカー(上位 20 件) | クエリあたりのモデル呼び出し回数 | クエリあたりの読み取り文字数 |
|---|---|---|
| クロスエンコーダー | 20 | 9,911 |
| LLM、ポイントワイズ | 20 | 17,864 |
| LLM、リストワイズ | 1 | 5,973 |
ポイントワイズは指示文を繰り返し、各商品を全文で示すため、読むテキストが最も多く、呼び出しは互いに独立している。リストワイズは読むテキストが最も少なく、呼び出しも 1 回で済むが、20 個の番号を順に書き出さなければならない。全ペアを両方の順序で比較するペアワイズ方式のプロンプトでは、クエリあたり 380 回の呼び出しが必要になり、上位 100 件に対する RankGPT のスライディングウィンドウでは 9 回が必要になる。TREC DL19 で公表されたある測定では、BM25 の上位 100 件を RTX A6000 GPU 1 枚上の Flan-T5-XL でリランクした場合、ポイントワイズの yes/no プロンプトは nDCG@10 が 0.650 でクエリあたり 1.5 秒、全ペアに対するペアワイズのプロンプトは 0.713 で 254.9 秒、LLM 呼び出しは 9,900 回だった(Zhuang et al., 2024b)。クロスエンコーダーを特徴量とする LambdaMART では、全候補についてクロスエンコーダーのスコアが必要になり、クエリあたりの中央値は 131 ペアである。
クロスエンコーダーでは、コストは深度に比例して増える。したがって問題は、品質の向上がどこで止まるかである。訓練クエリで深度を変えて調べる。
1"""Reranks semantic retrieval's top 5 to 75 with the recorded cross-encoder scores, on the train queries, and prints nDCG@102at each depth. The prose's point rests on the change from the depth before, which shows where the gain flattens and then3falls."""4from lib.metrics import evaluate5from lib.ranking import rerank6from lib.recorded import cross_encoder_scores7from lib.semantic import SemanticSearch8from lib.store import load_queries, qrels910queries, judgments = load_queries("train"), qrels("train") # choosing a depth is tuning: train queries11semantic = SemanticSearch().run(queries, k=100)12recorded = cross_encoder_scores()13# the recorded cross-encoder score of each (query, product) pair14score = dict(zip(zip(recorded["query_id"], recorded["product_id"]), recorded["score"]))15train_pairs = recorded[recorded["query_id"].isin(queries.index)]1617print(f"depth 0 (semantic alone), train ndcg@10: {evaluate(semantic, judgments)['ndcg@10'].mean():.3f}")18previous = round(evaluate(semantic, judgments)["ndcg@10"].mean(), 3) # changes between the printed values19for depth in [5, 10, 20, 30, 50, 75]:20 mean = round(evaluate(rerank(semantic, score, depth), judgments)["ndcg@10"].mean(), 3)21 print(f"depth {depth}, train ndcg@10: {mean:.3f}")22 print(f"depth {depth}, change from the depth before: {mean - previous:+.3f}")23 previous = mean24# every candidate needs its own cross-encoder score, so this counts the recorded pairs per query25print(f"pairs per query to score every candidate, median: {train_pairs.groupby('query_id').size().median():.0f}")
python rerank-depth-on-train.pydepth 0 (semantic alone), train ndcg@10: 0.772
depth 5, train ndcg@10: 0.782
depth 5, change from the depth before: +0.010
depth 10, train ndcg@10: 0.784
depth 10, change from the depth before: +0.002
depth 20, train ndcg@10: 0.802
depth 20, change from the depth before: +0.018
depth 30, train ndcg@10: 0.804
depth 30, change from the depth before: +0.002
depth 50, train ndcg@10: 0.793
depth 50, change from the depth before: -0.011
depth 75, train ndcg@10: 0.788
depth 75, change from the depth before: -0.005
pairs per query to score every candidate, median: 131
クロスエンコーダーの向上は深度 20 から 30 の間で頭打ちになる。これは LLM の記録に合わせて固定した深度の近くである。セマンティック検索の 0.772 から出発すると、上位 5 件のリランクで 0.782、上位 20 件で 0.802 となる。上位 30 件では +0.002 が加わるが、そのために必要なペア数は 5 割増えている。それより深くすると低下し、50 で 0.793、75 で 0.788 となる。
テストでは、深度 20 のリランクは向上を確立できなかった。したがってこのストアでは、最もコストの小さいリランカー、つまりリランカーを置かないことが、残すべき選択でもある。
実践ルール: 検討するリランカーごとに、各深度でクエリあたりの呼び出し回数と文字数を数え、深度は訓練クエリで決める。測定ではなく経験則だが、訓練時の nDCG@10 が改善しなくなる深度までしかリランクしない。それを超えた商品は同じコストがかかるのに何も加えないからである。そのうえで、その構成を自分のハードウェアとトラフィックで計測し、レイテンシ予算に収まる場合にのみ、その段階を残す。
クロスエンコーダを小さなランカーに蒸留する
予算に収まらないほど遅いリランカーも、高速なランカーを教えることができる。蒸留(distillation)は、小さなモデルである生徒(student)を訓練し、大きなモデルである教師(teacher)を模倣させる手法である。Hinton らは、これをアンサンブルの知識を単一のモデルに蒸留することとして述べている(Hinton et al., 2015)。ランキングでは、教師のスコアが生徒のラベルになる。2 つのモデルのスコアの大きさが異なる場合、たとえば異なるアーキテクチャのスコアではよくあることだが、生徒は 2 つの商品のスコアの差、すなわちマージンを学習することもできる。これには Margin-MSE と呼ばれる損失を用いる(Hofstätter et al., 2020)。蒸留は LLM から小さなモデルへも行われる。RankGPT は、蒸留した 440M のモデルが、BEIR ベンチマークで 3B の教師ありモデルを上回ったと報告している(Sun et al., 2023、BEIR)。最上級を含むクエリを対象としたある商品検索の研究では、LLM で直接リランキングすると遅延が許容できないほど大きくなるため、軽量なモデルが LLM の解釈を引き継いだ(Zhu et al., 2025)。教師は人手のラベルを必要とせず、すべての候補にラベルを付ける。
教師はクロスエンコーダである。生徒は、7の特徴量に対する LightGBM の回帰モデルである。これらの特徴量は、検索がすでに行っている 1 回のモデル呼び出しのほかには、モデル呼び出しを必要としない。cross_encoder以外のすべてがそれに当たる。ラベルは教師の生のスコアであり、木による回帰は教師自身のスケールでこれに合わせることができる。生徒は同じ候補をproduct_id順に並べ、木の大きさはLambdaMART のために選んだものを使う。まず、train クエリに対する交差検証で、生徒を、同じ特徴量で人手の評価から訓練した LambdaMART モデルと比較する。
1"""Trains a student LightGBM regression on the cross-encoder's scores, and a LambdaMART model on human grades with the same2features, both by cross-validation on the train queries. The prose's point rests on the student's score, and on its3difference from the human-graded ranker."""4from lightgbm import LGBMRanker, LGBMRegressor56from lib.metrics import evaluate7from lib.ranking import CHEAP_FEATURES, DETERMINISTIC, LAMBDARANK, TREES, cross_validate, feature_table, ranked8from lib.semantic import SemanticSearch9from lib.significance import compare10from lib.store import load_products, load_queries, qrels1112queries, judgments = load_queries("train"), qrels("train") # design work: train queries only13candidates = feature_table(load_products(), queries)14# unjudged candidates count as grade 0, as in the metric15candidates["grade or 0"] = candidates["grade"].fillna(0).astype(int)1617student = cross_validate(lambda: LGBMRegressor(**TREES, **DETERMINISTIC), candidates, candidates,18 "cross_encoder", CHEAP_FEATURES, judgments) # labels from the teacher19human = cross_validate(lambda: LGBMRanker(**LAMBDARANK, **TREES, **DETERMINISTIC), candidates, candidates,20 "grade or 0", CHEAP_FEATURES, judgments) # labels from people21# the teacher's own order of the candidates, ties broken by product_id22teacher = evaluate(ranked(candidates, candidates["cross_encoder"]), judgments)["ndcg@10"]23semantic = evaluate(SemanticSearch().run(queries, k=100), judgments)["ndcg@10"]24for name, scores in [("semantic", semantic), ("teacher", teacher), ("student, teacher's labels", student),25 ("lambdarank, human grades, same features", human)]:26 print(f"{name}, train ndcg@10: {scores.mean():.3f}")27# the student's score minus the human-graded ranker's, per query, tested by permutation28result = compare(student, human)29print(f"student minus the human-graded ranker: {result['mean difference']:.3f}")30print(f"student minus the human-graded ranker, permutation p: {result['permutation p']:.3f}")
python distillation-on-train.pysemantic, train ndcg@10: 0.772
teacher, train ndcg@10: 0.785
student, teacher's labels, train ndcg@10: 0.791
lambdarank, human grades, same features, train ndcg@10: 0.747
student minus the human-graded ranker: 0.044
student minus the human-graded ranker, permutation p: 0.002
これらの train クエリと特徴量では、すべての候補に付けた教師のラベルは、人手の評価よりも優れたランカーを訓練した。ただし評価のない候補は 0 として数えている。生徒のスコアは0.791であり、教師自身の候補の順序は0.785である。
人手の評価で訓練したモデルのスコアは0.747で、セマンティック検索の0.772を下回る。生徒からそれを引いた差は0.044で、置換検定の p 値は0.002である。これは train クエリでの調整のための確認であり、証拠ではない。
次に、LambdaMARTと同じ、train クエリから取り分けた検証用クエリで早期終了を行いながら、生徒を train クエリに当てはめる。そして、同じ閾値0.01で、最後の計画した比較を行う。
1"""Fits a student LightGBM regression on the teacher's scores for the train queries, then reports its gain shares, how2closely its order follows similarity and the teacher, and its planned comparison on the test queries. The prose's point3rests on that comparison and on the taus printed for each ranking."""4import numpy as np5from lightgbm import LGBMRegressor6from scipy.stats import kendalltau78from lib.metrics import evaluate9from lib.ranking import CHEAP_FEATURES, DETERMINISTIC, TREES, feature_table, fit, ranked, report_planned, stopping_split10from lib.semantic import SemanticSearch11from lib.store import load_products, load_queries, qrels1213products, train, test = load_products(), load_queries("train"), load_queries("test")14table = feature_table(products, train) # the teacher's score is the label: no human grades15print(f"train candidates the teacher labels: {len(table)}")16print(f"of them, judged by nobody: {table['grade'].isna().sum()}")17print(f"student features: {len(CHEAP_FEATURES)}")18# early stopping on the same 10 held-out train queries that LambdaMART stops on19student = fit(LGBMRegressor(**TREES, **DETERMINISTIC), table, "cross_encoder", CHEAP_FEATURES, *stopping_split(train.index))20print(f"trees kept by early stopping: {student.best_iteration_}")2122candidates, judgments = feature_table(products, test), qrels("test") # test queries: read once, to report23predicted = student.predict(candidates[CHEAP_FEATURES])24gain = student.booster_.feature_importance(importance_type="gain")25# the three features with the most gain26for name, share in sorted(zip(CHEAP_FEATURES, gain / gain.sum()), key=lambda pair: -pair[1])[:3]:27 print(f"student's share of gain, {name}: {share:.3f}")28for name in ["cross_encoder", "similarity"]: # whose order does the student follow?29 # candidates has a default index, so group.index picks this query's rows out of predicted30 tau = [kendalltau(group[name], predicted[group.index]).statistic for _, group in candidates.groupby("query_id")]31 print(f"tau between student and {name}, mean over test queries: {np.mean(tau):.3f}")32teacher = evaluate(ranked(candidates, candidates["cross_encoder"]), judgments)33print(f"teacher's own order of the candidates, test ndcg@10, not a planned comparison: {teacher['ndcg@10'].mean():.3f}")34after = evaluate(ranked(candidates, predicted), judgments)35before = evaluate(SemanticSearch().run(test, k=100), judgments)36for name, frame in [("semantic", before), ("distilled", after)]:37 print(f"{name}, test ndcg@10: {frame['ndcg@10'].mean():.3f}")38 print(f"{name}, test judged@10: {frame['judged@10'].mean():.3f}")39 # each query's top 10 is 10 places, so (1 - judged@10) times 10 counts its unjudged places40 print(f"{name}, unjudged places in the test top 10s: {round((1 - frame['judged@10']).sum() * 10)}")41report_planned("distilled", after["ndcg@10"], before["ndcg@10"])
python distilled-ranker-on-test.pytrain candidates the teacher labels: 6896
of them, judged by nobody: 1975
student features: 7
trees kept by early stopping: 202
student's share of gain, similarity: 0.812
student's share of gain, name_match: 0.126
student's share of gain, bm25: 0.055
tau between student and cross_encoder, mean over test queries: 0.466
tau between student and similarity, mean over test queries: 0.753
teacher's own order of the candidates, test ndcg@10, not a planned comparison: 0.782
semantic, test ndcg@10: 0.830
semantic, test judged@10: 0.990
semantic, unjudged places in the test top 10s: 3
distilled, test ndcg@10: 0.826
distilled, test judged@10: 0.987
distilled, unjudged places in the test top 10s: 4
distilled, mean difference: -0.004
distilled, wins, losses, ties: 10, 13, 7
distilled, permutation p: 0.720
distilled, established at 0.01: no
distilled, t-test p: 0.694
distilled, t-test agrees: yes
distilled, 99% interval: -0.034 to 0.017
distilled, interval agrees: yes
教師は6896件の train 候補すべてにラベルを付ける。そのうち1975件は誰も評価していない。早期終了で残る木は202本である。
生徒はsimilarityに強く依存しており、そのゲインの寄与率は0.812で、name_matchの0.126とbm25の0.055を上回る。test の候補では、生徒のスコアはクエリごとの平均 tau 0.753でセマンティックな類似度と一致し、教師のスコアとは0.466で一致する。つまり生徒は主にセマンティック検索の順序を学習し、名前と BM25 の一致によって教師の順序へ少し寄せられたのである。
生徒は向上を立証しない。蒸留したランカーの nDCG@10 は0.826で、セマンティック検索の0.830に対し、置換検定の p 値は0.720である(判定の表を参照)。この章が検定を計画していなかった、test 候補に対する教師自身の順序のスコアは0.782である。
Judged@10は、生徒が0.987、セマンティック検索が0.990である。また、生徒の上位 10 件には評価のない位置が4あり、セマンティック検索では3である。
実践ルール: リランカーが確かな向上をもたらすがクエリあたりのコストが高すぎる場合に蒸留する。train クエリのすべての候補に教師でラベルを付け、教師を使わずに計算できる特徴量で生徒を訓練し、保持したテスト用クエリで生徒を自分たちの検索と比較して検定する。生徒のゲインの重要度を読む。検索自身のスコアが支配的であれば、生徒は主に自分たちの検索そのものである。生徒が教師のスケールに合わせられない場合は、ペア間のマージンを用いる。このストアには、セマンティック検索を上回る教師がまだないため、コードは残すが、生徒は出荷しない。
本番環境で何が問題になるか
ライブのランカーは、買い物客が待っている間に特徴量を計算し、モデルを呼び出す。そのとき使われるコードとモデルのバージョンは、学習やテストに使ったものからずれていく可能性がある。
特徴量が学習時と提供時でずれる。 ライブシステムのトークン化が異なる場合、BM25 のスケールを変えた場合、埋め込みモデルをアップグレードした場合には、学習済みの重みは異なる数値に出会うことになるが、エラーは一切発生しない。特徴量は共通のコードパスで計算し、提供したランキングごとに記録する。
行の順序が学習に漏れ込む。 LightGBM の lambdarank は入力行の順序を出発点とし、ランキングを作る際には、何らかの方法で同順位が解消される。リランカーのスコアで並べたファイルは、そのリランカーのランキングをモデルにそのまま与えてしまう。学習のたびに、またスコアリングのたびに、候補を ID で並べるか、固定したシードでシャッフルする。
判定はプールから得られる。 WANDS が判定したのは、自らのシステムが提示した商品だけである(2章)。未判定の候補を無関連として数えると、それらのシステムが見つけられなかったものを降格させるようモデルに教え込むことになる。追加するランカーごとに judged@10 を確認し、改善を信頼する前に、未判定の商品に判定を付けてもらう。
LLM の判定者は LLM のリランカーを好む。 ある研究では、LLM の評価者から作られたランキングは、LLM ベースのリランカーを平均で 9 から 17 位も高く位置づける(Fröbe et al., 2025)。また、LLM を順位付けと判定の両方に使うことは循環であり、「LLM ベースの評価への依存がシステムのランキングをいかに歪め得るかを示している」(Clarke and Dietz, 2024)。LLM のリランカーは、この章で行っているように人手の判定で採点する。
LLM のリランカーは、1 つのモデルバージョンと 1 つのプロンプトに属する。 ダイジェスト(正確な重みを識別するハッシュ)、量子化(重みを保存する精度)、プロンプトの文言のいずれかが異なれば、すべての回答が変わり得る。位置バイアスがあるため、検索結果の順序を変えた場合も同様である。LLM の記録のマニフェストには、ダイジェストと両方のプロンプトが原文のまま記録されている。どちらかが変わるたびに再測定する。
なじみのないクエリでは成績が下がることがある。 22 種類のリランキング手法を比較した研究では、LLM のリランカーはなじみのあるクエリでは成績が良いが、「新規のクエリへの汎化能力はまちまちである」ことが分かった(Abdallah et al., 2025)。モデルの学習データより新しい商品やスタイルに関するクエリには注意する。
リランカーはどれも、買い物客の経路上に呼び出しを 1 つ加える。 予算で行ったように、クエリごとの呼び出し回数とテキスト量を数え、タイムアウトを設定し、リランカーが時間内に応答しない場合は検索の順序にフォールバックする。
実践ルール: ランキングごとに特徴量とモデルのバージョンを記録し、候補は情報を持たない順序に並べ、人手の判定で採点し、judged@10 を確認し、各リランキング段階に制限時間とフォールバックを設ける。
この部分はまだ翻訳されていないため、英語で表示している。
Decisions this chapter made
The chapter made each choice in the list below on the train queries, and only an established planned comparison on the test queries could change the store's search.
The 5 planned comparisons, each on nDCG@10 against semantic retrieval's 0.830, established below 0.01:
| Measure | LambdaMART | Cross-encoder | LLM pointwise | LLM listwise | Distilled |
|---|---|---|---|---|---|
| nDCG@10 | 0.817 | 0.813 | 0.837 | 0.819 | 0.826 |
| Mean difference | -0.012 | -0.016 | 0.008 | -0.010 | -0.004 |
| Wins / losses / ties | 14, 12, 4 | 11, 12, 7 | 11, 8, 11 | 12, 12, 6 | 10, 13, 7 |
| Permutation p | 0.260 | 0.178 | 0.241 | 0.580 | 0.720 |
| Established | no | no | no | no | no |
| t-test p | 0.278 | 0.171 | 0.244 | 0.498 | 0.694 |
| 99% interval | -0.044 to 0.011 | -0.058 to 0.006 | -0.007 to 0.026 | -0.070 to 0.016 | -0.034 to 0.017 |
| t-test agrees | yes | yes | yes | yes | yes |
| Interval agrees | yes | yes | yes | yes | yes |
- 5 planned comparisons against semantic retrieval, on human judgments, established below 0.01.
- Candidates: the union of BM25's and semantic retrieval's top 100, in
product_idorder, with score ties broken byproduct_id. - LambdaMART:
label_gain=[0, 1, 2], unjudged candidates as grade 0, 3 leaves at a learning rate of 0.1, chosen by cross-validation on train, with early stopping on held-out train queries througheval_Xandeval_y. Its gain isn't established on test, so it isn't used. - Rerank depth: semantic retrieval's top 20, fixed to match the LLM recording; on train, the cross-encoder's curve flattens between 20 and 30. No reranker's gain is established on test, so none is used.
- LLM rerankers are scored on human judgments only, and the listwise one is checked in two input orders. Its answer depends on the order shown.
- Costs are counted, not timed: calls and characters per query.
- Distillation: a LightGBM student on the cross-encoder's scores, with no model call beyond retrieval's. It learns mostly semantic retrieval's order, shows no established gain, and isn't used.
- The ranker to beat stays semantic retrieval alone, with BM25, unstemmed, as the baseline.
- What the chapter showed: no second stage earned its place; none shows an established gain.
この版における本章
第2026.11.10版2026年11月19日時点で例が通過この章の14 例件のうち14件が合格
英語版の書籍(版2026.11.10)から翻訳した。