Ranking

읽는 데 약 72분분 걸린다, 12절에 14개 예제이(가) 있음.

2026.11.10판 예제 통과 2026년 11월 19 132개 예제 중 132개 통과 변경 사항

AI로 번역함. 영어 원문을 사람이 검증함. 우리가 책을 만드는 방식

영어판 2026.11.10판을 번역한 것이다. 원문 읽기

이 부분은 아직 번역되지 않았으므로 영어로 표시된다.

By the end of this chapter you can describe each product that retrieval returns with features, train a learning-to-rank model with LightGBM on your own judgments, rerank the top of the list with a cross-encoder or an LLM, measure an LLM reranker's positional bias, set the rerank depth by counting its costs, and distil a slow reranker into a fast one. You can also decide, with a planned comparison on the test queries, whether any of them beats your retrieval.

Semantic retrieval alone is still the ranker to beat at the store, where every search is ordered by one embedding score and nothing else. It ranks products by embeddings from Snowflake/snowflake-arctic-embed-m-v1.5, and scores 0.830 nDCG@10 on the 30 test queries. No change in chapters 5 to 8 has beaten it by an established margin. This chapter asks whether a second stage can: a model trained on the store's own judgments, a reranker that reads the query and the product together, or an LLM. Chapter 1's cross-encoder reranked the hybrid run, not semantic retrieval, and never saw the store's judgments.

So this chapter tries all three on semantic retrieval's top results, each scored against the store's judgments. Reranking reorders the top of retrieval's list with a slower model. The store's rerankers are recorded: each ran once and saved its outputs, which the examples read.

The store's LLM is the open model gemma4:12b-it-q4_K_M, licensed Apache-2.0. It ran locally with chapter 4's settings, which make it take its likeliest next word every time, plus thinking off, so it answers without writing out reasoning, and a JSON schema, a machine-readable description of the answer's shape. recordings/llm_rerank.py reranked semantic retrieval's top 20 for the test queries only, so nothing about the LLM rerankers can be tuned. Every score uses the human judgments, because in one study LLM judges ranked LLM rerankers too high (What goes wrong in production). These results describe one model and two prompts, so treat them as the store's, not as a verdict on LLM rerankers.

The chapter fixed its 5 planned comparisons on the test queries before it read any test result. Each sets one method against semantic retrieval's run on nDCG@10:

  1. LambdaMART, a learned ranker trained on the train queries' judgments, ordering each query's candidate products
  2. the cross-encoder reranking semantic retrieval's top 20
  3. the LLM scoring each of the top 20 on its own
  4. the LLM ordering the top 20 in one prompt
  5. a small distilled ranker trained on the cross-encoder's scores

The bar is the book's rule: 5 planned comparisons split its 0.05, so each needs a paired permutation test p-value below 0.01. The paired t-test and a 99% bootstrap interval stand beside it.

특징으로 각 후보 설명하기

마지막으로 변경된 판: 2026.11.10판

이 장의 계획된 비교 가운데 첫 번째는 학습된 랭커이며, 이 랭커는 텍스트를 읽지 않는다. 학습된 랭커는 쿼리와 상품의 각 쌍을 숫자 목록, 즉 특징(feature)으로 점수화하고, 판정으로부터 그 특징들에 얼마나 가중치를 둘지 학습한다. 쿼리 의존 특징은 BM25 점수처럼 쿼리와 상품을 함께 설명한다. 쿼리 독립 특징은 몇 명의 구매자가 평가했는지처럼 상품만을 설명한다. 웹 검색용 특징 집합은 두 종류를 섞는다. 순위 학습의 벤치마크인 LETOR 4.0은 "BM25 of title", "PageRank", "Number of slash in URL" 등 46개의 특징을 갖는다(Qin and Liu, 2013). 랭커는 더 값싼 검색 단계가 반환한 상품인 후보만 점수화하므로, 특징의 계산 비용이 커도 된다.

상점의 각 쿼리에 대한 후보는 교차 인코더의 기록에 점수가 매겨진 상품이며, BM25 상위 100개와 의미 검색 상위 100개의 합집합이다. lib.ranking.feature_table는 각 후보에 8개의 특징을 부여한다.

  • bm25: lib.lexical에서 얻은 BM25 점수이며, 이 책의 어간 추출을 하지 않은 기준선이다. 쿼리와 공통된 용어가 없는 상품은 0이다.
  • similarity: 쿼리 임베딩과 상품 임베딩의 코사인 유사도
  • cross_encoder: 기록된 교차 인코더 점수
  • name_match와 attribute_match: 5장과 같은 방식으로 토큰화한 쿼리 용어 가운데 상품 이름에서 발견된 비율과 상품의 속성에서 발견된 비율
  • rating_count, average_rating, review_count: 상점의 데이터가 각 상품에 제공하는 값

카탈로그 상품의 2632에는 평점이 없으며, 이 상품들의 평점 특징 세 개는 결측으로 남는다. 표에는 각 후보의 사람 판정 등급도 들어 있으며, 아무도 판정하지 않았다면 빈칸으로 둔다. 학습 쿼리에 대한 표를 만들고, 쿼리 하나를 살펴본 다음, 등급별로 각 특징을 비교한다.

ranking-features-by-grade.py Python 3.12.15 · bm25s 0.3.16 · lightgbm 4.7.0 · numpy 2.5.5 · pandas 3.1.1 · pystemmer 3.1.0 · scikit-learn 1.9.1 · scipy 1.18.1 통과
"""Builds the feature table for the train queries, shows one query's candidates, and compares each feature's mean acrossthe human grades. The prose's point rests on the mean cross_encoder line for each grade and the mean of each feature by grade."""import pandas as pdfrom lib.ranking import FEATURES, feature_tablefrom lib.store import load_products, load_queriesproducts, queries = load_products(), load_queries("train")      # train queries: features are design worktable = feature_table(products, queries)print(f"features: {len(FEATURES)}")print(f"train candidates: {len(table)}")print(f"train candidates per query, median: {table.groupby('query_id').size().median():.0f}")print(f"share of train candidates nobody judged: {table['grade'].isna().mean():.3f}")query_id = queries.index[queries["query"] == "laundry basket with wheels"][0]one = table[table["query_id"] == query_id].sort_values("cross_encoder", ascending=False)one = one.assign(name=products.loc[one["product_id"], "name"].str[:38].to_numpy())pd.set_option("display.width", 200)print(f"\ncandidates for laundry basket with wheels: {len(one)}")print(one[["name", "bm25", "similarity", "cross_encoder", "name_match", "attribute_match", "rating_count", "grade"]]      .head(6).round(3).to_string(index=False))print(f"top 6 by cross-encoder, exact: {(one['grade'].head(6) == 2).sum()}")# grades 2, 1 and 0 are exact, partial and irrelevant; a missing grade means nobody judged the pairlabel = table["grade"].map({2: "exact", 1: "partial", 0: "irrelevant"}).fillna("unjudged")means = table[FEATURES].groupby(label).mean().loc[["exact", "partial", "irrelevant", "unjudged"]]print("\nmean of each feature by grade, train candidates:")print(means.round(3).to_string())for grade in means.index:    print(f"mean cross_encoder, {grade}: {means.at[grade, 'cross_encoder']:.3f}")relevance = ["bm25", "similarity", "cross_encoder", "name_match"]nearer = [name for name in relevance if abs(means.at["unjudged", name] - means.at["irrelevant", name])          < abs(means.at["unjudged", name] - means.at["partial", name])]print(f"relevance features where unjudged sits nearer irrelevant than partial: {len(nearer)} of {len(relevance)}")medians = table[["rating_count", "review_count"]].groupby(label).median().loc[means.index]  # counts are skewedfor grade in ["exact", "partial", "irrelevant"]:    print(f"rating_count, {grade}, mean: {means.at[grade, 'rating_count']:.3f}")    print(f"rating_count, {grade}, median: {medians.at[grade, 'rating_count']:.1f}")print(f"products with no rating: {products['rating_count'].isna().sum()}")
출력python ranking-features-by-grade.py
features: 8
train candidates: 6896
train candidates per query, median: 131
share of train candidates nobody judged: 0.286

candidates for laundry basket with wheels: 125
                                  name  bm25  similarity  cross_encoder  name_match  attribute_match  rating_count  grade
       laundry sorter basket on wheels 8.229       0.597          3.433       1.000              0.0           NaN    2.0
                vintage laundry basket 6.270       0.572          2.995       0.667              0.0         638.0    2.0
         rolling wicker laundry basket 5.494       0.588          2.974       0.667              0.0         110.0    2.0
aluminum frame laundry basket can be f 4.739       0.524          2.941       0.667              0.0          14.0    2.0
    rebrilliant rolling laundry hamper 5.537       0.558          2.804       0.333              0.0          17.0    2.0
4-bag laundry hamper sorter cart baske 6.012       0.543          2.739       0.667              0.0           NaN    2.0
top 6 by cross-encoder, exact: 6

mean of each feature by grade, train candidates:
             bm25  similarity  cross_encoder  name_match  attribute_match  rating_count  average_rating  review_count
grade
exact       4.317       0.517          2.191       0.562            0.024       466.879           4.537       333.589
partial     3.936       0.454          1.513       0.461            0.030       385.454           4.539       288.779
irrelevant  2.760       0.360          1.159       0.404            0.006       246.781           4.552       159.052
unjudged    2.323       0.369          1.050       0.140            0.018       339.904           4.520       237.027
mean cross_encoder, exact: 2.191
mean cross_encoder, partial: 1.513
mean cross_encoder, irrelevant: 1.159
mean cross_encoder, unjudged: 1.050
relevance features where unjudged sits nearer irrelevant than partial: 4 of 4
rating_count, exact, mean: 466.879
rating_count, exact, median: 43.0
rating_count, partial, mean: 385.454
rating_count, partial, median: 31.0
rating_count, irrelevant, mean: 246.781
rating_count, irrelevant, median: 35.5
products with no rating: 2632

학습 쿼리의 후보는 6896개이고 쿼리당 중앙값은 131개이며, 그중 0.286는 아무도 판정하지 않았다.

laundry basket with wheels의 후보는 125개이다. 교차 인코더 점수 상위 6개에는 exact 상품이 6개 들어 있다. 이 상품들의 attribute_match는 0인데, 속성에는 색상, 소재, 크기가 들어 있을 뿐 wheels 같은 단어는 들어 있지 않기 때문이다.

학습 후보 전체에서 교차 인코더 점수의 평균은 exact 상품의 2.191에서 partial 상품의 1.513, irrelevant 상품의 1.159로 떨어지며, bm25, similarity, name_match도 같은 방식으로 떨어진다. average_rating와 attribute_match는 거의 변하지 않는다.

rating_count의 평균은 등급에 따라 떨어지지만 중앙값은 그렇지 않다. 평가가 매우 많은 소수의 상품이 평균을 결정하기 때문이다.

등급 평점 수 평균 평점 수 중앙값
exact 466.879 43.0
partial 385.454 31.0
irrelevant 246.781 35.5

판정되지 않은 후보의 교차 인코더 점수 평균은 1.050이며, 등급에 따라 떨어지는 4 of 4개의 관련성 특징에서는 이들의 평균이 partial 상품보다 irrelevant 상품의 평균에 더 가깝다. 이는 판정되지 않은 후보가 모델에 무엇을 가르쳐야 하는지를 정하는 데 중요하다.

작업 규칙: 이미 계산하고 있는 특징에서 시작한다. 각 검색기의 점수, 있다면 재순위기의 점수, 이름 같은 필드의 일치, 인기 신호가 그 예이다. 학습하기 전에 학습 쿼리에서 등급별로 각 특징을 비교하되, 점수는 평균으로, 치우친 개수는 중앙값으로 비교한다. 등급에 따라 변하지 않는 특징은 모델에 가르칠 것이 거의 없다. 학습할 때와 서비스할 때 같은 코드로 특징을 계산한다.

이 부분은 아직 번역되지 않았으므로 영어로 표시된다.

Learn a ranking from pairs of products

마지막으로 변경된 판: 2026.11.10판

Learning to rank trains a model on judged queries to order candidates by their features. Its methods fall into three families, pointwise, pairwise and listwise (Liu, 2009). A pointwise model predicts each product's grade on its own, as a regression. A pairwise model learns which of two products should rank higher. A listwise model scores a whole ranking. Liu's 2009 comparison on the LETOR benchmark favoured the listwise approach, though tentatively. The obstacle for all three is the metric: nDCG changes only when two products swap places, so ranking metrics "are everywhere either discontinuous or flat, so gradient descent appears to be problematic" (Burges, 2010).

RankNet, a pairwise method, works around it with a smooth loss on pairs. For two products where i should rank above j, its gradient is large when the model scores them the wrong way round or nearly level, and small when it already has them well apart. LambdaRank keeps that gradient and multiplies it by how much nDCG would change if the two products swapped places (Burges, 2010, section 4.1). Burges writes the gradient with a minus sign, as the slope of a cost; here a positive λ pushes i up:

λ_ij    = σ / (1 + exp(σ · (s_i − s_j))) · |ΔnDCG_ij|      s: the model's scores, σ = 1
ΔnDCG_ij = (g_i − g_j) · (1 / log2(1 + r_i) − 1 / log2(1 + r_j)) / ideal DCG      g: grades, r: positions

A product's total is its λ summed over the pairs where it should rank higher, minus those where it should rank lower. Burges (2010) also reads the λ's as forces, pushing products up and down the list.

LambdaMART uses LambdaRank's λ as the gradients for gradient-boosted trees, which add small decision trees one at a time, each fitted to the gradients of the trees before it: "LambdaMART is the boosted tree version of LambdaRank, which is based on RankNet". An ensemble of LambdaMART rankers won Track 1 of the Learning to Rank Challenge that Yahoo ran in 2010 (Burges, 2010).

One train query, outdoor waterproof chest, shows the arithmetic. Let semantic similarity be the model's scores so far, compute every pair's λ in its top 10 by hand, check each |ΔnDCG| by swapping the pair and recomputing nDCG with lib.metrics, and add up the forces:

lambdas-for-one-query.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 통과
"""Works out LambdaRank's λ for every pair in one train query's top 10 by hand, and checks each pair's |ΔnDCG| againstswapping the pair in the metric. The prose's point rests on the strongest push up and down, and on the count of pairswhose formula matches swapping."""import mathfrom itertools import combinationsfrom lib.metrics import dcg_at_k, ndcg_at_kfrom lib.semantic import SemanticSearchfrom lib.store import load_queries, qrelsqueries, judgments = load_queries("train"), qrels("train")query_id = queries.index[queries["query"] == "outdoor waterproof chest"][0]judged = judgments[query_id]scores = SemanticSearch().search(query_id, k=10)       # the model so far: semantic similarityranking, similarity = list(scores.index), scores.to_numpy()# an unjudged product counts as grade 0, as it does in the metricgrades = [judged.get(product_id, 0) for product_id in ranking]# the ideal DCG takes every judged product, as lib.metrics does; sigma is 1, LightGBM's default sigmoidideal, sigma = dcg_at_k(sorted(judged.values(), reverse=True), 10), 1.0print("grades from the top:", " ".join(map(str, grades)))force, checks, pulls = [0.0] * len(ranking), [], []for i, j in combinations(range(len(ranking)), 2):      # i is above j    if grades[i] == grades[j]:        continue    better, worse = (i, j) if grades[i] > grades[j] else (j, i)    ranknet = 1 / (1 + math.exp(sigma * (similarity[better] - similarity[worse])))      # RankNet's pull    # i and j count from 0, so position p is i + 1 and its discount is 1 / log2(p + 1), which is 1 / log2(i + 2)    delta = abs((grades[i] - grades[j]) * (1 / math.log2(i + 2) - 1 / math.log2(j + 2))) / ideal    swapped = ranking.copy()    swapped[i], swapped[j] = swapped[j], swapped[i]    checks.append(math.isclose(delta, abs(ndcg_at_k(swapped, judged, 10) - ndcg_at_k(ranking, judged, 10))))    pulls.append(ranknet)    pair_lambda = sigma * ranknet * delta                         # LambdaRank: scaled by |delta nDCG|    force[better], force[worse] = force[better] + pair_lambda, force[worse] - pair_lambdaprint(f"pairs with different grades: {len(checks)}")print(f"pairs where delta ndcg by formula matches swapping the pair: {sum(checks)}")for position, push in enumerate(force, start=1):    print(f"force at position {position}: {push:+.3f}")# list indexes count from 0 and positions from 1, hence the + 1print(f"strongest push up, position: {force.index(max(force)) + 1}")print(f"strongest push down, position: {force.index(min(force)) + 1}")print(f"ranknet's pull, lowest: {min(pulls):.3f}\nranknet's pull, highest: {max(pulls):.3f}")
출력python lambdas-for-one-query.py
grades from the top: 1 2 2 0 1 2 0 2 0 0
pairs with different grades: 32
pairs where delta ndcg by formula matches swapping the pair: 32
force at position 1: +0.013
force at position 2: +0.159
force at position 3: +0.104
force at position 4: -0.083
force at position 5: -0.010
force at position 6: +0.063
force at position 7: -0.092
force at position 8: +0.063
force at position 9: -0.105
force at position 10: -0.111
strongest push up, position: 2
strongest push down, position: 10
ranknet's pull, lowest: 0.472
ranknet's pull, highest: 0.525

The formula's |ΔnDCG| matches the change from swapping the pair for 32 of the 32 pairs in the top 10 with different grades. The top 10's grades, from the top, are 1 2 2 0 1 2 0 2 0 0; an unjudged product counts as 0, as it does in the metric.

The strongest push up goes to position 2, an exact product, with +0.159; the strongest push down goes to position 10, an irrelevant product, with -0.111. The partial product at position 1 gets a net push of +0.013: the irrelevant products below push it up, and the exact ones push it down.

RankNet's factor in λ stays between 0.472 and 0.525, because the similarities lie close together, so |ΔnDCG| decides almost everything.

That example simplifies. It takes the ideal DCG from every judged product, as lib.metrics does, and looks at ten products. LightGBM takes the ideal from the query's training rows, and at 4.7.0 two defaults change the λ. lambdarank_truncation_level is 30, which limits training to the top 30 results: only pairs with a product in the current top 30 get a λ. lambdarank_norm is true, which normalizes the lambdas across queries; set it to false to get the original lambdarank algorithm (Parameters). Normalizing, as LightGBM's source does it, divides each pair's |ΔnDCG| by the gap between its scores plus 0.01, then rescales each query's λ (rank_objective.hpp). To check the formula against the library, write it as a custom LightGBM objective, grow five small trees on this query's candidates, and grow them again with LightGBM's lambdarank, with and without those defaults:

lambdas-match-lightgbm.py Python 3.12.15 · bm25s 0.3.16 · lightgbm 4.7.0 · numpy 2.5.5 · pandas 3.1.1 · pystemmer 3.1.0 · scikit-learn 1.9.1 · scipy 1.18.1 통과
"""Writes LambdaRank's gradients by hand as a custom LightGBM objective, then checks them against LightGBM's own lambdarankon one query, with and without its defaults. The prose's point rests on the two printed differences: a rounding-sized gapwithout the defaults, and a large one at them."""import numpy as npfrom lightgbm import LGBMRankerfrom lib.ranking import DETERMINISTIC, FEATURES, feature_tablefrom lib.store import load_products, load_queriesqueries = load_queries("train")one = queries[queries["query"] == "outdoor waterproof chest"]rows = feature_table(load_products(), one)                  # one query's candidates, in product_id order# unjudged candidates count as grade 0, as in the metricgrades = rows["grade"].fillna(0).astype(int).to_numpy()SIGMA = 1.0                                                   # LightGBM's default sigmoiddef lambdas(y_true, y_pred, weight, group):    """The chapter's lambda for every pair with different grades, as gradients and second derivatives."""    rank = np.empty(len(y_pred), dtype=int)    rank[np.argsort(-y_pred, kind="stable")] = np.arange(len(y_pred))   # LightGBM's stable sort on score    # rank counts from 0, so the product at position rank + 1 has the discount 1 / log2(rank + 2)    discount, ideal = 1 / np.log2(rank + 2), np.sum(np.sort(y_true)[::-1] / np.log2(np.arange(len(y_true)) + 2))    gradient, hessian = np.zeros(len(y_pred)), np.zeros(len(y_pred))    for i in range(len(y_true)):        for j in np.flatnonzero(y_true < y_true[i]):                      # i should rank above j            delta = (y_true[i] - y_true[j]) * abs(discount[i] - discount[j]) / ideal            pull = 1 / (1 + np.exp(SIGMA * (y_pred[i] - y_pred[j])))            gradient[i], gradient[j] = gradient[i] - SIGMA * pull * delta, gradient[j] + SIGMA * pull * delta            hessian[[i, j]] += SIGMA ** 2 * pull * (1 - pull) * delta    return gradient, hessian# five small trees; min_child_samples is 5 where LightGBM's default is 20, so the trees can split this query's candidatestree = {"n_estimators": 5, "num_leaves": 3, "min_child_samples": 5, **DETERMINISTIC}# turns off normalization and sets truncation to every candidate, so every pair gets a λ, as the formula has itplain = {"lambdarank_norm": False, "lambdarank_truncation_level": len(rows)}   # LightGBM's lambdas, untouchedmodels = {"by hand": LGBMRanker(objective=lambdas, **tree),          "lightgbm, no normalization or truncation": LGBMRanker(objective="lambdarank", label_gain=[0, 1, 2], **plain, **tree),          "lightgbm at its defaults": LGBMRanker(objective="lambdarank", label_gain=[0, 1, 2], **tree)}# raw scores: the values the gradients act on, before any sigmoidscores = {name: model.fit(rows[FEATURES], grades, group=[len(rows)]).predict(rows[FEATURES], raw_score=True)          for name, model in models.items()}print(f"candidates: {len(rows)}")for name in list(models)[1:]:    difference = np.abs(scores[name] - scores["by hand"]).max()    print(f"{name}, largest difference from the lambdas by hand after 5 trees: {difference:.6f}")
출력python lambdas-match-lightgbm.py
candidates: 180
lightgbm, no normalization or truncation, largest difference from the lambdas by hand after 5 trees: 0.000006
lightgbm at its defaults, largest difference from the lambdas by hand after 5 trees: 0.595667

Over the query's 180 candidates, LightGBM without normalization or truncation scores every product within 0.000006 of the custom objective's scores. The gap is rounding, since LightGBM reads its sigmoid from a precomputed table (rank_objective.hpp). At its defaults, scores differ by up to 0.595667.

Working rule: Train rankers with a gradient that weighs each pair by the metric you report, as LambdaMART does, rather than regressing on grades. Know which defaults of your library change that gradient, and check on train queries that it beats a pointwise model on your data, because the families' order in a benchmark doesn't transfer by itself.

이 부분은 아직 번역되지 않았으므로 영어로 표시된다.

Train LambdaMART with LightGBM

마지막으로 변경된 판: 2026.11.8판

LightGBM trains LambdaMART with LGBMRanker and objective="lambdarank". Four settings need care, and each is documented for 4.7.0:

  • Rows are grouped by query. "data should be ordered by the query", and group holds the number of rows of each query in that order, not query IDs, so its sizes sum to the number of rows (sum(group) = n_samples; Parameters; LGBMRanker).
  • Gains are exponential by default. label_gain defaults to 0,1,3,7,15,31,63,..., so "the gain of label 2 is 3", and every label must be smaller than the number of gains (Parameters). That is Burges's DCG, with gain 2^grade − 1. The book's nDCG uses linear gains, so set label_gain=[0, 1, 2].
  • nDCG is evaluated at positions 1 to 5 by default. Set eval_at=[10] to stop on nDCG@10, and report the final score with lib.metrics anyway.
  • Validation data goes in eval_X and eval_y. In 4.7.0, eval_set is "Deprecated since version 4.7.0", and eval_X and eval_y replace it (LGBMRanker; release notes). The examples pass eval_X, eval_y and eval_group.

Early stopping scores held-out train queries after every tree, and keeps the trees up to the best score once 30 more bring no gain. So that training repeats exactly, the examples use one thread, a fixed seed and deterministic=True, which "should ensure the stable results when using the same data" (Parameters), with force_row_wise=True, which the same entry recommends, to avoid potential instability from numerical issues. lib.ranking holds these settings, and its fit applies the grouping and early stopping.

Row order matters too. Before the first tree every score is 0, so LightGBM takes the rows' order as the starting ranking for the first λ, and equal scores need a tie-break. The cross-encoder's recording lists each query's candidates best first, which would hand the model the cross-encoder's ranking before it learned anything. So feature_table sorts candidates by product_id, and lib.ranking.ranked breaks ties by product_id.

Choose the tree size by cross-validation on the train queries: split them into five folds, rank each fold with a model fitted on three others and stopped early on the fifth, and score every query once. Try eight settings, from 3 to 31 leaves at two learning rates, counting unjudged candidates as grade 0 as the metric does:

lambdamart-tree-size-on-train.py Python 3.12.15 · bm25s 0.3.16 · lightgbm 4.7.0 · numpy 2.5.5 · pandas 3.1.1 · pystemmer 3.1.0 · scikit-learn 1.9.1 · scipy 1.18.1 통과
"""Cross-validates LambdaMART on the train queries over eight tree sizes and learning rates, and names the best. The prose'spoint rests on the best setting's score, and on the score beside it for LightGBM's default of 31 leaves at 0.1."""from lightgbm import LGBMRankerfrom lib.metrics import evaluatefrom lib.ranking import DETERMINISTIC, FEATURES, LAMBDARANK, TREE_GRID, TREES, cross_validate, feature_tablefrom lib.semantic import SemanticSearchfrom lib.store import load_products, load_queries, qrelsqueries, judgments = load_queries("train"), qrels("train")          # tuning: train queries onlycandidates = feature_table(load_products(), queries)candidates["grade or 0"] = candidates["grade"].fillna(0).astype(int)  # unjudged counts as irrelevant, as in nDCGprint(f"train queries: {len(queries)}")print(f"semantic, train ndcg@10: {evaluate(SemanticSearch().run(queries, k=100), judgments)['ndcg@10'].mean():.3f}")results = {}for setting in TREE_GRID:                                            # LightGBM's default is 31 leaves at 0.1    name = f"{setting['num_leaves']} leaves, learning rate {setting['learning_rate']}"    # up to 500 trees; early stopping inside cross_validate keeps fewer    model = lambda: LGBMRanker(**LAMBDARANK, n_estimators=500, **setting, **DETERMINISTIC)    # one nDCG@10 per train query, each ranked by a model that never saw that query    results[name] = cross_validate(model, candidates, candidates, "grade or 0", FEATURES, judgments)    print(f"{name}, cross-validated ndcg@10: {results[name].mean():.3f}")    # the trees early stopping kept in each fold    print(f"{name}, trees kept per fold: {', '.join(map(str, results[name].attrs['trees']))}")best = max(results, key=lambda name: results[name].mean())print(f"settings tried: {len(results)}")print(f"best setting: {best}")print(f"lib.ranking.TREES holds it: {'yes' if best == f'{TREES['num_leaves']} leaves, learning rate {TREES['learning_rate']}' else 'no'}")
출력python lambdamart-tree-size-on-train.py
train queries: 50
semantic, train ndcg@10: 0.772
3 leaves, learning rate 0.05, cross-validated ndcg@10: 0.780
3 leaves, learning rate 0.05, trees kept per fold: 18, 27, 52, 20, 85
3 leaves, learning rate 0.1, cross-validated ndcg@10: 0.798
3 leaves, learning rate 0.1, trees kept per fold: 99, 20, 8, 43, 38
7 leaves, learning rate 0.05, cross-validated ndcg@10: 0.744
7 leaves, learning rate 0.05, trees kept per fold: 2, 28, 30, 39, 55
7 leaves, learning rate 0.1, cross-validated ndcg@10: 0.772
7 leaves, learning rate 0.1, trees kept per fold: 80, 12, 13, 7, 20
15 leaves, learning rate 0.05, cross-validated ndcg@10: 0.720
15 leaves, learning rate 0.05, trees kept per fold: 4, 37, 175, 44, 120
15 leaves, learning rate 0.1, cross-validated ndcg@10: 0.728
15 leaves, learning rate 0.1, trees kept per fold: 19, 68, 136, 48, 53
31 leaves, learning rate 0.05, cross-validated ndcg@10: 0.679
31 leaves, learning rate 0.05, trees kept per fold: 15, 8, 91, 8, 22
31 leaves, learning rate 0.1, cross-validated ndcg@10: 0.737
31 leaves, learning rate 0.1, trees kept per fold: 11, 75, 166, 96, 58
settings tried: 8
best setting: 3 leaves, learning rate 0.1
lib.ranking.TREES holds it: yes

Of 8 settings, the best is 3 leaves, learning rate 0.1, at 0.798; LightGBM's default of 31 leaves at 0.1 (LGBMRanker) scores 0.737. The best of all the settings flatters itself, so expect less on held-out queries.

One choice depends on the store's judgments. Nobody judged a share of 0.286 of the train candidates. A model can count them as grade 0, or learn from judged candidates only; unjudged isn't irrelevant, but the model will rank them at query time either way. Compare both by cross-validation at the chosen tree size, with a pointwise regression on the grades beside them:

lambdamart-choices-on-train.py Python 3.12.15 · bm25s 0.3.16 · lightgbm 4.7.0 · numpy 2.5.5 · pandas 3.1.1 · pystemmer 3.1.0 · scikit-learn 1.9.1 · scipy 1.18.1 통과
"""Compares three ways of training on the train queries, each by cross-validation: LambdaRank with unjudged candidates as 0,LambdaRank on judged candidates only, and pointwise regression. The prose's point rests on the differences and thepermutation p-values printed under each setup."""from lightgbm import LGBMRanker, LGBMRegressorfrom lib.metrics import evaluatefrom lib.ranking import DETERMINISTIC, FEATURES, LAMBDARANK, TREES, cross_validate, feature_tablefrom lib.semantic import SemanticSearchfrom lib.significance import comparefrom lib.store import load_products, load_queries, qrelsqueries, judgments = load_queries("train"), qrels("train")          # tuning: train queries onlycandidates = feature_table(load_products(), queries)candidates["grade or 0"] = candidates["grade"].fillna(0).astype(int)  # unjudged counts as irrelevant# the rows the model learns from when unjudged candidates are left out; it still ranks every candidate at query timejudged_only = candidates.dropna(subset=["grade"]).astype({"grade": int})setups = {                                                          # (model, rows it learns from, label)    "lambdarank, unjudged as 0": (lambda: LGBMRanker(**LAMBDARANK, **TREES, **DETERMINISTIC), candidates, "grade or 0"),    "lambdarank, unjudged left out": (lambda: LGBMRanker(**LAMBDARANK, **TREES, **DETERMINISTIC), judged_only, "grade"),    "pointwise regression, unjudged as 0": (lambda: LGBMRegressor(**TREES, **DETERMINISTIC), candidates, "grade or 0"),}semantic = evaluate(SemanticSearch().run(queries, k=100), judgments)["ndcg@10"]print(f"semantic, train ndcg@10: {semantic.mean():.3f}")scores = {name: cross_validate(model, rows, candidates, label, FEATURES, judgments)          for name, (model, rows, label) in setups.items()}chosen = scores["lambdarank, unjudged as 0"]for name, per_query in scores.items():    print(f"{name}, cross-validated ndcg@10: {per_query.mean():.3f}")    if name != "lambdarank, unjudged as 0":        # the mean difference is the first argument minus the second, tested by permutation over the queries        result = compare(per_query, chosen)        print(f"{name}, minus the first: {result['mean difference']:.3f}")        print(f"{name}, minus the first, permutation p: {result['permutation p']:.3f}")result = compare(chosen, semantic)print(f"first setup minus semantic: {result['mean difference']:.3f}")print(f"first setup minus semantic, permutation p: {result['permutation p']:.3f}")
출력python lambdamart-choices-on-train.py
semantic, train ndcg@10: 0.772
lambdarank, unjudged as 0, cross-validated ndcg@10: 0.798
lambdarank, unjudged left out, cross-validated ndcg@10: 0.733
lambdarank, unjudged left out, minus the first: -0.064
lambdarank, unjudged left out, minus the first, permutation p: 0.005
pointwise regression, unjudged as 0, cross-validated ndcg@10: 0.769
pointwise regression, unjudged as 0, minus the first: -0.029
pointwise regression, unjudged as 0, minus the first, permutation p: 0.007
first setup minus semantic: 0.026
first setup minus semantic, permutation p: 0.242

Counting unjudged candidates as 0 scores higher than leaving them out, 0.798 against 0.733, a difference of -0.064 with a permutation p-value of 0.005. A model that never sees an unjudged candidate in training still has to rank them, and the features say they look like irrelevant products.

Pointwise regression scores 0.769, a difference of -0.029 (p = 0.007), so on the store's train queries a tuning check against a pointwise model favours LambdaMART.

Against semantic retrieval's 0.772, the chosen setup differs by 0.026, with a permutation p-value of 0.242: even on the queries that chose it, it shows no gain the train queries can tell from chance.

Working rule: Sort rows by query and pass group, set label_gain to the gains your metric uses, stop early on held-out train queries through eval_X and eval_y, and fix the seed and thread count. Order each query's candidates by product ID, as chapter 3 advises, and break score ties the same way. Count unjudged candidates as grade 0 unless cross-validation on train queries says otherwise. With few judged queries, try small trees.

이 부분은 아직 번역되지 않았으므로 영어로 표시된다.

Report LambdaMART on the test queries

마지막으로 변경된 판: 2026.11.10판

A model tuned on train queries is reported once on the test queries, by a comparison planned before looking. Feature importance shows what a model learned: LightGBM's "gain" importance is the "total gains of splits which use the feature" (Booster), so a feature with no gain plays no part in the score.

The first planned comparison uses the setup chosen on train: LambdaMART on all the features, with 3 leaves at a learning rate of 0.1, unjudged candidates as grade 0, and candidates in product_id order. It fits on 40 train queries, stops early on 10 others, and ranks each test query's candidates. A gain needs a permutation p-value below 0.01:

lambdamart-on-test.py Python 3.12.15 · bm25s 0.3.16 · lightgbm 4.7.0 · numpy 2.5.5 · pandas 3.1.1 · pystemmer 3.1.0 · scikit-learn 1.9.1 · scipy 1.18.1 통과
"""Fits LambdaMART on the train queries with the chosen setup, reports its early stopping and gain shares, then reports itsplanned comparison on the test queries against semantic retrieval. The prose's point rests on that comparison's permutationp-value, printed last, against the threshold printed first."""from lightgbm import LGBMRankerfrom lib.metrics import evaluatefrom lib.ranking import (DETERMINISTIC, FEATURES, LAMBDARANK, PLANNED_TEST_COMPARISONS, THRESHOLD, TREES, feature_table,                         fit, ranked, report_planned, stopping_split)from lib.semantic import SemanticSearchfrom lib.store import load_products, load_queries, qrelsproducts, train, test = load_products(), load_queries("train"), load_queries("test")print(f"planned test comparisons: {len(PLANNED_TEST_COMPARISONS)}")print(f"p-value threshold: {THRESHOLD:.2f}")table = feature_table(products, train)table["grade or 0"] = table["grade"].fillna(0).astype(int)            # the setup chosen on train# 10 train queries are held out, at random with a fixed seed, to stop onfit_ids, stop_ids = stopping_split(train.index)model = fit(LGBMRanker(**LAMBDARANK, **TREES, **DETERMINISTIC), table, "grade or 0", FEATURES, fit_ids, stop_ids)print(f"queries fitted: {len(fit_ids)}\nqueries held out for stopping: {len(stop_ids)}")curve = model.evals_result_["valid_0"]["ndcg@10"]                     # LightGBM's nDCG@10, one value per treeprint(f"trees tried: {len(curve)}\ntrees kept by early stopping: {model.best_iteration_}")print(f"stopping queries' ndcg@10, first tree: {curve[0]:.3f}")# best_iteration_ counts trees from 1 and the curve from 0, so the last tree kept is at index best_iteration_ - 1print(f"stopping queries' ndcg@10, trees kept: {curve[model.best_iteration_ - 1]:.3f}")gain = model.booster_.feature_importance(importance_type="gain")      # total gain of the splits on each feature# a feature with no gain was never split on, so it plays no part in the scoreprint(f"features the trees use: {(gain > 0).sum()} of {len(FEATURES)}")for name, share in sorted(zip(FEATURES, gain / gain.sum()), key=lambda pair: -pair[1]):    print(f"share of gain, {name}: {share:.3f}")candidates, judgments = feature_table(products, test), qrels("test")   # test queries: read once, to reportafter = evaluate(ranked(candidates, model.predict(candidates[FEATURES])), judgments)before = evaluate(SemanticSearch().run(test, k=100), judgments)for name, frame in [("semantic", before), ("lambdamart", after)]:    print(f"{name}, test ndcg@10: {frame['ndcg@10'].mean():.3f}")    print(f"{name}, test judged@10: {frame['judged@10'].mean():.3f}")    # each query's top 10 is 10 places, so (1 - judged@10) times 10 counts its unjudged places    print(f"{name}, unjudged places in the test top 10s: {round((1 - frame['judged@10']).sum() * 10)}")report_planned("lambdamart", after["ndcg@10"], before["ndcg@10"])
출력python lambdamart-on-test.py
planned test comparisons: 5
p-value threshold: 0.01
queries fitted: 40
queries held out for stopping: 10
trees tried: 243
trees kept by early stopping: 213
stopping queries' ndcg@10, first tree: 0.568
stopping queries' ndcg@10, trees kept: 0.794
features the trees use: 8 of 8
share of gain, cross_encoder: 0.641
share of gain, similarity: 0.172
share of gain, name_match: 0.097
share of gain, bm25: 0.043
share of gain, review_count: 0.021
share of gain, rating_count: 0.021
share of gain, attribute_match: 0.002
share of gain, average_rating: 0.001
semantic, test ndcg@10: 0.830
semantic, test judged@10: 0.990
semantic, unjudged places in the test top 10s: 3
lambdamart, test ndcg@10: 0.817
lambdamart, test judged@10: 0.990
lambdamart, unjudged places in the test top 10s: 3
lambdamart, mean difference: -0.012
lambdamart, wins, losses, ties: 14, 12, 4
lambdamart, permutation p: 0.260
lambdamart, established at 0.01: no
lambdamart, t-test p: 0.278
lambdamart, t-test agrees: yes
lambdamart, 99% interval: -0.044 to 0.011
lambdamart, interval agrees: yes

Of 243 trees tried, early stopping keeps 213. On the queries held out for stopping, LightGBM's own nDCG@10, whose ideal comes from their candidates alone, rises from 0.568 after the first tree to 0.794 at the trees kept.

The trees use 8 of 8 features, but they lean on the cross-encoder, which takes a share of 0.641 of the gain, ahead of similarity at 0.172, name_match at 0.097 and bm25 at 0.043. So serving the model means a cross-encoder call for every candidate.

LambdaMART doesn't establish a gain on the test queries: it scores 0.817 nDCG@10, against semantic retrieval's 0.830, at a permutation p-value of 0.260 (all five comparisons). Judged@10 is 0.990 for LambdaMART and 0.990 for semantic retrieval.

Working rule: Adopt a learned ranker only when its planned test comparison is established. Read its gain importance before you trust it, because the features it leans on are the ones you'll pay for at query time. Early stopping on a few held-out queries keeps very different numbers of trees from one split to the next, from 99, 20, 8, 43, 38 across the cross-validation folds to 213 here. A rule of thumb, not a measurement: with this few queries, judge more queries before you tune further, and treat any one stopping point as noise.

이 부분은 아직 번역되지 않았으므로 영어로 표시된다.

Rerank semantic retrieval with a cross-encoder

마지막으로 변경된 판: 2026.11.10판

Semantic retrieval uses an embedding model, also called a bi-encoder, which reads the query and each product separately, so product embeddings are computed ahead of time. A cross-encoder reads the query and one product together and outputs one score (Nogueira and Cho, 2019). It can relate every query word to every product word, and it pays for that. It is often slower than a Sentence Transformer model, because it computes for each pair rather than each text, and so "Cross Encoders are often used to re-rank the top-k results" (Sentence Transformers). The store's cross-encoder (see the opening) is small for a language model, with 149M parameters (model card). It is the open model Alibaba-NLP/gte-reranker-modernbert-base, whose manifest, the file saved with its recording, gives its licence as Apache-2.0. We chose it because anyone can download it and rerun the recording. recordings/cross_encoder.py scored the union of BM25's and semantic retrieval's top 100 for every query (for two queries, its semantic list differs from lib.semantic's by a near-tie at the 100th place).

Chapter 1 reranked the hybrid run. Here the cross-encoder reranks the ranker to beat, semantic retrieval's top 20: the same products the LLM rerankers read, so the chapter's three rerankers start from identical lists. The chapter fixed that depth before any test result, and doesn't tune it. Reorder the top 20 by the recorded scores and keep the rest in place. That is the second planned comparison, which needs a p-value below 0.01:

cross-encoder-on-semantic.py Python 3.12.15 · bm25s 0.3.16 · lightgbm 4.7.0 · numpy 2.5.5 · pandas 3.1.1 · pystemmer 3.1.0 · scikit-learn 1.9.1 · scipy 1.18.1 통과
"""Reranks semantic retrieval's top 20 with the recorded cross-encoder scores, and compares the two on the test queries.The prose's point rests on the planned comparison's permutation p-value, and on the judged@10 lines, which show what thererank does to the judgments it can score."""from lib.metrics import evaluatefrom lib.ranking import DEPTH, rerank, report_plannedfrom lib.recorded import cross_encoder_scores, llm_listwise, manifestfrom lib.semantic import SemanticSearchfrom lib.store import load_queries, qrelsqueries, judgments = load_queries("test"), qrels("test")       # scored on the human judgmentssemantic = SemanticSearch().run(queries, k=100)recorded = cross_encoder_scores()# the recorded score of each (query, product) pairscore = dict(zip(zip(recorded["query_id"], recorded["product_id"]), recorded["score"]))print("cross-encoder model:", manifest("cross-encoder")["model"])print("licence in the manifest:", manifest("cross-encoder")["model_licence"])# the set of products the LLM was shown for each query, from the listwise recordingshown = llm_listwise().groupby("query_id")["product_id"].apply(set)  # what the LLM rerankers sawsame = sum(set(semantic[query_id][:DEPTH]) == shown[query_id] for query_id in queries.index)print(f"rerank depth: {DEPTH}")print(f"queries whose top {DEPTH} holds the products the llm rerankers saw: {same} of {len(queries)}")reranked = rerank(semantic, score, DEPTH)                       # top 20 reordered, the rest in placebefore, after = evaluate(semantic, judgments), evaluate(reranked, judgments)for name, frame in [("semantic", before), ("cross-encoder", after)]:    print(f"{name}, test ndcg@10: {frame['ndcg@10'].mean():.3f}")    print(f"{name}, test judged@10: {frame['judged@10'].mean():.3f}")    print(f"{name}, unjudged places in the test top 10s: {round((1 - frame['judged@10']).sum() * 10)}")# index 0 is the first placemoved = sum(reranked[query_id][0] != semantic[query_id][0] for query_id in queries.index)print(f"queries whose first product changes: {moved}")report_planned("cross-encoder", after["ndcg@10"], before["ndcg@10"])
출력python cross-encoder-on-semantic.py
cross-encoder model: Alibaba-NLP/gte-reranker-modernbert-base
licence in the manifest: Apache-2.0
rerank depth: 20
queries whose top 20 holds the products the llm rerankers saw: 30 of 30
semantic, test ndcg@10: 0.830
semantic, test judged@10: 0.990
semantic, unjudged places in the test top 10s: 3
cross-encoder, test ndcg@10: 0.813
cross-encoder, test judged@10: 0.983
cross-encoder, unjudged places in the test top 10s: 5
queries whose first product changes: 23
cross-encoder, mean difference: -0.016
cross-encoder, wins, losses, ties: 11, 12, 7
cross-encoder, permutation p: 0.178
cross-encoder, established at 0.01: no
cross-encoder, t-test p: 0.171
cross-encoder, t-test agrees: yes
cross-encoder, 99% interval: -0.058 to 0.006
cross-encoder, interval agrees: yes

Reranking the top 20 doesn't establish a gain: the rerank scores 0.813 against 0.830, with a permutation p-value of 0.178 (see the decisions table). The top 20 holds the products the LLM rerankers saw for 30 of 30 test queries, and the reranker changes the first product for 23 of them.

Judged@10 falls from 0.990 to 0.983, and the unjudged places in the test top 10s go from 3 to 5.

On the train queries, the same rerank scores 0.802 against semantic retrieval's 0.772 (depth sweep). The train queries are another sample, and their verdict repeats what chapter 3 found: two close rankers' order can flip between sets of queries.

Working rule: Rerank your strongest retrieval, at a depth fixed before you look at test results, and adopt the reranker only on an established gain on held-out queries. The store's general-purpose cross-encoder stays out of the pipeline.

이 부분은 아직 번역되지 않았으므로 영어로 표시된다.

Score each product with an LLM

마지막으로 변경된 판: 2026.11.10판

An LLM can rerank too, in three ways. Pointwise, an LLM judges each product alone and returns a label or a score. Pairwise, an LLM says which of two products fits the query better. Listwise, an LLM orders a whole list in one answer (Qin et al., 2024). In one study, on eight datasets from BEIR, a collection of retrieval benchmarks used to test models on data they weren't trained for, pointwise prompts did better with graded answers than with yes or no, because "the lack of intermediate relevance label options may cause the LLM to provide noisy or biased answers" (Zhuang et al., 2024a).

The store's LLM, the open model named in the chapter's opening, saw the query and one product at a time: name, class, category, description and attributes (what the LLM saw). The prompt asks for a label, exact, partial or irrelevant, and "a score from 0 to 100 for how well it matches". Reorder semantic retrieval's top 20 by that score; when two products tie, they keep semantic retrieval's order, a rule fixed with the recording. That reorder is the third planned comparison, held to a p-value below 0.01:

llm-pointwise-rerank.py Python 3.12.15 · bm25s 0.3.16 · lightgbm 4.7.0 · numpy 2.5.5 · pandas 3.1.1 · pystemmer 3.1.0 · scikit-learn 1.9.1 · scipy 1.18.1 통과
"""Reads the pointwise LLM's answers for the test queries, one call per product, and reranks semantic retrieval's top 20 bytheir scores. The prose's point rests on the distinct-scores line, which shows how coarse the scores are, and on theplanned comparison's p-value."""from lib.metrics import evaluatefrom lib.ranking import DEPTH, rerank, report_plannedfrom lib.recorded import llm_pointwise, manifestfrom lib.semantic import SemanticSearchfrom lib.store import load_queries, qrelsqueries, judgments = load_queries("test"), qrels("test")       # scored on the human judgments, not the LLM'ssemantic = SemanticSearch().run(queries, k=100)answers = llm_pointwise()                                       # one call per product: a label and a scoreprint("llm:", manifest("llm-rerank")["model"])print("licence in the manifest:", manifest("llm-rerank")["model_licence"])print(f"calls per query: {len(answers) // answers['query_id'].nunique()}")print(f"answers without a score: {answers['score'].isna().sum()}")print(f"distinct scores per query, median: {answers.groupby('query_id')['score'].nunique().median():.0f}")print(f"share of products scored 100: {(answers['score'] == 100).mean():.3f}")# each answered product's human grade, or missing when nobody judged itpeople = answers.apply(lambda row: judgments[row["query_id"]].get(row["product_id"]), axis=1)# grades 2, 1 and 0 are exact, partial and irrelevant, the labels the LLM was asked forfor label, grade in [("exact", 2), ("partial", 1), ("irrelevant", 0)]:    print(f"products the llm labels {label}: {(answers['label'] == label).sum()}")    print(f"products people judged {label}: {(people == grade).sum()}")print(f"products nobody judged: {people.isna().sum()}")score = dict(zip(zip(answers["query_id"], answers["product_id"]), answers["score"]))reranked = rerank(semantic, score, DEPTH)                       # equal scores keep semantic retrieval's ordercounts = answers.groupby("query_id")["score"].value_counts()          # products per score, for each queryshared = {query_id: sum(counts[(query_id, score[(query_id, product_id)])] > 1 for product_id in reranked[query_id][:10])          for query_id in queries.index}                                # top-10 products that share their score# more than 5 of the top 10 share a score: most of the list is decided by the tie-breakprint(f"queries where most of the reranked top 10 shares its score: {sum(n > 5 for n in shared.values())}")before, after = evaluate(semantic, judgments), evaluate(reranked, judgments)for name, frame in [("semantic", before), ("llm pointwise", after)]:    print(f"{name}, test ndcg@10: {frame['ndcg@10'].mean():.3f}")    print(f"{name}, test judged@10: {frame['judged@10'].mean():.3f}")    print(f"{name}, unjudged places in the test top 10s: {round((1 - frame['judged@10']).sum() * 10)}")report_planned("llm pointwise", after["ndcg@10"], before["ndcg@10"])
출력python llm-pointwise-rerank.py
llm: gemma4:12b-it-q4_K_M
licence in the manifest: Apache-2.0
calls per query: 20
answers without a score: 0
distinct scores per query, median: 4
share of products scored 100: 0.390
products the llm labels exact: 251
products people judged exact: 310
products the llm labels partial: 345
products people judged partial: 265
products the llm labels irrelevant: 4
products people judged irrelevant: 7
products nobody judged: 18
queries where most of the reranked top 10 shares its score: 30
semantic, test ndcg@10: 0.830
semantic, test judged@10: 0.990
semantic, unjudged places in the test top 10s: 3
llm pointwise, test ndcg@10: 0.837
llm pointwise, test judged@10: 0.983
llm pointwise, unjudged places in the test top 10s: 5
llm pointwise, mean difference: 0.008
llm pointwise, wins, losses, ties: 11, 8, 11
llm pointwise, permutation p: 0.241
llm pointwise, established at 0.01: no
llm pointwise, t-test p: 0.244
llm pointwise, t-test agrees: yes
llm pointwise, 99% interval: -0.007 to 0.026
llm pointwise, interval agrees: yes

The recording made 20 calls per query, and 0 answers came back without a score. The scores are coarse. The median query gets 4 distinct scores for its 20 products, and a share of 0.390 of all products score 100. In 30 queries, most of the reranked top 10 shares its score with another product, so the tie-break, semantic retrieval's order, decides much of the list.

With this prompt, on these lists, the LLM calls fewer products exact than people did:

Label LLM People
exact 251 310
partial 345 265
irrelevant 4 7
nobody judged – 18

On these lists that holds, but chapter 4 shows that the direction of an LLM's disagreement with people changes with the pairs judged and the prompt, so don't carry this one elsewhere.

The pointwise rerank doesn't establish a gain either: it scores 0.837 against 0.830, at a permutation p-value of 0.241 (see the decisions table).

The LLM's order puts more unjudged products into the top 10s: 5 places instead of 3, and judged@10 falls from 0.990 to 0.983.

Working rule: If you rerank pointwise with an LLM, ask for a graded label or a score, count the distinct scores it gives per query, and decide the tie-break in advance: retrieval's order is a safe default. Score the result with human judgments, never with the same LLM's grades.

20개 상품을 LLM 프롬프트 하나로 정렬하기

마지막으로 변경된 판: 2026.11.8판

세 번째 방법인 리스트와이즈(listwise) 프롬프트는 후보에 번호를 매기고, LLM에게 관련도 순서대로 그 번호를 반환하게 한다. RankGPT가 이를 순열 생성(permutation generation)으로 소개했다(Sun et al., 2023). 프롬프트 하나에는 상품을 일정 개수만큼만 담을 수 있으므로, 긴 목록은 슬라이딩 윈도(sliding window) 방식으로 재순위화한다. RankGPT는 BM25의 상위 100개를 "윈도 크기 20, 스텝 크기 10"으로 목록의 아래쪽에서 위쪽으로 재순위화한다(3.2절과 6.1절). 답은 유효한 순서여야 하는데, 이것이 실패할 수 있다. 2023년의 중간 규모 LLM, 예를 들어 200억 파라미터의 Flan-UL2에서는 리스트와이즈 프롬프트가 거의 항상 실패했다. 순위가 빠지거나 반복되거나 거부되었기 때문이다(Qin et al., 2024).

상점의 LLM은 프롬프트마다 상품 20개, 즉 윈도 하나를 보았다. 프롬프트는 각 상품을 이름, 클래스, 속성, 설명의 처음 160자로 이루어진 짧은 한 줄로 보여 준다. 마지막 지시문은 다음과 같다. "Order all 20 products from the best match for the query to the worst. Return the product numbers, each exactly once, in JSON." JSON 스키마는 정확히 20개의 숫자를 요구하지만 서로 다른 20개일 것까지 요구하지는 않는다. 그래서 기록 단계에서 숫자가 반복되거나 빠진 답을 보정하고 그 사실을 표시한다. 시맨틱 검색의 상위 20개를 재순위화하되, 상품을 시맨틱 순서로 보여 주었을 때 LLM이 내놓은 답을 따른다. 이 재순위화는 네 번째 계획된 비교이며, p-값 기준은 0.01 미만이다.

llm-listwise-rerank.py Python 3.12.15 · bm25s 0.3.16 · lightgbm 4.7.0 · numpy 2.5.5 · pandas 3.1.1 · pystemmer 3.1.0 · scikit-learn 1.9.1 · scipy 1.18.1 통과
"""Reads the listwise LLM's answers for the test queries, in both orders, and reranks semantic retrieval's top 20 by theanswer given in semantic order. The prose's point rests on the planned comparison's p-value, and on the repair count,which shows how many answers needed a fix."""from lib.metrics import evaluatefrom lib.ranking import DEPTH, report_plannedfrom lib.recorded import llm_listwise, manifestfrom lib.semantic import SemanticSearchfrom lib.store import load_queries, qrelsqueries, judgments = load_queries("test"), qrels("test")       # scored on the human judgments, not the LLM'ssemantic = SemanticSearch().run(queries, k=100)answers = llm_listwise()                                        # one call per query and order shown# the answers hold one row per product, so keep one row per call: one call per query and ordercalls = answers.drop_duplicates(["query_id", "order"])print(f"listwise calls, both orders: {len(calls)}")# valid is false where the recording repaired a repeated or missing numberprint(f"answers that needed repair: {(~calls['valid']).sum()}")print(f"products per call: {answers.groupby(['query_id', 'order']).size().max()}")print("prompt asks for:", manifest("llm-rerank")["prompts"]["listwise"].splitlines()[-1].replace("{n}", str(DEPTH)))shown_in_semantic_order = answers[answers["order"] == "semantic"].sort_values(["query_id", "position"])order = shown_in_semantic_order.groupby("query_id")["product_id"].apply(list)# the reranked top 20, then the rest of semantic retrieval's list in place: ranking[DEPTH:] starts at the 21st productreranked = {query_id: order[query_id] + ranking[DEPTH:] for query_id, ranking in semantic.items()}before, after = evaluate(semantic, judgments), evaluate(reranked, judgments)for name, frame in [("semantic", before), ("llm listwise", after)]:    print(f"{name}, test ndcg@10: {frame['ndcg@10'].mean():.3f}")    print(f"{name}, test judged@10: {frame['judged@10'].mean():.3f}")    print(f"{name}, unjudged places in the test top 10s: {round((1 - frame['judged@10']).sum() * 10)}")moved = sum(reranked[query_id][0] != semantic[query_id][0] for query_id in queries.index)print(f"queries whose first product changes: {moved}")report_planned("llm listwise", after["ndcg@10"], before["ndcg@10"])
출력python llm-listwise-rerank.py
listwise calls, both orders: 60
answers that needed repair: 0
products per call: 20
prompt asks for: Order all 20 products from the best match for the query to the worst. Return the product numbers, each exactly once, in JSON.
semantic, test ndcg@10: 0.830
semantic, test judged@10: 0.990
semantic, unjudged places in the test top 10s: 3
llm listwise, test ndcg@10: 0.819
llm listwise, test judged@10: 0.983
llm listwise, unjudged places in the test top 10s: 5
queries whose first product changes: 19
llm listwise, mean difference: -0.010
llm listwise, wins, losses, ties: 12, 12, 6
llm listwise, permutation p: 0.580
llm listwise, established at 0.01: no
llm listwise, t-test p: 0.498
llm listwise, t-test agrees: yes
llm listwise, 99% interval: -0.070 to 0.016
llm listwise, interval agrees: yes

기록 단계에서 리스트와이즈 호출은 테스트 질의당 두 번씩 모두 60회 이루어졌고, 답 0개가 보정이 필요했다. 재순위화로 첫 번째 상품이 바뀌는 질의는 19개이다.

리스트와이즈 재순위화의 향상은 입증되지 않았다. 점수는 0.819로, 0.830와 비교되며 순열 p-값은 0.580이다(다섯 가지 비교 전체).

Judged@10도 여기서 0.990에서 0.983로 떨어지며, 판정되지 않은 자리는 5개이다. 시맨틱 검색은 3개이다.

작업 규칙: 리스트와이즈 답은 사용하기 전에 모두 검사한다. 빠진 번호, 반복된 번호, 알 수 없는 번호를 세고, 보정이 필요했던 답의 수를 기록한다. 실행 환경이 지원하면 스키마로 답의 형태를 제약한다. 측정하기 전에 윈도, 스텝, 상품을 보여 주는 순서를 고정한다. 다음 점검에서 보듯이 순서에 따라 답이 달라지기 때문이다.

이 부분은 아직 번역되지 않았으므로 영어로 표시된다.

Reverse the input to measure positional bias

마지막으로 변경된 판: 2026.11.10판

A listwise answer can suffer from positional bias: an LLM's tendency to let a product's place in the prompt affect its rank, not the click position bias of shoppers. "Large language models (LLMs) exhibit positional bias in how they use context, which especially complicates listwise ranking" (Tang et al., 2024). On TREC DL19, the 2019 queries of TREC's Deep Learning track, a passage-ranking benchmark, gpt-3.5-turbo reranked BM25's top 100 to an nDCG@10 of 65.80 when shown BM25's order, 32.77 when shown it reversed and 25.17 when shown a random order (Sun et al., 2023, Table 5). The direct check is to show the same products in two orders and compare the answers. Kendall's tau measures how alike two orderings are, from 1 for the same order to −1 for the reverse, by counting the pairs they put the same way round (chapter 4).

The recording sent each test query's listwise prompt to the same LLM twice. The first prompt showed semantic retrieval's top 20 in semantic order, as the listwise rerank used; the second showed them reversed. Out of the 30 test queries, how often will the LLM put the same product first both times? Write your number down.

listwise-position-bias.py Python 3.12.15 · bm25s 0.3.16 · lightgbm 4.7.0 · numpy 2.5.5 · pandas 3.1.1 · pystemmer 3.1.0 · scikit-learn 1.9.1 · scipy 1.18.1 통과
"""Shows the listwise LLM the same test products in two orders, and measures how far its answers agree and how far eachanswer follows the order it was shown. The prose's point rests on the mean tau between the two answers, and on the twotaus between each answer and its own input order."""import numpy as npfrom scipy.stats import kendalltaufrom lib.metrics import evaluatefrom lib.ranking import DEPTHfrom lib.recorded import llm_listwisefrom lib.semantic import SemanticSearchfrom lib.store import load_queries, qrelsqueries, judgments = load_queries("test"), qrels("test")semantic = SemanticSearch().run(queries, k=DEPTH)               # the order the first prompt showedanswers = llm_listwise().sort_values(["query_id", "order", "position"])# the products in the order the LLM returned them, for each query and each order shownanswer = {key: list(group["product_id"]) for key, group in answers.groupby(["query_id", "order"])}def tau(first: list, second: list) -> float:    """Kendall's tau between two orderings of the same products."""    place = {product_id: rank for rank, product_id in enumerate(second)}    # each product's place in the second ordering    # the first ordering's positions, against where the second ordering puts each product    return kendalltau(range(len(first)), [place[product_id] for product_id in first]).statisticbetween, shown_semantic, shown_reversed, overlap, same_first = [], [], [], [], 0for query_id, shown in semantic.items():    forward, backward = answer[(query_id, "semantic")], answer[(query_id, "reversed")]    between.append(tau(forward, backward))    shown_semantic.append(tau(forward, shown))    shown_reversed.append(tau(backward, shown[::-1]))     # the reversed input, as the second prompt showed it    overlap.append(len(set(forward[:10]) & set(backward[:10])))    same_first += forward[0] == backward[0]               # index 0 is the first placeprint(f"test queries: {len(semantic)}")print(f"tau between the two answers, mean: {np.mean(between):.3f}")print(f"tau between the two answers, lowest: {min(between):.3f}\ntau between the two answers, highest: {max(between):.3f}")print(f"top-10 products in common, mean: {np.mean(overlap):.3f}")print(f"top-10 products in common, fewest: {min(overlap)}")print(f"queries with the same first product: {same_first}")print(f"share of queries with the same first product: {same_first / len(semantic):.3f}")print(f"tau between the order shown and the answer, semantic order: {np.mean(shown_semantic):.3f}")print(f"tau between the order shown and the answer, reversed order: {np.mean(shown_reversed):.3f}")# an answer that ignored the order would repeat itself, so its tau with the reversed input is minus its tau with the semantic orderprint(f"the same, if the order shown made no difference: {-np.mean(shown_semantic):.3f}")reversed_run = {query_id: answer[(query_id, "reversed")] for query_id in semantic}print(f"reversed answer, test ndcg@10, not a planned comparison: {evaluate(reversed_run, judgments)['ndcg@10'].mean():.3f}")
출력python listwise-position-bias.py
test queries: 30
tau between the two answers, mean: 0.311
tau between the two answers, lowest: -0.484
tau between the two answers, highest: 0.811
top-10 products in common, mean: 6.967
top-10 products in common, fewest: 3
queries with the same first product: 13
share of queries with the same first product: 0.433
tau between the order shown and the answer, semantic order: 0.414
tau between the order shown and the answer, reversed order: 0.045
the same, if the order shown made no difference: -0.414
reversed answer, test ndcg@10, not a planned comparison: 0.826

The same product comes first for 13 of the 30 queries, so the expectation that a capable model finds the same best match whichever way the list comes holds for only a share of 0.433 of the queries.

The two answers agree with a mean tau of 0.311, and per query the tau runs from -0.484 to 0.811. Their top 10s share 6.967 products on average, as few as 3.

The answers also lean toward the order shown. Each answer's tau against its own input order is 0.414 for the semantic order and 0.045 for the reversed one. An LLM that ignored the order would give the same answer both times, so its reversed answer would have a tau of -0.414 against the reversed input. The reversed answer sits far nearer its own input order than that: positional bias, for this model, prompt and these queries. The reversed answer scores 0.826 nDCG@10; it is not a planned comparison, so the chapter doesn't test it.

Three remedies exist. Pairwise ranking prompting asks the LLM about each pair twice, once in each order, and counts a disagreement as a tie (Qin et al., 2024). Permutation self-consistency reranks several shuffled orders and aggregates the answers, which raised scores by up to 7 to 18% for GPT-3.5 and 8 to 16% for LLaMA v2 (70B) (Tang et al., 2024). And rerankers fine-tuned for listwise ranking, such as the open RankZephyr, are reported to be "resilient against variations in initial document ordering" (Pradeep et al., 2023); the store didn't record one.

Working rule: Before you trust a listwise LLM reranker, run each query in at least two orders and report how much the answers agree. When agreement is low, aggregate several orders or ask each pair in both orders, and count those calls against your budget. In production, show the products in retrieval's order every time, so that the same list at least gets the same answer.

예산에 맞춰 재순위화 깊이 정하기

마지막으로 변경된 판: 2026.11.8판

검색 파이프라인은 단계별로 실행되며, 각 단계는 앞 단계보다 느리고 받는 후보 수는 더 적다. 이 장의 모든 재순위화기는 단계를 하나씩 더한다. 다단계 순위화기는 "각 파이프라인 단계로 들어가는 후보의 수를 조절함으로써 품질과 지연 시간 사이에서 균형을 잡을 수 있는 능력"을 제공한다(Nogueira et al., 2019). 따라서 재순위화기가 쿼리마다 읽는 상품의 수인 재순위화 깊이는 예산을 두고 내리는 결정이다. 실행 시간은 하드웨어와 실행 환경의 선택에 따라 달라지므로(Peng et al., 2025), 이 절에서는 실행 시간 대신 이를 좌우하는 요소를 센다. 쿼리마다 모델을 호출하는 횟수와 모델이 읽는 텍스트의 양을 세고, 여기에 모델의 크기를 곱한다. 이 상점의 교차 인코더는 매개변수가 149M개이고(model card), LLM은 약 120억 개로(model card) 약 80배에 이른다.

깊이 20에서 각 재순위화기가 테스트 쿼리 하나당 하는 모델 호출 횟수와 읽는 문자 수를 센다. 문자 수는 상품 텍스트와 기록된 프롬프트에서 구한다. 여기에 상점이 기록하지 않은 두 설계에 필요한 호출 횟수도 더한다.

reranker-prompt-sizes.py Python 3.12.15 · bm25s 0.3.16 · lightgbm 4.7.0 · numpy 2.5.5 · pandas 3.1.1 · pystemmer 3.1.0 · scikit-learn 1.9.1 · scipy 1.18.1 통과
"""Counts the characters each reranker reads per test query at depth 20, from the products' text and the recorded prompts,and the model calls each design makes. The prose's point rests on the characters and calls lines, and on the two designsthe store didn't record."""import numpy as npfrom lib.agreement import as_the_llm_sawfrom lib.ranking import DEPTH, listwise_linefrom lib.recorded import manifestfrom lib.semantic import SemanticSearchfrom lib.store import load_products, load_queries, product_textproducts, queries = load_products(), load_queries("test")semantic = SemanticSearch().run(queries, k=DEPTH)                # the 20 products each reranker readsprompts, text = manifest("llm-rerank")["prompts"], product_text(products)cross_encoder, pointwise, listwise = [], [], []                  # characters read per queryfor query_id, query in queries["query"].items():    shown = products.loc[semantic[query_id]]    # one call per product, each reading the query and that product's text together    cross_encoder.append(sum(len(query) + len(text[product_id]) for product_id in semantic[query_id]))    # the pointwise prompt, filled in with the query and the product as the LLM saw it    pointwise.append(sum(len(prompts["pointwise"].format(query=query, product=as_the_llm_saw(product)))                         for _, product in shown.iterrows()))    # products numbered from 1, as the listwise prompt shows them    lines = "\n".join(f"[{number}] {listwise_line(product)}" for number, (_, product) in enumerate(shown.iterrows(), 1))    listwise.append(len(prompts["listwise"].format(query=query, products=lines, n=DEPTH)))# the cross-encoder and the pointwise LLM make one call per product; the listwise LLM makes one call per queryfor name, calls, characters in [("cross-encoder", DEPTH, cross_encoder), ("llm pointwise", DEPTH, pointwise),                                ("llm listwise", 1, listwise)]:    print(f"{name}, model calls per query: {calls}")    print(f"{name}, characters read per query, mean: {np.mean(characters):,.0f}")print(f"llm pairwise, every pair in both orders, calls per query: {DEPTH * (DEPTH - 1)}")# a sliding window of 20 products, moved 10 at a time: its starts are list positions 0, 10, ..., 80window, step, depth = 20, 10, 100                                # a sliding window over a deeper listprint(f"llm listwise over the top {depth}, window {window}, step {step}, calls per query: {(depth - window) // step + 1}")
출력python reranker-prompt-sizes.py
cross-encoder, model calls per query: 20
cross-encoder, characters read per query, mean: 9,911
llm pointwise, model calls per query: 20
llm pointwise, characters read per query, mean: 17,864
llm listwise, model calls per query: 1
llm listwise, characters read per query, mean: 5,973
llm pairwise, every pair in both orders, calls per query: 380
llm listwise over the top 100, window 20, step 10, calls per query: 9
재순위화기, 상위 20 쿼리당 모델 호출 쿼리당 읽는 문자 수
교차 인코더 20 9,911
LLM, 포인트와이즈 20 17,864
LLM, 리스트와이즈 1 5,973

포인트와이즈 방식은 지시문을 반복하고 각 상품을 전체로 보여 주므로, 독립된 호출로 가장 많은 텍스트를 읽는다. 리스트와이즈 방식은 한 번의 호출로 가장 적은 텍스트를 읽지만, 20개의 번호를 순서대로 써야 한다. 모든 쌍을 양쪽 순서로 비교하는 페어와이즈 프롬프트는 쿼리당 380번 호출이 필요하고, 상위 100개에 대한 RankGPT의 슬라이딩 윈도는 9번 호출이 필요하다. 발표된 한 측정에서는 TREC DL19에서 BM25의 상위 100개를 RTX A6000 GPU 한 대로 Flan-T5-XL을 써서 재순위화했다. 포인트와이즈 예/아니오 프롬프트는 쿼리당 1.5초에 nDCG@10 0.650을 얻었고, 모든 쌍에 대한 페어와이즈 프롬프트는 254.9초에 0.713을 얻었으며 LLM 호출은 9,900번이었다(Zhuang et al., 2024b). 교차 인코더를 특징으로 쓰는 LambdaMART는 모든 후보에 대해 교차 인코더 점수가 필요하며, 쿼리당 중앙값은 131쌍이다.

교차 인코더는 비용이 깊이에 비례해 늘어나므로, 문제는 품질 향상이 어디에서 멈추느냐이다. 학습 쿼리에서 깊이를 바꿔 가며 확인한다.

rerank-depth-on-train.py Python 3.12.15 · bm25s 0.3.16 · lightgbm 4.7.0 · numpy 2.5.5 · pandas 3.1.1 · pystemmer 3.1.0 · scikit-learn 1.9.1 · scipy 1.18.1 통과
"""Reranks semantic retrieval's top 5 to 75 with the recorded cross-encoder scores, on the train queries, and prints nDCG@10at each depth. The prose's point rests on the change from the depth before, which shows where the gain flattens and thenfalls."""from lib.metrics import evaluatefrom lib.ranking import rerankfrom lib.recorded import cross_encoder_scoresfrom lib.semantic import SemanticSearchfrom lib.store import load_queries, qrelsqueries, judgments = load_queries("train"), qrels("train")      # choosing a depth is tuning: train queriessemantic = SemanticSearch().run(queries, k=100)recorded = cross_encoder_scores()# the recorded cross-encoder score of each (query, product) pairscore = dict(zip(zip(recorded["query_id"], recorded["product_id"]), recorded["score"]))train_pairs = recorded[recorded["query_id"].isin(queries.index)]print(f"depth 0 (semantic alone), train ndcg@10: {evaluate(semantic, judgments)['ndcg@10'].mean():.3f}")previous = round(evaluate(semantic, judgments)["ndcg@10"].mean(), 3)     # changes between the printed valuesfor depth in [5, 10, 20, 30, 50, 75]:    mean = round(evaluate(rerank(semantic, score, depth), judgments)["ndcg@10"].mean(), 3)    print(f"depth {depth}, train ndcg@10: {mean:.3f}")    print(f"depth {depth}, change from the depth before: {mean - previous:+.3f}")    previous = mean# every candidate needs its own cross-encoder score, so this counts the recorded pairs per queryprint(f"pairs per query to score every candidate, median: {train_pairs.groupby('query_id').size().median():.0f}")
출력python rerank-depth-on-train.py
depth 0 (semantic alone), train ndcg@10: 0.772
depth 5, train ndcg@10: 0.782
depth 5, change from the depth before: +0.010
depth 10, train ndcg@10: 0.784
depth 10, change from the depth before: +0.002
depth 20, train ndcg@10: 0.802
depth 20, change from the depth before: +0.018
depth 30, train ndcg@10: 0.804
depth 30, change from the depth before: +0.002
depth 50, train ndcg@10: 0.793
depth 50, change from the depth before: -0.011
depth 75, train ndcg@10: 0.788
depth 75, change from the depth before: -0.005
pairs per query to score every candidate, median: 131

교차 인코더의 향상은 20과 30 사이에서 평탄해지며, 이는 LLM 기록에 맞추어 정한 깊이 근처이다. 의미 검색의 0.772에서 시작해, 상위 5개를 재순위화하면 0.782이 되고 상위 20개를 재순위화하면 0.802이 된다. 상위 30개는 쌍을 절반만큼 더 쓰고 +0.002을 더한다. 더 깊어지면 떨어져서 50에서는 0.793, 75에서는 0.788이다.

테스트에서 깊이 20의 재순위화는 향상을 입증하지 못했으므로, 이 상점에서는 가장 저렴한 재순위화기인 '없음'이 유지할 만한 선택이기도 하다.

작업 규칙: 검토하는 각 재순위화기에 대해 깊이별로 쿼리당 호출 횟수와 문자 수를 세고, 깊이는 학습 쿼리에서 정한다. 측정이 아닌 경험칙으로, 학습 nDCG@10이 더 이상 오르지 않는 깊이까지만 재순위화한다. 그 깊이를 넘는 상품은 비용이 같은데도 얻는 것이 없기 때문이다. 그런 다음 그 구성을 자신의 하드웨어와 트래픽에서 시간을 재어 보고, 지연 시간 예산 안에 들어올 때만 그 단계를 유지한다.

교차 인코더를 작은 랭커로 증류하기

마지막으로 변경된 판: 2026.11.10판

예산에 맞추기에는 너무 느린 재순위화 모델이 빠른 모델을 가르칠 수 있다. 증류(distillation)는 작은 모델인 학생(student)이 큰 모델인 교사(teacher)를 모방하도록 학습시키는 방법이다. Hinton 등은 이를 앙상블의 지식을 단일 모델로 증류하는 것으로 설명한다(Hinton et al., 2015). 랭킹에서는 교사의 점수가 학생의 레이블이 된다. 서로 다른 아키텍처의 점수가 흔히 그렇듯 두 모델의 점수 크기가 다르면, 학생은 Margin-MSE라는 손실로 두 상품 점수의 차이(마진)를 대신 학습할 수 있다(Hofstätter et al., 2020). 증류는 LLM에서 더 작은 모델로 내려가는 방향으로도 이루어진다. RankGPT는 증류한 440M 모델이 BEIR 벤치마크에서 3B 지도 학습 모델보다 높은 성능을 보였다고 보고한다(Sun et al., 2023; BEIR). 최상급 표현이 들어간 질의를 다룬 한 상품 검색 연구에서는, LLM으로 직접 재순위화하면 지연 시간이 지나치게 길어지는 상황에서 경량 모델이 LLM의 해석을 넘겨받았다(Zhu et al., 2025). 교사는 사람이 매긴 레이블이 필요 없고 모든 후보에 레이블을 붙인다.

교사는 교차 인코더다. 학생은 7 특징에 대한 LightGBM 회귀 모델이며, 이 특징은 검색이 이미 한 번 수행하는 호출 외에 추가 모델 호출이 필요 없는 것들로, cross_encoder를 제외한 모든 것이다. 레이블은 교사의 원점수이며, 트리 회귀는 이를 교사 자신의 척도에서 맞출 수 있다. 학생은 같은 후보를 product_id 순서로 순위화하며, 트리 크기는 LambdaMART에 선택한 값을 따른다. 먼저 학습 질의에 대한 교차 검증으로, 학생을 같은 특징으로 사람의 평가 등급을 학습한 LambdaMART 모델과 비교한다.

distillation-on-train.py Python 3.12.15 · bm25s 0.3.16 · lightgbm 4.7.0 · numpy 2.5.5 · pandas 3.1.1 · pystemmer 3.1.0 · scikit-learn 1.9.1 · scipy 1.18.1 통과
"""Trains a student LightGBM regression on the cross-encoder's scores, and a LambdaMART model on human grades with the samefeatures, both by cross-validation on the train queries. The prose's point rests on the student's score, and on itsdifference from the human-graded ranker."""from lightgbm import LGBMRanker, LGBMRegressorfrom lib.metrics import evaluatefrom lib.ranking import CHEAP_FEATURES, DETERMINISTIC, LAMBDARANK, TREES, cross_validate, feature_table, rankedfrom lib.semantic import SemanticSearchfrom lib.significance import comparefrom lib.store import load_products, load_queries, qrelsqueries, judgments = load_queries("train"), qrels("train")          # design work: train queries onlycandidates = feature_table(load_products(), queries)# unjudged candidates count as grade 0, as in the metriccandidates["grade or 0"] = candidates["grade"].fillna(0).astype(int)student = cross_validate(lambda: LGBMRegressor(**TREES, **DETERMINISTIC), candidates, candidates,                         "cross_encoder", CHEAP_FEATURES, judgments)          # labels from the teacherhuman = cross_validate(lambda: LGBMRanker(**LAMBDARANK, **TREES, **DETERMINISTIC), candidates, candidates,                       "grade or 0", CHEAP_FEATURES, judgments)               # labels from people# the teacher's own order of the candidates, ties broken by product_idteacher = evaluate(ranked(candidates, candidates["cross_encoder"]), judgments)["ndcg@10"]semantic = evaluate(SemanticSearch().run(queries, k=100), judgments)["ndcg@10"]for name, scores in [("semantic", semantic), ("teacher", teacher), ("student, teacher's labels", student),                     ("lambdarank, human grades, same features", human)]:    print(f"{name}, train ndcg@10: {scores.mean():.3f}")# the student's score minus the human-graded ranker's, per query, tested by permutationresult = compare(student, human)print(f"student minus the human-graded ranker: {result['mean difference']:.3f}")print(f"student minus the human-graded ranker, permutation p: {result['permutation p']:.3f}")
출력python distillation-on-train.py
semantic, train ndcg@10: 0.772
teacher, train ndcg@10: 0.785
student, teacher's labels, train ndcg@10: 0.791
lambdarank, human grades, same features, train ndcg@10: 0.747
student minus the human-graded ranker: 0.044
student minus the human-graded ranker, permutation p: 0.002

이 학습 질의와 이 특징에서는 모든 후보에 붙인 교사의 레이블이 사람의 평가 등급보다 더 나은 랭커를 학습시켰다. 이때 평가되지 않은 후보는 0으로 센다. 학생의 점수는 0.791이고, 교사 자신의 후보 순서는 0.785이다.

사람의 평가 등급으로 학습한 모델의 점수는 0.747으로 시맨틱 검색의 0.772보다 낮으며, 학생에서 이를 뺀 값은 0.044이고 순열 p-값은 0.002이다. 이는 학습 질의에서 한 튜닝 점검일 뿐 증거가 아니다.

그다음 LambdaMART와 같은 학습 질의 보류 집합으로 조기 종료를 적용해 학습 질의에서 학생을 적합시키고, 같은 임곗값 0.01에서 마지막 계획된 비교를 수행한다.

distilled-ranker-on-test.py Python 3.12.15 · bm25s 0.3.16 · lightgbm 4.7.0 · numpy 2.5.5 · pandas 3.1.1 · pystemmer 3.1.0 · scikit-learn 1.9.1 · scipy 1.18.1 통과
"""Fits a student LightGBM regression on the teacher's scores for the train queries, then reports its gain shares, howclosely its order follows similarity and the teacher, and its planned comparison on the test queries. The prose's pointrests on that comparison and on the taus printed for each ranking."""import numpy as npfrom lightgbm import LGBMRegressorfrom scipy.stats import kendalltaufrom lib.metrics import evaluatefrom lib.ranking import CHEAP_FEATURES, DETERMINISTIC, TREES, feature_table, fit, ranked, report_planned, stopping_splitfrom lib.semantic import SemanticSearchfrom lib.store import load_products, load_queries, qrelsproducts, train, test = load_products(), load_queries("train"), load_queries("test")table = feature_table(products, train)                       # the teacher's score is the label: no human gradesprint(f"train candidates the teacher labels: {len(table)}")print(f"of them, judged by nobody: {table['grade'].isna().sum()}")print(f"student features: {len(CHEAP_FEATURES)}")# early stopping on the same 10 held-out train queries that LambdaMART stops onstudent = fit(LGBMRegressor(**TREES, **DETERMINISTIC), table, "cross_encoder", CHEAP_FEATURES, *stopping_split(train.index))print(f"trees kept by early stopping: {student.best_iteration_}")candidates, judgments = feature_table(products, test), qrels("test")       # test queries: read once, to reportpredicted = student.predict(candidates[CHEAP_FEATURES])gain = student.booster_.feature_importance(importance_type="gain")# the three features with the most gainfor name, share in sorted(zip(CHEAP_FEATURES, gain / gain.sum()), key=lambda pair: -pair[1])[:3]:    print(f"student's share of gain, {name}: {share:.3f}")for name in ["cross_encoder", "similarity"]:                   # whose order does the student follow?    # candidates has a default index, so group.index picks this query's rows out of predicted    tau = [kendalltau(group[name], predicted[group.index]).statistic for _, group in candidates.groupby("query_id")]    print(f"tau between student and {name}, mean over test queries: {np.mean(tau):.3f}")teacher = evaluate(ranked(candidates, candidates["cross_encoder"]), judgments)print(f"teacher's own order of the candidates, test ndcg@10, not a planned comparison: {teacher['ndcg@10'].mean():.3f}")after = evaluate(ranked(candidates, predicted), judgments)before = evaluate(SemanticSearch().run(test, k=100), judgments)for name, frame in [("semantic", before), ("distilled", after)]:    print(f"{name}, test ndcg@10: {frame['ndcg@10'].mean():.3f}")    print(f"{name}, test judged@10: {frame['judged@10'].mean():.3f}")    # each query's top 10 is 10 places, so (1 - judged@10) times 10 counts its unjudged places    print(f"{name}, unjudged places in the test top 10s: {round((1 - frame['judged@10']).sum() * 10)}")report_planned("distilled", after["ndcg@10"], before["ndcg@10"])
출력python distilled-ranker-on-test.py
train candidates the teacher labels: 6896
of them, judged by nobody: 1975
student features: 7
trees kept by early stopping: 202
student's share of gain, similarity: 0.812
student's share of gain, name_match: 0.126
student's share of gain, bm25: 0.055
tau between student and cross_encoder, mean over test queries: 0.466
tau between student and similarity, mean over test queries: 0.753
teacher's own order of the candidates, test ndcg@10, not a planned comparison: 0.782
semantic, test ndcg@10: 0.830
semantic, test judged@10: 0.990
semantic, unjudged places in the test top 10s: 3
distilled, test ndcg@10: 0.826
distilled, test judged@10: 0.987
distilled, unjudged places in the test top 10s: 4
distilled, mean difference: -0.004
distilled, wins, losses, ties: 10, 13, 7
distilled, permutation p: 0.720
distilled, established at 0.01: no
distilled, t-test p: 0.694
distilled, t-test agrees: yes
distilled, 99% interval: -0.034 to 0.017
distilled, interval agrees: yes

교사는 학습 후보 6896개 전부에 레이블을 붙이며, 그중 1975개는 아무도 평가하지 않은 것이다. 조기 종료는 트리 202개를 남긴다.

학생은 similarity에 크게 의존하며, 이 특징이 이득의 0.812를 차지해 0.126를 차지하는 name_match와 0.055를 차지하는 bm25보다 앞선다. 테스트 후보에서 학생의 점수는 질의당 평균 tau 0.753로 시맨틱 유사도와 일치하고, 교사의 점수와는 0.466로 일치한다. 따라서 학생은 주로 시맨틱 검색의 순서를 학습했고, 이름 일치와 BM25 일치로 교사의 순서 쪽으로 조금 조정된 것이다.

학생은 이득을 입증하지 못한다. 증류한 랭커의 nDCG@10은 0.826이고 시맨틱 검색은 0.830이며, 순열 p-값은 0.720이다(결정 표 참조). 이 장에서 검정하기로 계획하지 않았던 테스트 후보에 대한 교사 자신의 순서는 0.782를 기록한다.

Judged@10은 학생이 0.987, 시맨틱 검색이 0.990이며, 학생의 상위 10개 결과에는 평가되지 않은 자리가 4개 있고 시맨틱 검색에는 3개 있다.

작업 규칙: 재순위화 모델이 확립된 이득을 내지만 질의당 비용이 너무 클 때 증류한다. 학습 질의의 모든 후보에 교사로 레이블을 붙이고, 교사 없이 계산할 수 있는 특징으로 학생을 학습시킨 뒤, 보류한 질의에서 학생을 자신의 검색과 비교해 검정한다. 학생의 이득 중요도를 읽는다. 검색 자체의 점수가 이를 지배한다면 학생은 대체로 자신의 검색이다. 학생이 교사의 척도를 맞출 수 없을 때는 쌍 사이의 마진을 사용한다. 이 상점에는 아직 시맨틱 검색을 능가하는 교사가 없으므로 코드는 유지하되 학생은 배포하지 않는다.

운영 환경에서 무엇이 잘못되는가

마지막으로 변경된 판: 2026.11.8판

실시간 랭커는 쇼핑객이 기다리는 동안 특징을 계산하고 모델을 호출한다. 이때 사용하는 코드와 모델 버전은 학습과 테스트에 쓴 것과 달라질 수 있다.

특징이 학습 단계와 서빙 단계 사이에서 어긋난다. 실시간 시스템이 토큰화를 다르게 하거나, BM25의 척도를 다시 조정하거나, 임베딩 모델을 업그레이드하면 학습된 가중치는 다른 수치를 만나게 되고, 어떤 오류도 발생하지 않는다. 특징은 하나의 공유 코드 경로로 계산하고, 제공한 각 랭킹과 함께 기록한다.

행 순서가 학습으로 새어 든다. LightGBM의 lambdarank는 입력 행의 순서에서 출발하며, 랭킹은 동점을 어떤 식으로든 처리한다. 리랭커의 점수로 정렬된 파일은 그 리랭커의 랭킹을 모델에 넘겨 준다. 매번 학습하고 점수를 매기기 전에 후보를 ID로 정렬하거나 고정된 시드로 섞는다.

판정은 풀(pool)에서 나온다. WANDS는 자체 시스템이 제시한 상품만 판정했다(2장). 판정되지 않은 후보를 무관한 것으로 세면, 모델은 그 시스템들이 찾지 못한 상품을 모두 강등하도록 학습한다. 추가하는 모든 랭커에 대해 judged@10을 확인하고, 향상을 신뢰하기 전에 판정되지 않은 상품을 판정받도록 한다.

LLM 판정자는 LLM 리랭커를 선호한다. 한 연구에서 LLM 평가자로 만든 랭킹은 LLM 기반 리랭커를 평균 9~17위 지나치게 높게 매겼다(Fröbe et al., 2025). 또한 LLM으로 랭킹과 판정을 모두 하는 것은 순환적이며, 이는 "LLM 기반 평가에 의존하는 것이 시스템 랭킹을 어떻게 왜곡할 수 있는지를 보여 주는" 사례다(Clarke and Dietz, 2024). LLM 리랭커는 이 장에서처럼 사람의 판정으로 점수를 매긴다.

LLM 리랭커는 하나의 모델 버전과 하나의 프롬프트에 속한다. 다이제스트(정확한 가중치를 식별하는 해시), 양자화(가중치를 저장하는 정밀도), 프롬프트 문구가 다르면 모든 답이 달라질 수 있으며, 위치 편향이 있으므로 검색 순서가 바뀌어도 마찬가지다. LLM 녹화본의 매니페스트에는 다이제스트와 두 프롬프트가 그대로 기록되어 있다. 둘 중 하나라도 바뀌면 다시 측정한다.

익숙하지 않은 질의에서는 더 나쁠 수 있다. 22가지 리랭킹 방법을 비교한 연구에 따르면 LLM 리랭커는 익숙한 질의에서 더 잘하지만, "새로운 질의에 대한 일반화 능력은 일정하지 않다"(Abdallah et al., 2025). 모델의 학습 데이터보다 최근에 나온 상품과 스타일에 관한 질의를 주시한다.

모든 리랭커는 쇼핑객의 경로에 호출을 하나 더한다. 예산에서처럼 질의당 호출 수와 텍스트 양을 세고, 타임아웃을 설정하며, 리랭커가 제때 응답하지 않으면 검색의 순서로 되돌아간다.

작업 규칙: 모든 랭킹과 함께 특징과 모델 버전을 기록하고, 후보를 정보를 담지 않는 순서로 배치하고, 사람의 판정으로 점수를 매기고, judged@10을 확인하며, 각 리랭킹 단계에 시간 제한과 대체 경로를 둔다.

이 장이 내린 결정

마지막으로 변경된 판: 2026.11.8판

이 장은 아래 목록의 각 선택을 train 질의에서 내렸고, 상점의 검색을 바꿀 수 있는 것은 test 질의에서 성립이 확인된 계획된 비교뿐이었다.

5개의 계획된 비교는 각각 의미 기반 검색의 nDCG@10인 0.830에 대해 nDCG@10으로 수행했으며, 0.01 아래에서 성립이 확인되었다.

측정값 LambdaMART 교차 인코더 LLM 포인트와이즈 LLM 리스트와이즈 증류
nDCG@10 0.817 0.813 0.837 0.819 0.826
평균 차이 -0.012 -0.016 0.008 -0.010 -0.004
승 / 패 / 무 14, 12, 4 11, 12, 7 11, 8, 11 12, 12, 6 10, 13, 7
순열 p 0.260 0.178 0.241 0.580 0.720
성립 여부 no no no no no
t-검정 p 0.278 0.171 0.244 0.498 0.694
99% 구간 -0.044 to 0.011 -0.058 to 0.006 -0.007 to 0.026 -0.070 to 0.016 -0.034 to 0.017
t-검정 일치 yes yes yes yes yes
구간 일치 yes yes yes yes yes
  • 5개의 계획된 비교는 의미 기반 검색을 상대로 사람의 판정을 기준으로 수행했으며, 0.01 아래에서 성립이 확인되었다.
  • 후보: BM25와 의미 기반 검색의 상위 100개를 합한 집합이며, product_id 순서로 두고 점수가 같으면 product_id로 순서를 정한다.
  • LambdaMART: label_gain=[0, 1, 2], 판정되지 않은 후보는 등급 0, 학습률 0.1에 리프 3개이며, train에서 교차 검증으로 골랐고 eval_X와 eval_y를 통해 train에서 떼어 둔 질의로 조기 종료를 적용했다. 이 모델의 이득은 test에서 성립이 확인되지 않았으므로 사용하지 않는다.
  • 재순위 깊이: 의미 기반 검색의 상위 20개이며, LLM 기록에 맞추어 고정했다. train에서 교차 인코더의 곡선은 20과 30 사이에서 평평해진다. test에서 이득이 성립한 재순위기는 없으므로 어느 것도 사용하지 않는다.
  • LLM 재순위기는 사람의 판정만으로 점수를 매기며, 리스트와이즈 방식은 두 가지 입력 순서에서 확인한다. 이 방식의 답은 제시된 순서에 따라 달라진다.
  • 비용은 시간을 재지 않고 센다. 질의당 호출 수와 문자 수를 센다.
  • 증류: 교차 인코더의 점수를 학습하는 lightgbm 학생 모델이며, 검색 외에는 모델 호출이 없다. 이 모델은 주로 의미 기반 검색의 순서를 학습하고, 성립이 확인된 이득을 보이지 않으며, 사용하지 않는다.
  • 이겨야 할 순위기는 여전히 의미 기반 검색 단독이며, 어간 추출을 하지 않은 BM25를 기준선으로 둔다.
  • 이 장이 보여 준 것: 어떤 두 번째 단계도 제자리를 얻지 못했다. 어느 것도 성립이 확인된 이득을 보이지 않는다.

이 판의 이 장

2026.11.10판예제 통과 2026년 11월 19이 장에서 14개 예제개 중 14개가 통과한다

영어판 2026.11.10판을 번역한 것이다.

이 절 인용

형식

각 형식은 이 번역이 따르는 영문 원서를 출처로 표기한다.

모든 형식은 이 판을 명시한다. 링크는 현재 판을 열며, 여기서 절의 제목은 본문이 마지막으로 바뀐 시점을 알려 준다. 접근 날짜와 검색 날짜는 이 자료를 여는 날짜다.

이 페이지의 내용

11장 중 9장: Ranking

  1. 특징으로 각 후보 설명하기
  2. Learn a ranking from pairs of products
  3. Train LambdaMART with LightGBM
  4. Report LambdaMART on the test queries
  5. Rerank semantic retrieval with a cross-encoder
  6. Score each product with an LLM
  7. 20개 상품을 LLM 프롬프트 하나로 정렬하기
  8. Reverse the input to measure positional bias
  9. 예산에 맞춰 재순위화 깊이 정하기
  10. 교차 인코더를 작은 랭커로 증류하기
  11. 운영 환경에서 무엇이 잘못되는가
  12. 이 장이 내린 결정

맨 위로 돌아가기전체 장