人員與 LLM 的判斷
閱讀約需 78 分鐘 分鐘,13 節 中共有 11 個範例。
讀完本章,你將能撰寫判斷準則,用 kappa 與加權 kappa 衡量兩位評判者的一致程度,從混淆矩陣解讀 LLM 評判者的錯誤,並判斷 LLM 的判斷在哪些情況下可以取代人員的判斷,在哪些情況下不行。
本書中的每個分數都建立在判斷之上:人員閱讀在商店搜尋框中輸入的查詢,並對照它所回傳的商品,替這組配對評分。這些判斷就是 WANDS 的標籤,每個標籤都是最多三位受過訓練的標註者多數決的結果,而且存在缺漏:在測試查詢上,BM25 的前 10 名中有 22 個商品沒有人判斷過(計算未判斷的商品)。這些判斷也耗費人力,而且 WANDS 標註者之間的一致程度,在四個月的校準期間內逐步提高(預期評判者會有分歧)。LLM 能替任何一組它看得到的查詢與商品評分。在商店讓 LLM 擔任評判者之前,必須先知道 LLM 與商店的評判者有多常意見一致,以及它的評分是否會導向與人員相同的排序器決策。
LLM 的評分來自一份錄製結果:recordings/llm_judgments.py 在本機以溫度 0 及固定的種子,執行開放模型 gemma4:12b-it-q4_K_M 一次,並儲存其回答,因此範例透過 lib.recorded.llm_judgments() 讀取這些回答,而不需呼叫模型。我們選擇這個模型,是因為它是開放的,採用 Apache 2.0 授權,並且能在本機執行,所以任何人都可以重新執行這份錄製。JSON 結構描述(schema)將每個回答限制為三個標籤之一。這份錄製使用兩種提示詞替每組配對評分,分別是 short 與 guidelines(撰寫判斷準則),其配對取自兩個集合,本章稱之為池(pool)。
決定排序器優劣的分數仍然採用人員的評分。本章在測試查詢上規劃的比較,是第 3 章的三項比較,並以各提示詞的 LLM 評分重新計分,採用第 3 章的門檻 0.0167 進行成對的排列檢定:語意檢索對 BM25、混合式檢索對語意檢索,以及重新排序的混合式檢索對混合式檢索。
撰寫評判準則
本章將提供判斷的人,無論是真人或 LLM,一律稱為評判者,而 等級則是代表該判斷的數字。第 1 章將判斷定義為人所給的標註;本章把它擴大,也涵蓋 LLM 給的標註。判斷取決於背後的指示。評判準則說明每個等級的意義、如何在各等級之間做選擇,以及如何處理落在兩個等級之間的案例。
WANDS 公開了其標註者使用的準則:共八頁,採用 MIT 授權(標註準則)。針對每一組配對,準則列出評判者所看到的內容:查詢,以及商品的 ID、名稱、圖片與網頁。準則定義了三個等級,另外附上兩項說明:一項要求評判者先理解查詢,即「強烈建議先用 Google 搜尋」,另一項則說明要忽略拼字錯誤,並依原樣標註該配對。接著準則以範例教學,為每個等級給出查詢、商品與理由:沒有分隔板的鐵絲籃,對 wire basket with dividers 而言是部分相關,這是該商店自己的訓練查詢之一;而美容院用的抗疲勞地墊,對 salon chair 而言是不相關,因為顧客要找的是美容院座椅,而不是地墊。準則也會隨評判者的使用而改變。WANDS 的作者指出,其標註者之間的一致性在四個月內有所提高,他們列出的原因包括:每日討論相互衝突標註的例行作業,以及參考標註者的意見來修訂準則(Chen et al., 2022, 第 5 節)。某個網路搜尋引擎的評分者準則也是如此,其 2025 年 9 月 11 日版本長達 182 頁(Search Quality Rater Guidelines)。
該次記錄中的兩個提示詞,就是給 LLM 的評判準則。short 僅包含 WANDS 的三項定義;guidelines 保留這三項定義,並加上五條規則,因此請依該次記錄送出的原樣印出:
1"""Prints the model, the word count of each prompt and the guidelines prompt verbatim, from the recording's manifest. The prose's point rests on the rules at the end of that prompt."""2from lib.recorded import manifest34recording = manifest("llm-judgments") # the recording's model, settings and both prompts, verbatim5short, guidelines = recording["prompts"]["short"], recording["prompts"]["guidelines"]67print("model:", recording["model"])8print("what the model saw:", recording["sees"])9print("words in the short prompt:", len(short.split()))10print("words in the guidelines prompt:", len(guidelines.split()))11print()12print(guidelines)
python print-judging-guidelines.pymodel: gemma4:12b-it-q4_K_M
what the model saw: lib.store text plus attributes; no images
words in the short prompt: 59
words in the guidelines prompt: 206
You judge product search results for an online furniture and home-goods store. A shopper typed the query and was shown the product. Decide how well the product matches what the shopper asked for.
Exact: the product fully matches the search query.
Partial: the product only matches the target entity of the query, but does not satisfy the modifiers for the query.
Irrelevant: the product is not relevant to the query.
Guidelines:
- First find the target entity: the kind of product the query asks for (a sofa, a lamp, a rug).
- Then find the modifiers: everything else the query asks for, such as colour, size, material, style, brand or a feature.
- Exact needs the right kind of product and every modifier. Partial is the right kind of product with at least one modifier missing or contradicted. Irrelevant is the wrong kind of product, even when it shares words with the query, such as an accessory for the product asked for.
- Read misspelled queries as the shopper meant them.
- Judge only from the product's text and attributes. If a modifier isn't mentioned, decide from what is written, and don't assume it is met.
Query: {query}
Product:
{product}
Answer with the label in JSON.
guidelines 提示詞是兩者中較長的一個,有 206 個字,而 short 則為 59 個字。它的前兩條規則闡明了 WANDS 對部分相關的定義:先找出實體,再看修飾語。配件規則處理了 WANDS 自己的標籤也出現分歧的案例:對於 clear shower curtain,WANDS 將同一組浴簾掛鉤列了兩次,一次標為部分相關,一次標為不相關(預期評判者會有分歧)。拼字錯誤規則則依循 WANDS 的說明。最後一條規則在 WANDS 的準則中沒有對應項目,因為該商店的資料不含圖片,所以該次記錄只向 LLM 呈現文字。兩個提示詞都沒有範例,而 WANDS 的準則正是仰賴範例。
實用原則: 將準則寫成定義、程序、為每個等級附上理由的範例,以及針對評判者曾經產生分歧的每一種案例所訂的規則。記錄分歧,並在某一類分歧不斷重現時修訂準則。給 LLM 評判者的準則應與給真人的相同,再加上一條針對它看不到的內容所訂的規則,然後衡量它實際的表現,因為你寫下的規則,並不等於評判者會遵守的規則。
以 kappa 衡量超出機率的一致性
在信任第二位評審之前,先衡量它與第一位評審有多常意見一致。兩位評審對同一組配對給出相同等級的比例,也就是觀察一致率(observed agreement),是最直觀的數字,但它會讓結果看起來比實際好。兩位各自獨立亂猜的評審,也會有一部分配對因機率而一致;當兩者都常使用同一個標籤時,一致的配對還會更多。Cohen's kappa 會針對機率修正觀察一致率:它拿觀察一致率與「各評審保持自己的標籤比例,但彼此獨立選擇」時預期的一致率相比較。kappa 為 1 表示完全一致,0 則表示「沒有超出機率預期的一致性」(scikit-learn 文件):
kappa = (observed - expected) / (1 - expected)
expected = sum over grades g of (share of pairs judge A gives g) × (share of pairs judge B gives g)
用來衡量一致性的訓練池(train pool),包含人員針對訓練查詢所評判的配對,並依標籤平均抽取:每個查詢的每種標籤最多 4 筆,共 550 個配對。測試池(test pool)包含 LLM 為了替商店的排序器評分而要評等的配對:對每個測試查詢,取第 1 章四種排序器前 10 名中的每一項商品,不論是否已由人員評判,共 590 個配對。這四種排序器是 BM25、以 Snowflake/snowflake-arctic-embed-m-v1.5 的嵌入向量進行的語意檢索、兩者的混合式排序,以及由交叉編碼器 Alibaba-NLP/gte-reranker-modernbert-base 對混合式結果重新排序的版本。對 LLM 評審的每項衡量,只適用於進行衡量的那個池,因此本章每次都會指明是哪個池。
問題在於,該次紀錄中的 LLM 有多常以商店評審的方式為配對評等。使用short 提示(僅包含 WANDS 的三項定義)的訓練池可以回答這個問題。以手工建立 3×3 的計數表即可算出 kappa,再用 scikit-learn 的 cohen_kappa_score 檢查結果:
1"""Builds the 3×3 table of the store's grades against the short prompt's on the train pool, then computes observed agreement, chance agreement and kappa from it. The prose's point rests on the printed kappa by hand, which matches the scikit-learn line."""2import numpy as np3from sklearn.metrics import cohen_kappa_score45from lib.agreement import graded_pairs67pairs = graded_pairs("train") # up to 4 human-judged pairs per label per train query8# the grades are nullable integers; plain ones compare and count the way the formulas below expect9human, llm = pairs["human"].astype(int), pairs["short"].astype(int)10grades = [2, 1, 0] # exact, partial, irrelevant11counts = np.array([[((human == row) & (llm == column)).sum() for column in grades] for row in grades])12print("pairs:", counts.sum())13print("rows are the human grades 2, 1, 0 and columns the LLM's, short prompt")14print(counts)1516observed = np.trace(counts) / counts.sum() # the share of pairs on the diagonal17human_share, llm_share = counts.sum(axis=1) / counts.sum(), counts.sum(axis=0) / counts.sum()18expected = (human_share * llm_share).sum() # agreement if the two judged independently19kappa = (observed - expected) / (1 - expected)20print(f"observed agreement, short prompt: {observed:.3f}")21print(f"agreement expected by chance, short prompt: {expected:.3f}")22print(f"kappa by hand, short prompt: {kappa:.3f}")23print(f"kappa from scikit-learn, short prompt: {cohen_kappa_score(human, llm):.3f}")2425other = pairs["guidelines"].astype(int)26print(f"observed agreement, guidelines prompt: {(human == other).mean():.3f}")27print(f"kappa from scikit-learn, guidelines prompt: {cohen_kappa_score(human, other):.3f}")
python kappa-by-hand.pypairs: 550
rows are the human grades 2, 1, 0 and columns the LLM's, short prompt
[[102 85 9]
[ 22 132 46]
[ 3 42 109]]
observed agreement, short prompt: 0.624
agreement expected by chance, short prompt: 0.337
kappa by hand, short prompt: 0.432
kappa from scikit-learn, short prompt: 0.432
observed agreement, guidelines prompt: 0.633
kappa from scikit-learn, guidelines prompt: 0.454
LLM 與商店評審對 0.624 的訓練配對給出相同等級,而僅憑機率就會產生 0.337。kappa 先從觀察一致率中減去這個機率份額,再除以超出機率之上所剩的空間,得到 short 的 0.432,與 scikit-learn 的 0.432 相同。
guidelines 與評審一致的頻率略高於 short,在 0.633 的配對上一致,kappa 為 0.454。
這樣算好嗎?把 kappa 稱為「中等」或「相當高」的量表,是特定領域的慣例。Artstein 與 Poesio 的調查發現,為特定目的選擇適當的一致性水準,「仍然近乎一門黑魔法」(Artstein and Poesio, 2008)。
已發表的數字提供的是背景,而不是門檻。WANDS 的標註者彼此之間的 kappa,在最初兩個月為 0.467,到第四個月為 0.826(Chen et al., 2022, Table 2)。
Thomas 等人針對 TREC(Text REtrieval Conference 的共同評測)的分層配對樣本發現,依提示不同,GPT-4 與評估者之間的 kappa 低至 0.20,高至 0.64。兩組受過訓練的人類評估者,在 TREC ad hoc 文件上對自己第二次評判的結果,達到 0.52(Cormack et al., 1998, as reported by Thomas et al., 2024)。每個數字都來自不同的樣本,而且 kappa 取決於樣本的標籤組成。
實用原則: 報告 kappa 時,要並列觀察一致率,絕不只報告觀察一致率。以人員針對訓練查詢評判的配對來衡量,並將所用的任何門檻標示為經驗法則。若評審超過兩位或有缺漏的標籤,請使用 Krippendorff's alpha,它比較觀察到的不一致與機率造成的不一致,並能處理任意數量的觀察者與不完整的資料(Krippendorff, 2011)。
以加權 kappa 給予差一點的判斷部分分數
Kappa 把每一種不一致都視為相同。商店的等級是有順序的,完全相關高於部分相關,部分相關又高於不相關,而把完全相關的商品判為部分相關,錯誤比判為不相關小。在 nDCG@10(為一個排名評分)中,前者損失的增益只有後者的一半:差一個等級,而不是兩個等級。加權 kappa(weighted kappa)依距離為每一種不一致指定權重,再將加權後的不一致,與偶然預期的加權不一致相比較(Artstein and Poesio, 2008, section 2.6)。採用線性權重時,相差一個等級的不一致權重為 0.5,相差兩個等級的權重為 1:
weighted kappa = 1 - (sum of weight × observed count) / (sum of weight × count expected by chance)
若每一種不一致的權重都是 1,此公式得到的就是一般的 kappa。scikit-learn 的 cohen_kappa_score 以這種方式計算三種情形:預設的權重為 1,weights="linear" 採用等級之間的距離,weights="quadratic" 則採用距離的平方(scikit-learn 原始碼)。權重整體縮放不會改變結果。
針對兩個提示詞,這個範例在〈衡量超越偶然的一致性〉中均勻抽取的訓練配對上,以手動方式建立線性權重,再將每個結果與 scikit-learn 核對:
1"""Computes linear weighted kappa by hand for both prompts on the train pool and checks each against scikit-learn. The prose's point rests on the two linear weighted kappa lines, which match scikit-learn's, and on the one-grade-apart shares beside them."""2import numpy as np3from sklearn.metrics import cohen_kappa_score, confusion_matrix45from lib.agreement import PROMPTS, graded_pairs67pairs = graded_pairs("train") # up to 4 human-judged pairs per label per train query8grades = [2, 1, 0] # exact, partial, irrelevant9cost = np.abs(np.subtract.outer(grades, grades)) / 2 # 0 on the diagonal, 0.5 one grade apart, 1 two apart10print("disagreement weights, rows and columns 2, 1, 0")11print(cost)1213for prompt in PROMPTS:14 human, llm = pairs["human"].astype(int), pairs[prompt].astype(int)15 # labels fixes the rows and columns to 2, 1, 0 in that order, not the order the grades first appear16 counts = confusion_matrix(human, llm, labels=grades) # rows human, columns LLM17 expected = np.outer(counts.sum(axis=1), counts.sum(axis=0)) / counts.sum() # counts if independent18 weighted = 1 - (cost * counts).sum() / (cost * expected).sum()19 near = cost == 0.5 # cells one grade apart20 print()21 # row 0 is exact and column 2 is irrelevant, so these two corners are the far misses22 print(f"pairs two grades apart, {prompt} prompt: {counts[0, 2] + counts[2, 0]}")23 print(f"disagreements one grade apart, share, {prompt} prompt: {counts[near].sum() / counts[cost > 0].sum():.3f}")24 print(f"one grade apart, share expected by chance, {prompt} prompt: {expected[near].sum() / expected[cost > 0].sum():.3f}")25 print(f"kappa, {prompt} prompt: {cohen_kappa_score(human, llm):.3f}")26 print(f"linear weighted kappa by hand, {prompt} prompt: {weighted:.3f}")27 print(f"linear weighted kappa from scikit-learn, {prompt} prompt: "28 f"{cohen_kappa_score(human, llm, weights='linear'):.3f}")
python weighted-kappa-by-hand.pydisagreement weights, rows and columns 2, 1, 0
[[0. 0.5 1. ]
[0.5 0. 0.5]
[1. 0.5 0. ]]
pairs two grades apart, short prompt: 12
disagreements one grade apart, share, short prompt: 0.942
one grade apart, share expected by chance, short prompt: 0.742
kappa, short prompt: 0.432
linear weighted kappa by hand, short prompt: 0.523
linear weighted kappa from scikit-learn, short prompt: 0.523
pairs two grades apart, guidelines prompt: 30
disagreements one grade apart, share, guidelines prompt: 0.851
one grade apart, share expected by chance, guidelines prompt: 0.664
kappa, guidelines prompt: 0.454
linear weighted kappa by hand, guidelines prompt: 0.531
linear weighted kappa from scikit-learn, guidelines prompt: 0.531
加權使 short 的 kappa 從 0.432 升到 0.523,使 guidelines 的 kappa 從 0.454 升到 0.531,且兩者都與 scikit-learn 的結果相符。數值會上升,是因為不一致落在相差一個等級的頻率高於偶然的預期:short 為 0.942,偶然預期為 0.742;guidelines 為 0.851,偶然預期為 0.664。
兩個提示詞在嚴重偏差上也有差異,也就是相差兩個等級的配對:short 為 12,guidelines 為 30。哪一個提示詞的一致性較好,則是區間要回答的問題。
**實用原則:**對於有順序的等級,在一般 kappa 之外也報告線性加權 kappa,並另外計算嚴重偏差的數量。每一種不一致的權重,應依其造成的增益差距而定。本書採用線性增益時,這就是線性權重。若採用 0、1、3 的指數增益(選擇指標),差距為 1、2、3,這是 scikit-learn 的兩個選項都無法提供的,因此要像範例建立線性矩陣那樣,自行建立該矩陣。
對照標籤組成來解讀 kappa
Kappa 的校正取決於各個評審使用各標籤的頻率,因此同一位評審在標籤組成不同的樣本上,會得到不同的 kappa。當某一個標籤占多數時,觀察一致率可能很高,kappa 卻仍然偏低。Artstein 與 Poesio 稱之為盛行率問題(prevalence problem):「標註者可能在很高比例的項目上達成一致,而且標註本身在很高程度上確實正確,但信度係數仍然偏低」(Artstein and Poesio, 2008)。
該次記錄對訓練池採平均抽樣,每個訓練查詢的每種標籤最多取 4 組配對,因此訓練池中完全相關與不相關的配對,比資料庫的判斷結果所含的更多。要了解這對 kappa 有何影響,可以替每組配對加權,使訓練池具有所有訓練查詢判斷結果的標籤組成,然後重新測量:
1"""Measures each prompt's agreement with the store's judges on the train pool, as drawn and reweighted to the label mix of all the train judgments. The prose's point rests on the kappa lines for each mix and on the higher-kappa lines, which change with the mix."""2import numpy as np3from sklearn.metrics import cohen_kappa_score45from lib.agreement import LABELS, PROMPTS, graded_pairs6from lib.store import load_labels, load_queries78pairs = graded_pairs("train") # drawn evenly: up to 4 pairs per label per train query9drawn = pairs["human"].value_counts(normalize=True)10labels = load_labels()11# every judged label for the train queries, not only the pairs that were drawn12natural = labels[labels["query_id"].isin(load_queries("train").index)]["grade"].value_counts(normalize=True)13top = graded_pairs("test")["human"].dropna().value_counts(normalize=True) # the rankers' top 10s, judged pairs14for grade, label in LABELS.items():15 print(f"{label} share, as drawn: {drawn[grade]:.3f}")16 print(f"{label} share, in the train judgments: {natural[grade]:.3f}")17 print(f"{label} share, judged test-pool pairs: {top[grade]:.3f}")1819# each pair counts for its label's share in the judgments divided by its share as drawn20weight = pairs["human"].map(natural / drawn).astype(float) # count each pair as if drawn in that mix21kappas = {}22for prompt in PROMPTS:23 human, llm = pairs["human"].astype(int), pairs[prompt].astype(int)24 print()25 print(f"observed agreement, {prompt} prompt, as drawn: {(human == llm).mean():.3f}")26 print(f"kappa, {prompt} prompt, as drawn: {cohen_kappa_score(human, llm):.3f}")27 print(f"observed agreement, {prompt} prompt, train judgments' mix: {np.average(human == llm, weights=weight):.3f}")28 # sample_weight counts each pair that many times; it is not the linear weights of the last section29 kappas[prompt] = cohen_kappa_score(human, llm), cohen_kappa_score(human, llm, sample_weight=weight)30 print(f"kappa, {prompt} prompt, train judgments' mix: {kappas[prompt][1]:.3f}")31print()32for position, mix in enumerate(["as drawn", "train judgments' mix"]):33 # on a tie, max returns the first prompt, short34 print(f"higher kappa, {mix}: {max(PROMPTS, key=lambda prompt: kappas[prompt][position])}")
python kappa-under-two-label-mixes.pyexact share, as drawn: 0.356
exact share, in the train judgments: 0.234
exact share, judged test-pool pairs: 0.543
partial share, as drawn: 0.364
partial share, in the train judgments: 0.613
partial share, judged test-pool pairs: 0.427
irrelevant share, as drawn: 0.280
irrelevant share, in the train judgments: 0.153
irrelevant share, judged test-pool pairs: 0.030
observed agreement, short prompt, as drawn: 0.624
kappa, short prompt, as drawn: 0.432
observed agreement, short prompt, train judgments' mix: 0.635
kappa, short prompt, train judgments' mix: 0.370
observed agreement, guidelines prompt, as drawn: 0.633
kappa, guidelines prompt, as drawn: 0.454
observed agreement, guidelines prompt, train judgments' mix: 0.573
kappa, guidelines prompt, train judgments' mix: 0.345
higher kappa, as drawn: guidelines
higher kappa, train judgments' mix: short
部分相關在原樣抽取的訓練池中占 0.364,在訓練判斷結果中則占 0.613,所以重新加權後,部分相關的配對權重較高,完全相關與不相關的配對權重較低。
在判斷結果的組成下,short 的觀察一致率由 0.624 變為 0.635,kappa 由 0.432 變為 0.370。至於 guidelines,觀察一致率由 0.633 變為 0.573,kappa 由 0.454 變為 0.345。
kappa 較高的提示詞,在原樣抽取時是 guidelines,在判斷結果的組成下則是 short。
已發表的數字都帶有各自的標籤組成。我們找到的唯一一項以 WANDS 評估 LLM 評審的研究,在三種標籤間平均抽取了 5,000 組查詢與商品的配對。它報告的準確率,從最佳開放模型(700 億參數)的 0.726,到最佳封閉模型的 0.837,皆為各自最佳設定下的結果,並且每組配對都檢索了 16 個範例作為示範(Sachdev et al., 2025, Table 3)。該資料庫的提示詞沒有示範範例。這個準確率是在平衡樣本上的觀察一致率,與訓練池的情況相同,因此兩個數字都無法套用到資料庫判斷結果的標籤組成上。
資料庫將要請 LLM 判斷的配對,是各排序器放入前 10 名的商品,也就是測試池。在其已判斷的配對中,完全相關占 0.543,部分相關占 0.427,不相關占 0.030。這是第三種組成,也是另一種類型的配對,解讀 LLM 的錯誤方向會對此進行測量。
實用原則: 每報告一個 kappa,都要同時報告標籤組成,並且不要比較來自不同組成樣本的 kappa,無論是自己的或已發表的。我們建議依照評審的實際用法來抽取一致性樣本:如果它將用來評分排序器的前 10 名,就從訓練查詢的前 10 名中抽取配對。
解讀 LLM 錯誤的方向
Kappa說明兩位評審的一致程度,而不是兩者如何不同。混淆矩陣(confusion matrix)則呈現兩位評審的差異:這是一張計數表,每一列對應商店的一種評分,每一欄對應 LLM 的一種評分,順序皆為完全相關、部分相關、不相關。對角線上方表示 LLM 給的分數低於商店的評審;對角線下方則表示高於商店的評審。錯誤多落在下方的評審是寬鬆的(lenient),多落在上方的則是嚴格的(strict)。
數項研究發現 LLM 評審偏寬鬆。在 TREC 2024 的 RAG 賽道(評估檢索增強生成,即根據檢索到的文字撰寫答案)中,整體而言人類評估者似乎比開源 LLM 評審 UMBRELA 更嚴格(Upadhyay et al., 2024a)。Arabzadeh 與 Clarke 指出,以 GPT-4o 為基礎建構的 UMBRELA(Upadhyay et al., 2024b),在 TREC 2019 至 2021 年的 Deep Learning 賽道中,標為完全相關的文件比人類多出超過 26%,分別為 3,063 份與 2,429 份(Arabzadeh and Clarke, 2025a)。Yu et al. 則指出 LLM 評審「一貫給出偏高的相關性分數」(Yu et al., 2026)。
並非所有研究都得到相同結論。Thomas et al. 以其 32 種提示詞之一,在依標籤分層的樣本上發現,LLM 比 TREC 的評估者更常將配對判為不相關,比例為 44% 對 33%(Thomas et al., 2024)。
為每種提示詞在每個評分池上各做一張矩陣,可以看出 LLM 往哪個方向出錯,以及這個方向在從均衡的訓練配對轉到排序器的前 10 名時是否仍然成立:訓練池依標籤均勻抽樣,測試池中已評分的配對則是第 1 章四個排序器的前 10 名中須由 LLM 評分的部分。測試池的矩陣報告的是測試查詢的情形,並不用來選擇提示詞:
1"""Prints the confusion matrix of the store's grades against each prompt's grades, on the train pool and on the judged test pairs, with the counts of lower and higher grades. The prose's point rests on the 'stricter' line for each pool, read with the largest error printed beside it."""2import pandas as pd3from sklearn.metrics import cohen_kappa_score45from lib.agreement import LABELS, PROMPTS, graded_pairs67order = list(LABELS.values()) # rows: the store's grades; columns: the LLM's; both exact first8for pool in ["train", "test"]:9 pairs = graded_pairs(pool).dropna(subset=["human"]) # in the test pool, the pairs people judged10 human = pairs["human"].astype(int)11 print(f"\njudged pairs, {pool} pool: {len(pairs)}\nmean human grade, {pool} pool: {human.mean():.3f}")12 for prompt in PROMPTS:13 llm, name = pairs[prompt].astype(int), f"{pool} pool, {prompt} prompt"14 table = pd.crosstab(human.map(LABELS).rename("human"), llm.map(LABELS).rename(f"llm, {name}"))15 table = table.reindex(index=order, columns=order, fill_value=0)16 errors = {f"{row} called {column}": table.at[row, column] for row in order for column in order if row != column}17 # on a tie, max returns the first cell, in row order18 largest = max(errors, key=errors.get)19 print(table)20 print(f"kappa, {name}: {cohen_kappa_score(human, llm):.3f}\nmean llm grade, {name}: {llm.mean():.3f}")21 print(f"llm grade higher than human, {name}: {(llm > human).sum()}")22 print(f"llm grade lower than human, {name}: {(llm < human).sum()}")23 # a tie counts as not stricter24 print(f"stricter than the store's judges, {name}: {'yes' if (llm < human).sum() > (llm > human).sum() else 'no'}")25 print(f"largest error, {name}: {largest} ({errors[largest]})")
python llm-confusion-matrix.py
judged pairs, train pool: 550
mean human grade, train pool: 1.076
llm, train pool, short prompt exact partial irrelevant
human
exact 102 85 9
partial 22 132 46
irrelevant 3 42 109
kappa, train pool, short prompt: 0.432
mean llm grade, train pool, short prompt: 0.933
llm grade higher than human, train pool, short prompt: 67
llm grade lower than human, train pool, short prompt: 140
stricter than the store's judges, train pool, short prompt: yes
largest error, train pool, short prompt: exact called partial (85)
llm, train pool, guidelines prompt exact partial irrelevant
human
exact 118 53 25
partial 30 98 72
irrelevant 5 17 132
kappa, train pool, guidelines prompt: 0.454
mean llm grade, train pool, guidelines prompt: 0.862
llm grade higher than human, train pool, guidelines prompt: 52
llm grade lower than human, train pool, guidelines prompt: 150
stricter than the store's judges, train pool, guidelines prompt: yes
largest error, train pool, guidelines prompt: partial called irrelevant (72)
judged pairs, test pool: 564
mean human grade, test pool: 1.512
llm, test pool, short prompt exact partial irrelevant
human
exact 252 53 1
partial 57 177 7
irrelevant 2 6 9
kappa, test pool, short prompt: 0.571
mean llm grade, test pool, short prompt: 1.521
llm grade higher than human, test pool, short prompt: 65
llm grade lower than human, test pool, short prompt: 61
stricter than the store's judges, test pool, short prompt: no
largest error, test pool, short prompt: partial called exact (57)
llm, test pool, guidelines prompt exact partial irrelevant
human
exact 261 41 4
partial 76 133 32
irrelevant 3 4 10
kappa, test pool, guidelines prompt: 0.470
mean llm grade, test pool, guidelines prompt: 1.521
llm grade higher than human, test pool, guidelines prompt: 83
llm grade lower than human, test pool, guidelines prompt: 77
stricter than the store's judges, test pool, guidelines prompt: no
largest error, test pool, guidelines prompt: partial called exact (76)
在訓練池上,LLM 是否比商店的評審更嚴格?使用 short 時:yes。使用 guidelines 時:yes。使用 short 時,它給出較低分的配對有 140 組,較高分的有 67 組;使用 guidelines 時,較低分的有 150 組,較高分的有 52 組。最大的誤差在 short 下為 exact called partial (85),在 guidelines 下為 partial called irrelevant (72)。
在測試池上,使用 short 時答案是 no,使用 guidelines 時是 no。使用 short 時,LLM 給較高分的配對有 65 組,較低分的有 61 組;使用 guidelines 時,較高分的有 83 組,較低分的有 77 組。它的平均評分在 short 下為 1.521,在 guidelines 下為 1.521,而評審的平均為 1.512。它的最大誤差在 short 下為 partial called exact (57),在 guidelines 下為 partial called exact (76)。
Kappa 在 short 下為 0.571,在 guidelines 下為 0.470;訓練池上則分別為 0.432 與 0.454。方向、最大誤差與 kappa 都隨評分池而變,正如標籤組成所警示的。比較 LLM 看到的內容找到一個同時符合兩個評分池的模式。
錯誤的方向也無法說明排序器的分數會如何變化。寬鬆的評審會把更多商品算作相關,這會提高 precision@10(計算相關商品數)。nDCG@10 除以由同一組評分建構的理想前 10 名(為一個排序評分),因此提高評分也可能同時抬高理想值,淨效果可能朝任一方向,且因排序器而異。排序器順序的檢查在測試池上量測這一點。
實用原則: 先看混淆矩陣,再看 kappa,且要用與你將來使用評審的方式相同的方法抽取配對,並指出該評分池的方向與最大誤差。不要把一個評分池的方向帶到另一個評分池,也不要據此預測排序器的 nDCG@10。排序器之間應在相同的評分下相互比較,切勿拿 LLM 評分下的分數與人工評分下的分數相比。
比較 LLM 所見與評審所見
評審只能依據呈現在眼前的證據評分。WANDS 的評審看到的是查詢,以及產品的 ID、名稱、圖片和網頁,而評分指引要求他們先理解查詢:「Strongly recommend google it first」(評分指引)。錄製的內容則讓 LLM 看到每項產品的名稱、類別、分類、描述和屬性,格式由 lib.agreement.as_the_llm_saw 寫出。商店的樣本會把描述截斷為約 400 個字元(資料卡),也不含圖片,因此錄製時只送給 LLM 純文字。部分分歧可能來自這個落差,而不是來自 LLM 的評分能力;這是我們的推論。其中一項後果可以檢驗:若 LLM 依據文字內容評分,它應該會給文字中沒有寫出查詢所要求內容的完全相關(exact)產品較低的評分,並給文字中有寫出的部分相關(partial)產品較高的評分。
這項檢查取一組訓練配對、測試查詢 clear shower curtain,以及兩個資料池中所有已評分的完全相關與部分相關配對(錄製內容)。它會在 LLM 所見的內容中尋找每個查詢詞,並以 第 2 章的檢查 的方式將內容切分為小寫單字:
1"""Checks whether the LLM's grades follow the words in each product's text, for one train pair, for the clear shower curtain products, and across both pools. The prose's point rests on the 'share graded lower' and 'share graded higher' lines, which compare pairs whose text has every query word with pairs that miss one."""2import re34from lib.agreement import PROMPTS, as_the_llm_saw, graded_pairs5from lib.store import load_products, load_queries678def words(text):9 """Lowercase word tokens, as chapter 2 matches query words against product text."""10 # runs of letters, digits and underscores, so 7qt is one word and '7 qt' is two11 return set(re.findall(r"\w+", text.lower()))121314products, queries = load_products(), load_queries()15# query 163 is a train query, and product 20718 is the pair the prose follows16print("query:", queries.at[163, "query"]) # a train query17print(as_the_llm_saw(products.loc[20718]))18print("grades from people, short, guidelines:", *graded_pairs("train").set_index(["query_id", "product_id"]).loc[(163, 20718)])1920curtains = graded_pairs("test").query("query_id == 246") # clear shower curtain: every product in a top 1021# whole words only, so 'unclear' doesn't count as saying clear22says = curtains["product_id"].map(lambda product_id: "clear" in words(as_the_llm_saw(products.loc[product_id])))23for kind, group in [("say clear", curtains[says]), ("never say clear", curtains[~says])]:24 print(f"\nclear shower curtain products that {kind}: {len(group)}")25 for column, judge in [("human", "people"), ("short", "the short prompt"), ("guidelines", "the guidelines prompt")]:26 print(f"products that {kind}, graded exact by {judge}: {(group[column] == 2).sum()}")2728for pool in ["train", "test"]:29 pairs = graded_pairs(pool).dropna(subset=["human"]) # in the test pool, the pairs people judged30 # subset: every query word appears in the text the LLM saw31 pairs["every word"] = [words(queries.at[query_id, "query"]) <= words(as_the_llm_saw(products.loc[product_id]))32 for query_id, product_id in zip(pairs["query_id"], pairs["product_id"])]33 # share of each group that the LLM grades below people (exact) or above them (partial), for each prompt34 for label, grade, every_word, direction in [("exact", 2, False, "lower"), ("exact", 2, True, "lower"),35 ("partial", 1, True, "higher"), ("partial", 1, False, "higher")]:36 group = pairs[(pairs["human"] == grade) & (pairs["every word"] == every_word)]37 kind = f"{label} {pool} pairs {'with every query word' if every_word else 'missing a query word'}"38 print(f"\n{kind}: {len(group)}")39 for prompt in PROMPTS:40 moved = group[prompt] < grade if direction == "lower" else group[prompt] > grade41 print(f"share graded {direction}, {prompt} prompt, {kind}: {moved.mean():.3f}")
python llm-grades-follow-written-words.pyquery: dual monitor stand
Name: abramowski monitor stand
Class: Desktop Organizers & Desk Pads
Category:
Description: the monitor stand riser can help !
Attributes: color: white; material: wood; style: modern; width: 23.5; height: 4.64; depth: 9.2
grades from people, short, guidelines: 2 1 1
clear shower curtain products that say clear: 9
products that say clear, graded exact by people: 5
products that say clear, graded exact by the short prompt: 7
products that say clear, graded exact by the guidelines prompt: 7
clear shower curtain products that never say clear: 9
products that never say clear, graded exact by people: 1
products that never say clear, graded exact by the short prompt: 0
products that never say clear, graded exact by the guidelines prompt: 0
exact train pairs missing a query word: 149
share graded lower, short prompt, exact train pairs missing a query word: 0.584
share graded lower, guidelines prompt, exact train pairs missing a query word: 0.483
exact train pairs with every query word: 47
share graded lower, short prompt, exact train pairs with every query word: 0.149
share graded lower, guidelines prompt, exact train pairs with every query word: 0.128
partial train pairs with every query word: 8
share graded higher, short prompt, partial train pairs with every query word: 0.875
share graded higher, guidelines prompt, partial train pairs with every query word: 0.875
partial train pairs missing a query word: 192
share graded higher, short prompt, partial train pairs missing a query word: 0.078
share graded higher, guidelines prompt, partial train pairs missing a query word: 0.120
exact test pairs missing a query word: 128
share graded lower, short prompt, exact test pairs missing a query word: 0.305
share graded lower, guidelines prompt, exact test pairs missing a query word: 0.281
exact test pairs with every query word: 178
share graded lower, short prompt, exact test pairs with every query word: 0.084
share graded lower, guidelines prompt, exact test pairs with every query word: 0.051
partial test pairs with every query word: 42
share graded higher, short prompt, partial test pairs with every query word: 0.548
share graded higher, guidelines prompt, partial test pairs with every query word: 0.667
partial test pairs missing a query word: 199
share graded higher, short prompt, partial test pairs missing a query word: 0.171
share graded higher, guidelines prompt, partial test pairs missing a query word: 0.241
對於訓練查詢 dual monitor stand,商店的評審將 abramowski monitor stand 評為完全相關,而兩種提示都將它評為部分相關(依序為 2 1 1);LLM 所見的內容中,沒有任何地方說明這個支架可容納兩台螢幕。
對於 clear shower curtain,LLM 給的完全相關評分會隨該詞而變動。在測試池中,LLM 能讀到 clear 一詞的 9 項產品裡,評審評為完全相關的有 5,short 提示評為完全相關的有 7,guidelines 提示評為完全相關的有 7。在從未出現該詞的 9 項產品中,評審評為完全相關的有 1,short 提示評為完全相關的有 0,guidelines 提示評為完全相關的有 0。
就整個資料池統計,這個模式依然成立。使用 short 提示時,文字中缺少某個查詢詞的完全相關配對,LLM 給予較低評分的比例在訓練池為 0.584,在測試池為 0.305;文字中具備所有查詢詞的完全相關配對則分別為 0.149 和 0.084。
在測試池中,文字中具備所有查詢詞的部分相關配對獲得較高評分的比例為 0.548,缺少某個詞的配對則為 0.171。在測試池中,guidelines 提示也有同樣的分野:完全相關配對為 0.281 對 0.051,部分相關配對為 0.667 對 0.241。
兩個資料池中,常見的情況並不相同。訓練池有 149 組缺少某個詞的完全相關配對,以及 8 組具備所有詞的部分相關配對;測試池則分別為 128 和 42。依我們的解讀,這就是錯誤方向在兩個資料池之間不同的原因。這項檢查相當粗略:複數形或拼字錯誤也會被算作缺少該詞。guidelines 提示要求降低評分:「If a modifier isn't mentioned, decide from what is written, and don't assume it is met.」而 short 提示沒有這樣的規則,同樣會降低評分。
實用原則: 請將你的評審所依據、且可轉為文字的每個欄位都提供給 LLM 評審,例如屬性和完整描述,並寫明它看不到哪些內容。在把分歧算作 LLM 的錯誤之前,先檢查證據是否出現在它所見的內容中。若你的評審所依據的證據是視覺性的,請衡量 LLM 是否會給完全相關的產品較低的評分(如本例在兩個資料池中的情形),或者在模型支援圖片時,直接提供圖片給它。
衡量提示詞對標籤的影響程度
LLM 評審是模型與提示詞的組合。Thomas 等人針對同一提示詞的 42 種改寫版本進行比較,發現平均 kappa 的分布從 0.50(他們稱之為中等一致性)一路到 0.72。他們提醒,任何單一提示詞的分數,都只是評審可能達到的範圍中的一次抽樣(Thomas et al., 2024)。
Arabzadeh 與 Clarke 在 TREC Deep Learning 資料集上,以三個 LLM 和 72 種提示詞進行實驗,發現二元式與成對式方法對提示詞措辭最不敏感,而分級相關性判斷(也就是商店所採用的類型)則對措辭高度敏感(Arabzadeh and Clarke, 2025b)。沒有任何單一提示詞對所有 LLM 都是最佳,不過 UMBRELA 在不同 LLM 的分級評估上表現一直不錯。
這兩種提示詞共用相同的定義,而 guidelines 在其上再加五條規則(撰寫評判指引)。錄製資料中的每個資料池都會顯示兩種提示詞在哪些地方不一致,而訓練資料池則可顯示能否判斷哪一種提示詞與商店的評審者一致性較高。同一查詢的配對並非彼此獨立,因此區間是以整個訓練查詢為單位重新抽樣,如同第 3 章的 bootstrap對查詢重新抽樣一樣:
1"""Counts where the two prompts disagree, their kappa with each other, and how far their kappas with people differ across resampled train queries. The prose's point rests on the 95% interval line, which includes 0 for this recording."""2import numpy as np3import pandas as pd4from sklearn.metrics import cohen_kappa_score56from lib.agreement import LABELS, graded_pairs7from lib.store import load_products, load_queries8910def kappa_gap(sample):11 """Kappa with people under the guidelines prompt, minus kappa under the short prompt."""12 human = sample["human"].astype(int)13 return cohen_kappa_score(human, sample["guidelines"].astype(int)) - cohen_kappa_score(human, sample["short"].astype(int))141516order = list(LABELS.values()) # exact, partial, irrelevant17for pool in ["train", "test"]:18 # the test pool includes pairs nobody judged, so these counts cover every pair19 pairs = graded_pairs(pool)20 print(f"\n{pool} pairs where the prompts disagree: {(pairs['short'] != pairs['guidelines']).sum()} of {len(pairs)}\n"21 f"kappa between the prompts, {pool} pool: {cohen_kappa_score(pairs['short'], pairs['guidelines']):.3f}")22 table = pd.crosstab(pairs["short"].map(LABELS).rename(f"short, {pool}"), pairs["guidelines"].map(LABELS).rename("guidelines"))23 table = table.reindex(index=order, columns=order, fill_value=0)24 cells = {f"{row} under short, {column} under guidelines": table.at[row, column] for row in order for column in order if row != column}25 print(f"{table}\nlargest disagreement, {pool} pool: {max(cells, key=cells.get)} ({max(cells.values())})")2627pairs, products, queries = graded_pairs("train"), load_products(), load_queries()28moved = pairs[(pairs["short"] == 1) & (pairs["guidelines"] == 0)] # partial under one prompt, irrelevant under the other29print("\n".join(f"moved train pairs judged {label} by people: {(moved['human'] == grade).sum()}" for grade, label in LABELS.items()))30# the first five queries in order, not a random sample31for query_id, product_id in moved[["query_id", "product_id"]].drop_duplicates("query_id").head(5).itertuples(index=False):32 print(f" {queries.at[query_id, 'query']:<28} {products.at[product_id, 'name'][:50]}")3334# a fixed seed, so the interval is the same on every run35rng, train_queries = np.random.default_rng(2026), pairs["query_id"].unique()36by_query = {query_id: group for query_id, group in pairs.groupby("query_id")}37# each resample draws train queries with replacement, so some repeat and others are left out38gaps = [kappa_gap(pd.concat([by_query[query_id] for query_id in rng.choice(train_queries, len(train_queries))]))39 for _ in range(2000)] # resample whole train queries40low, high = np.percentile(gaps, [2.5, 97.5])41print(f"kappa with people, guidelines minus short: {kappa_gap(pairs):.3f}\n95% interval over resampled train queries: {low:.3f} to {high:.3f}")
python prompt-disagreement.py
train pairs where the prompts disagree: 122 of 550
kappa between the prompts, train pool: 0.668
guidelines exact partial irrelevant
short, train
exact 122 4 1
partial 31 153 75
irrelevant 0 11 153
largest disagreement, train pool: partial under short, irrelevant under guidelines (75)
test pairs where the prompts disagree: 93 of 590
kappa between the prompts, test pool: 0.712
guidelines exact partial irrelevant
short, test
exact 303 11 1
partial 42 174 37
irrelevant 0 2 20
largest disagreement, test pool: partial under short, exact under guidelines (42)
moved train pairs judged exact by people: 18
moved train pairs judged partial by people: 31
moved train pairs judged irrelevant by people: 26
toddler couch fold out eulah 79 '' cotton rolled arm sofa bed
70s inspired furniture angels moroccan window curtains , oriental wavy cu
7qt slow cooker berghoff international eclipse 7 qt non-stick cove
gnome fairy garden fairy house tea pot fairy garden
wall sconce with usb port 2 piece wall sconce set
kappa with people, guidelines minus short: 0.022
95% interval over resampled train queries: -0.042 to 0.086
兩種提示詞在 122 of 550 組訓練配對與 93 of 590 組測試配對上意見不一致,彼此之間的 kappa 在訓練資料池為 0.668,在測試資料池為 0.712。
最大的不一致之處因資料池而異。在訓練資料池中是 partial under short, irrelevant under guidelines (75);在測試資料池(即各排序器的前 10 名)中則是 partial under short, exact under guidelines (42)。
在 short 判為部分相關、而 guidelines 判為不相關的訓練配對中,商店的評審者判為不相關的有 26 組,這與 guidelines 相符;判為部分相關的有 31 組,這與 short 相符;判為完全相符的有 18 組。此範例所列出的被改判配對中,包括對應 toddler couch fold out 的沙發床,以及對應 70s inspired furniture 的窗簾。錄製資料只保留標籤而不保留理由,因此我們從這類名稱推斷,這種改判來自 guidelines 中把錯誤產品類別判為不相關的那條規則。
就與商店評審者的 kappa 而言,guidelines 比 short 高出 0.022,而以重新抽樣的訓練查詢計算的 95% 區間為 -0.042 to 0.086。該區間包含 0,所以訓練資料池無法區分這兩種提示詞,商店因此兩者都保留。
實用原則: 把提示詞視為評審的一部分。逐字記錄每一個提示詞,並連同模型版本、設定與回答結構描述一併記錄,就像這份錄製資料在其資訊清單與腳本中所做的那樣。提示詞一旦更動,就視為不同的評審,並重新衡量其一致性。至少執行兩種提示詞,而且只有在兩者都得出相同結論時,才相信由 LLM 評分得出的結論。要在兩種提示詞之間擇一,須依據訓練配對,而且只有在兩者差異的區間不包含 0 時才可如此選擇。
此部分尚未翻譯,因此以英文呈現。
Fill unjudged pairs with LLM grades
We consider filling the holes in the human judgments the least risky use of an LLM judge. Keep every human grade, and use the LLM only for pairs nobody judged, which a metric would otherwise count as grade 0 (Count unjudged products). The LLM then decides only what was a guess before.
Of the 590 pairs in the top 10s of chapter 1's four rankers, 26 are unjudged, and BM25's top 10s hold 22 of them, against 3 for semantic retrieval and 6 for the hybrid.
Let each prompt fill them, rescore the four rankers on nDCG@10 (Score one ranking), and rerun chapter 3's planned comparisons on the filled grades:
1"""Fills the pairs nobody judged with each prompt's grades, then rescores the four rankers and the planned verdicts with the filled grades. The prose's point rests on the 'that change' lines, which list the rankers and verdicts that move when the holes are filled."""2from lib.agreement import LABELS, PLANNED, PROMPTS, chapter_one_runs, fill_unjudged, graded_pairs, llm_qrels, planned_verdicts3from lib.metrics import evaluate4from lib.store import load_products, load_queries, qrels567def order(scores):8 """Rankers by mean nDCG@10 over the test queries, best first."""9 return sorted(scores, key=lambda name: scores[name].mean(), reverse=True)101112products, queries, people = load_products(), load_queries("test"), qrels("test")13runs = chapter_one_runs(products, queries)14pairs = graded_pairs("test") # every product in the top 10 of the four runs, per test query15unjudged = pairs[pairs["human"].isna()]16print(f"pairs in the test pool: {len(pairs)}\nof them, unjudged: {len(unjudged)}")17for name, run in runs.items():18 # people[query_id] holds the human judgments for a query, so a product missing from it was never judged19 missing = sum(product_id not in people[query_id] for query_id, ranking in run.items() for product_id in ranking[:10])20 print(f"unjudged products in the top 10s, {name}: {missing}")21lights = unjudged[unjudged["query_id"] == 133] # outdoor light fixtures22print(f"unjudged products for outdoor light fixtures: {len(lights)}\n"23 f"of them, with bulb in the name: {lights['product_id'].map(products['name']).str.contains('bulb').sum()}")24threshold = 0.05 / len(PLANNED) # chapter 3's rule25human = {name: evaluate(run, people)["ndcg@10"] for name, run in runs.items()} # per-query scores26print(f"order, people's grades only: {' > '.join(order(human))}")2728for prompt in PROMPTS:29 counts = ", ".join(str((unjudged[prompt] == grade).sum()) for grade in LABELS) # grades 2, 1 and 030 print(f"\nunjudged pairs graded exact, partial, irrelevant, {prompt} prompt: {counts}")31 print(f"unjudged bulbs graded irrelevant, {prompt} prompt: {(lights[prompt] == 0).sum()}")32 filled = {name: evaluate(run, fill_unjudged(people, llm_qrels(prompt)))["ndcg@10"] for name, run in runs.items()}33 print("\n".join(f"{name} ndcg@10, filled by the {prompt} prompt: {filled[name].mean():.3f}" for name in runs))34 print(f"order, filled by the {prompt} prompt: {' > '.join(order(filled))}")35 # compares the two orders place by place36 moved = [name for name, place in zip(order(filled), order(human)) if name != place]37 print(f"rankers that change place when filled, {prompt} prompt: {', '.join(moved) or 'none'}")38 changed = [pair for pair, established in planned_verdicts(filled, threshold).items()39 if established != planned_verdicts(human, threshold)[pair]]40 print(f"verdicts that change when filled, {prompt} prompt: {'; '.join(' minus '.join(pair) for pair in changed) or 'none'}")
python fill-unjudged-pairs.pypairs in the test pool: 590
of them, unjudged: 26
unjudged products in the top 10s, bm25: 22
unjudged products in the top 10s, semantic: 3
unjudged products in the top 10s, hybrid: 6
unjudged products in the top 10s, reranked: 6
unjudged products for outdoor light fixtures: 8
of them, with bulb in the name: 8
order, people's grades only: semantic > hybrid > reranked > bm25
unjudged pairs graded exact, partial, irrelevant, short prompt: 4, 17, 5
unjudged bulbs graded irrelevant, short prompt: 0
bm25 ndcg@10, filled by the short prompt: 0.754
semantic ndcg@10, filled by the short prompt: 0.833
hybrid ndcg@10, filled by the short prompt: 0.821
reranked ndcg@10, filled by the short prompt: 0.813
order, filled by the short prompt: semantic > hybrid > reranked > bm25
rankers that change place when filled, short prompt: none
verdicts that change when filled, short prompt: none
unjudged pairs graded exact, partial, irrelevant, guidelines prompt: 5, 9, 12
unjudged bulbs graded irrelevant, guidelines prompt: 7
bm25 ndcg@10, filled by the guidelines prompt: 0.745
semantic ndcg@10, filled by the guidelines prompt: 0.834
hybrid ndcg@10, filled by the guidelines prompt: 0.821
reranked ndcg@10, filled by the guidelines prompt: 0.813
order, filled by the guidelines prompt: semantic > hybrid > reranked > bm25
rankers that change place when filled, guidelines prompt: none
verdicts that change when filled, guidelines prompt: none
The prompts disagree about the unjudged pairs. Counted as exact, partial and irrelevant, short grades them 4, 17, 5, and guidelines 5, 9, 12. They part on the unjudged products for outdoor light fixtures in BM25's top 10 (Score every test query): 8 of the 8 have bulb in their names.
guidelines calls 7 of the bulbs irrelevant, and short calls 0. Chapter 1 argued that a bulb isn't a light fixture. Which reading the store's judges would take, nobody knows until they judge the bulbs.
With the holes filled, BM25 scores 0.754 with short and 0.745 with guidelines, against 0.725 on human grades alone, and semantic retrieval scores 0.833 and 0.834, against 0.830.
The order on human grades alone is semantic > hybrid > reranked > bm25. The rankers that change place when short fills the holes are none, and when guidelines does, none. The planned comparisons' verdicts that change are none under short and none under guidelines.
Under human grades, the rankers' mean judged@10, the share of each top 10 that people judged, runs from 0.927 for BM25 to 0.990 for semantic retrieval (Compare runs in a harness).
How far can the filled grades be trusted? The closest evidence is the judged pairs of the same pool, where the LLM's kappa with the store's judges is 0.571 for short and 0.470 for guidelines. On those judged pairs, its grade is lower than the judges' on 61 pairs and higher on 65 with short, and lower on 77 and higher on 83 with guidelines (Read the direction of the LLM's errors).
Working rule: Fill the unjudged pairs in your rankers' top 10s with LLM grades as a diagnostic, under every prompt you run, and report the filled score beside the human-only one with the number of pairs filled. If filling changes the order of two rankers or a verdict, have people judge those pairs. Never let a filled grade replace a human one.
檢查 LLM 評分是否以相同方式排序各排序器
兩組判斷即使在許多標籤上意見不一,仍可能對系統做出相近的排名。Voorhees 以不同的相關性判斷集合為 TREC 的執行結果評分,發現「使用不同相關性判斷集合所產生的系統排名之間,相關性非常高」(Voorhees, 1998)。LLM 評審也呈現同樣的分歧。在 TREC 2023 Deep Learning 賽道的 42 組 LLM 標籤中,kappa 約從 0.06 到 0.29,而系統排名的相關性則維持在約 0.83 到 0.95 之間(Rahmani et al., 2025, Table 3)。
這種相關性就是 Kendall's tau。當兩種排序對一對系統的先後順序相同時,這一對稱為一致對(concordant);當兩種排序對這一對系統的先後順序相反時,則稱為不一致對(discordant)。tau 等於一致對數減去不一致對數,再除以所有配對數,範圍從 1(順序相同)到 −1(順序完全相反)。SciPy 的 kendalltau 計算的是 τ_b,這種形式也能處理同分的情況(SciPy documentation)。
Yilmaz et al. 指出,tau 達到 0.9 以上的排名,至少在經驗上常被視為實質等價(Yilmaz et al., 2008)。Sanderson 與 Soboroff 發現,以這類門檻做判斷,其可靠程度不如該領域原先的假設(Sanderson and Soboroff, 2007)。
以 nDCG@10 為第 1 章的四個排序器評分,分別採用人工評分與每個提示詞產生的 LLM 評分,就能看出 LLM 對它們的排序是否與人一致。LLM 只評判了測試池,也就是這些排序器前 10 名中的商品,因此另設一組對照,同樣以人工評分為它們評分,但只涵蓋測試池中的配對。四個排序器共有 6 組配對,所以交換一組配對會使 tau 變動 0.333,tau 也就只能取七個值。這樣相當粗略。一個在單一配對上與人差距這麼大的評審,仍有可能以和人相同的方式排序這些排序器。實際上會如此嗎?請先寫下你的答案:
1"""Scores the four rankers under human grades, under people's grades on the LLM's pairs as a control, and under each prompt, then counts the swapped pairs for each set. The prose's point rests on the 'swapped under' lines, read with the tau lines beside them."""2from itertools import combinations34from scipy.stats import kendalltau56from lib.agreement import chapter_one_runs, llm_qrels7from lib.metrics import evaluate8from lib.store import load_products, load_queries, qrels910products, queries, people = load_products(), load_queries("test"), qrels("test")11runs = chapter_one_runs(products, queries)12short, guidelines = llm_qrels("short"), llm_qrels("guidelines")13pooled = {query_id: {product_id: grade for product_id, grade in people[query_id].items() if product_id in short[query_id]}14 for query_id in people} # people's grades for the pairs the LLM judged, and no others15judgment_sets = {"people": people, "people on the LLM's pairs": pooled, "the short prompt": short,16 "the guidelines prompt": guidelines}17means = {name: {ranker: evaluate(run, judgments)["ndcg@10"].mean() for ranker, run in runs.items()}18 for name, judgments in judgment_sets.items()}19for name, scores in means.items():20 print(f"order under {name}: {' > '.join(sorted(scores, key=scores.get, reverse=True))}")21 for ranker, score in scores.items():22 print(f"{ranker} ndcg@10 under {name}: {score:.3f}")2324orders = [sorted(scores, key=scores.get, reverse=True) for scores in means.values()]25steady = [ranker for position, ranker in enumerate(orders[0]) if all(order[position] == ranker for order in orders)]26print(f"rankers in the same place under every set of grades: {', '.join(steady) or 'none'}")27control = means["people on the LLM's pairs"] # the same pool, human grades28for name in ["the short prompt", "the guidelines prompt"]:29 print("\n".join(f"{ranker}, {name} minus people on the LLM's pairs: {means[name][ranker] - control[ranker]:+.3f}" for ranker in runs))30# people's scores are the reference that every other set of grades is compared with31pairs, reference = list(combinations(runs, 2)), means["people"]32# one swap turns a concordant pair into a discordant one, which moves tau by 2 over the number of pairs33print(f"\npairs of rankers: {len(pairs)}\nchange in tau from one swapped pair: {2 / len(pairs):.3f}")34for name, scores in list(means.items())[1:]:35 swapped = [(first, second) for first, second in pairs # discordant: opposite signs36 if (reference[first] - reference[second]) * (scores[first] - scores[second]) < 0]37 print(f"swapped under {name}: {'; '.join(f'{first} and {second}' for first, second in swapped) or 'none'}")38 # concordant minus discordant, over all pairs; the discordant count is the number of swaps39 print(f"tau by hand, people against {name}: {(len(pairs) - 2 * len(swapped)) / len(pairs):.3f}")40 # SciPy's tau_b, which handles ties; .statistic is the tau value41 print(f"tau from scipy, people against {name}: {kendalltau(list(reference.values()), list(scores.values())).statistic:.3f}")
python kendall-tau-four-rankers.pyorder under people: semantic > hybrid > reranked > bm25
bm25 ndcg@10 under people: 0.725
semantic ndcg@10 under people: 0.830
hybrid ndcg@10 under people: 0.812
reranked ndcg@10 under people: 0.798
order under people on the LLM's pairs: semantic > hybrid > reranked > bm25
bm25 ndcg@10 under people on the LLM's pairs: 0.795
semantic ndcg@10 under people on the LLM's pairs: 0.904
hybrid ndcg@10 under people on the LLM's pairs: 0.885
reranked ndcg@10 under people on the LLM's pairs: 0.869
order under the short prompt: reranked > semantic > hybrid > bm25
bm25 ndcg@10 under the short prompt: 0.832
semantic ndcg@10 under the short prompt: 0.903
hybrid ndcg@10 under the short prompt: 0.900
reranked ndcg@10 under the short prompt: 0.906
order under the guidelines prompt: reranked > semantic > hybrid > bm25
bm25 ndcg@10 under the guidelines prompt: 0.805
semantic ndcg@10 under the guidelines prompt: 0.878
hybrid ndcg@10 under the guidelines prompt: 0.876
reranked ndcg@10 under the guidelines prompt: 0.886
rankers in the same place under every set of grades: bm25
bm25, the short prompt minus people on the LLM's pairs: +0.037
semantic, the short prompt minus people on the LLM's pairs: -0.001
hybrid, the short prompt minus people on the LLM's pairs: +0.016
reranked, the short prompt minus people on the LLM's pairs: +0.037
bm25, the guidelines prompt minus people on the LLM's pairs: +0.010
semantic, the guidelines prompt minus people on the LLM's pairs: -0.026
hybrid, the guidelines prompt minus people on the LLM's pairs: -0.008
reranked, the guidelines prompt minus people on the LLM's pairs: +0.017
pairs of rankers: 6
change in tau from one swapped pair: 0.333
swapped under people on the LLM's pairs: none
tau by hand, people against people on the LLM's pairs: 1.000
tau from scipy, people against people on the LLM's pairs: 1.000
swapped under the short prompt: semantic and reranked; hybrid and reranked
tau by hand, people against the short prompt: 0.333
tau from scipy, people against the short prompt: 0.333
swapped under the guidelines prompt: semantic and reranked; hybrid and reranked
tau by hand, people against the guidelines prompt: 0.333
tau from scipy, people against the guidelines prompt: 0.333
人工評分給出的順序是 semantic > hybrid > reranked > bm25。對照組,也就是以人工評分檢視 LLM 的配對,維持了這個順序,tau 為 1.000,因此把評分限定在測試池內並不會使任何排序器的位置移動。不過它確實提高了每個分數,例如 BM25 從 0.725 升到 0.795,原因在於理想的前 10 名只能包含進入測試池的商品。
short 之下的順序是 reranked > semantic > hybrid > bm25,guidelines 之下則是 reranked > semantic > hybrid > bm25。short 交換了 semantic and reranked; hybrid and reranked,guidelines 交換了 semantic and reranked; hybrid and reranked。
short 的 tau 以手算為 0.333,以 SciPy 計算為 0.333;guidelines 則為 0.333。
與對照組相比,LLM 評分使四個排序器的變動幅度與方向並不相同。在 short 之下,BM25 變動 +0.037,語意檢索變動 -0.001,混合式排序器變動 +0.016,經重排序的混合式排序器變動 +0.037。在 guidelines 之下,變動分別為 +0.010、-0.026、-0.008 與 +0.017。評審提高了某些排序器的分數,也降低了另一些的分數,使它們彼此超越。
在 short 之下,語意檢索、混合式排序器與經重排序的混合式排序器幾乎同分,分別為 0.903、0.900 與 0.906。tau 不考慮兩個排序器的分數相差多少,因此近乎同分者之間的一次交換,與最佳和最差排序器之間的一次交換,權重相同。
tau 也把頂端的交換與底端的交換視為同等重要,這是 Yilmaz et al. 指出的弱點(Yilmaz et al., 2008)。Yilmaz et al. 針對他們比較的 TREC 執行結果指出,這些系統「表現非常相近。因此,Kendall's τ 統計量可能會受這些系統影響」(Yilmaz et al., 2012)。
在每一組評分下都保持原位的排序器是 bm25。順序的差異只出現在人工評分給出 0.830、0.812 與 0.798 的三個排序器之間,而這些差距並不是任何預先規劃的比較所能確立的(重新執行預先規劃的比較)。
實用原則: 報告 tau 時,務必附上被交換的配對清單,不要單獨呈報。排序器只有少數幾個時,一次交換就使 tau 變動 0.333,因此不要拿它與 0.9 比較。接著再以各組評分下預先規劃的比較檢驗這些被交換的配對是否真有差異。
以 LLM 評分重跑計畫內比較
決策來自本書規則下的判定,而不是來自tau 所比較的排序。如果改用 LLM 評分,商店會得到相同的判定嗎?可預期問題會出現在排名前段:Otero 等人發現「以 LLM 為基礎的判斷在排序表現最佳的系統時並不公平」,且「在統計差異方面出現極高比率的偽陽性」(Otero et al., 2025)。
第 3 章的三項計畫內比較使用 0.0167 的門檻,以及 0.9833 水準的區間。我們在測試池上,分別以人工評分與各提示的評分重跑這些比較,並加入第 3 章中重排序混合檢索與語意檢索的探索性比較,以檢視商店是否會上線相同的排序器:
1"""Runs chapter 3's three planned comparisons, and the exploratory one, under human grades and under each prompt's grades on the test pool. The prose's point rests on the 'verdicts that differ' lines and the count of disagreeing checks."""2from lib.agreement import PLANNED, PROMPTS, chapter_one_runs, llm_qrels # PLANNED: chapter 3's, each stage against the last3from lib.metrics import evaluate4from lib.significance import compare, verdict5from lib.store import load_products, load_queries, qrels67EXPLORATORY = [("reranked", "semantic")] # chosen after seeing test means8threshold = 0.05 / len(PLANNED) # chapter 3's threshold, so that a verdict means the same either way9print(f"planned comparisons: {len(PLANNED)}\np-value threshold: {threshold:.4f}\ninterval level: {1 - threshold:.4f}")1011runs = chapter_one_runs(load_products(), load_queries("test"))12judgment_sets = {"people": qrels("test"), **{f"{prompt} prompt": llm_qrels(prompt) for prompt in PROMPTS}}13established, disagreements = {}, {judges: 0 for judges in judgment_sets}14for judges, judgments in judgment_sets.items():15 ndcg = {ranker: evaluate(run, judgments)["ndcg@10"] for ranker, run in runs.items()}16 for after, before in PLANNED + EXPLORATORY:17 # the interval's level matches the threshold, so each interval and p-value follow one rule18 result, name = compare(ndcg[after], ndcg[before], confidence=1 - threshold), f"{after} minus {before}, {judges}"19 print(f"\n{name}: {result['mean difference']:.3f}\n{name}, permutation p: {result['permutation p']:.3f}")20 # the exploratory comparison gets no t-test, interval or verdict21 if (after, before) in PLANNED:22 print(f"{name}, t-test p: {result['t-test p']:.3f}\n{name}, interval: {result['ci low']:.3f} to {result['ci high']:.3f}")23 checks = verdict(result, threshold)24 established[(judges, after, before)] = checks["established"]25 disagree = [check for check in ["t-test", "interval"] if not checks[f"{check} agrees"]]26 disagreements[judges] += bool(disagree)27 print(f"{name}, established: {'yes' if checks['established'] else 'no'}\n"28 f"{name}, disagrees with the permutation test: {' and '.join(disagree) or 'neither'}")2930print()31for judges, count in disagreements.items():32 print(f"planned comparisons where the t-test or interval disagrees, {judges}: {count}")33for prompt in PROMPTS:34 differ = [pair for pair in PLANNED if established[(f"{prompt} prompt", *pair)] != established[("people", *pair)]]35 print(f"verdicts that differ from people's, {prompt} prompt: {'; '.join(f'{after} minus {before}' for after, before in differ) or 'none'}")
python planned-comparisons-llm-grades.pyplanned comparisons: 3
p-value threshold: 0.0167
interval level: 0.9833
semantic minus bm25, people: 0.105
semantic minus bm25, people, permutation p: 0.001
semantic minus bm25, people, t-test p: 0.003
semantic minus bm25, people, interval: 0.047 to 0.205
semantic minus bm25, people, established: yes
semantic minus bm25, people, disagrees with the permutation test: neither
hybrid minus semantic, people: -0.018
hybrid minus semantic, people, permutation p: 0.125
hybrid minus semantic, people, t-test p: 0.123
hybrid minus semantic, people, interval: -0.048 to 0.006
hybrid minus semantic, people, established: no
hybrid minus semantic, people, disagrees with the permutation test: neither
reranked minus hybrid, people: -0.014
reranked minus hybrid, people, permutation p: 0.382
reranked minus hybrid, people, t-test p: 0.375
reranked minus hybrid, people, interval: -0.053 to 0.020
reranked minus hybrid, people, established: no
reranked minus hybrid, people, disagrees with the permutation test: neither
reranked minus semantic, people: -0.032
reranked minus semantic, people, permutation p: 0.048
semantic minus bm25, short prompt: 0.070
semantic minus bm25, short prompt, permutation p: 0.010
semantic minus bm25, short prompt, t-test p: 0.014
semantic minus bm25, short prompt, interval: 0.016 to 0.146
semantic minus bm25, short prompt, established: yes
semantic minus bm25, short prompt, disagrees with the permutation test: neither
hybrid minus semantic, short prompt: -0.002
hybrid minus semantic, short prompt, permutation p: 0.851
hybrid minus semantic, short prompt, t-test p: 0.861
hybrid minus semantic, short prompt, interval: -0.029 to 0.036
hybrid minus semantic, short prompt, established: no
hybrid minus semantic, short prompt, disagrees with the permutation test: neither
reranked minus hybrid, short prompt: 0.006
reranked minus hybrid, short prompt, permutation p: 0.727
reranked minus hybrid, short prompt, t-test p: 0.714
reranked minus hybrid, short prompt, interval: -0.029 to 0.042
reranked minus hybrid, short prompt, established: no
reranked minus hybrid, short prompt, disagrees with the permutation test: neither
reranked minus semantic, short prompt: 0.003
reranked minus semantic, short prompt, permutation p: 0.847
semantic minus bm25, guidelines prompt: 0.073
semantic minus bm25, guidelines prompt, permutation p: 0.045
semantic minus bm25, guidelines prompt, t-test p: 0.057
semantic minus bm25, guidelines prompt, interval: 0.006 to 0.183
semantic minus bm25, guidelines prompt, established: no
semantic minus bm25, guidelines prompt, disagrees with the permutation test: interval
hybrid minus semantic, guidelines prompt: -0.001
hybrid minus semantic, guidelines prompt, permutation p: 0.929
hybrid minus semantic, guidelines prompt, t-test p: 0.945
hybrid minus semantic, guidelines prompt, interval: -0.045 to 0.037
hybrid minus semantic, guidelines prompt, established: no
hybrid minus semantic, guidelines prompt, disagrees with the permutation test: neither
reranked minus hybrid, guidelines prompt: 0.010
reranked minus hybrid, guidelines prompt, permutation p: 0.657
reranked minus hybrid, guidelines prompt, t-test p: 0.609
reranked minus hybrid, guidelines prompt, interval: -0.024 to 0.076
reranked minus hybrid, guidelines prompt, established: no
reranked minus hybrid, guidelines prompt, disagrees with the permutation test: neither
reranked minus semantic, guidelines prompt: 0.009
reranked minus semantic, guidelines prompt, permutation p: 0.581
planned comparisons where the t-test or interval disagrees, people: 0
planned comparisons where the t-test or interval disagrees, short prompt: 0
planned comparisons where the t-test or interval disagrees, guidelines prompt: 1
verdicts that differ from people's, short prompt: none
verdicts that differ from people's, guidelines prompt: semantic minus bm25
| 量測 | 評分 | 語意檢索減 BM25 | 混合檢索減語意檢索 | 重排序減混合檢索 |
|---|---|---|---|---|
| 平均差異 | 人工 | 0.105 | -0.018 | -0.014 |
| 平均差異 | short |
0.070 | -0.002 | 0.006 |
| 平均差異 | guidelines |
0.073 | -0.001 | 0.010 |
| 區間 | 人工 | 0.047 to 0.205 | -0.048 to 0.006 | -0.053 to 0.020 |
| 區間 | short |
0.016 to 0.146 | -0.029 to 0.036 | -0.029 to 0.042 |
| 區間 | guidelines |
0.006 to 0.183 | -0.045 to 0.037 | -0.024 to 0.076 |
| 排列檢定 p 值 | 人工 | 0.001 | 0.125 | 0.382 |
| 排列檢定 p 值 | short |
0.010 | 0.851 | 0.727 |
| 排列檢定 p 值 | guidelines |
0.045 | 0.929 | 0.657 |
| t 檢定 p 值 | 人工 | 0.003 | 0.123 | 0.375 |
| t 檢定 p 值 | short |
0.014 | 0.861 | 0.714 |
| t 檢定 p 值 | guidelines |
0.057 | 0.945 | 0.609 |
| 是否成立 | 人工 | yes | no | no |
| 是否成立 | short |
yes | no | no |
| 是否成立 | guidelines |
no | no | no |
商店唯一成立的結果能否在 LLM 評分下維持,取決於提示。在 guidelines 下,判定不同的是 semantic minus bm25,也就是第 3 章所確立的那項結果。在 short 下,與人工判定不同的是 none。
在人工評分下,t 檢定或區間與排列檢定不一致的計畫內比較有 0 項;在 short 下有 0 項,在 guidelines 下有 1 項。就 guidelines 下的語意檢索減 BM25 而言,不一致的檢查是interval。
比較排序器的順序時發現兩組互換的配對。其一是重排序混合檢索對混合檢索,屬於計畫內比較,而表中顯示它在任何一組評分下都未成立。另一組是重排序混合檢索對語意檢索,屬於探索性比較。在人工評分下,重排序混合檢索減語意檢索為 -0.032(排列檢定 p 值 0.048)。在 LLM 評分下,使用 short 時為 0.003,使用 guidelines 時為 0.009。第 3 章是在看過測試平均值之後才選定此比較,因此無論結果如何,它都不能確立任何結論。
LLM 評審也可能偏好使用 LLM 的排序器:Balog 等人報告了「LLM 評審對以 LLM 為基礎的排序器表現出顯著偏見的首項實證證據」(Balog et al., 2025)。第 1 章的排序器不使用 LLM,因此本章無法量測這種偏見。本書以人工評分來為第 8 章與第 9 章中以 LLM 為基礎的排序器評分。
實用原則: 切勿僅憑 LLM 評分得出的判定就上線變更。用 LLM 評分來決定哪些候選值得評判,並以人工評分做決策。當兩種提示給出不同判定時,LLM 無法裁定該項比較,必須由人來裁定。
以人工樣本校正 LLM 分數
折衷的做法是兩種評分並用:以 LLM 為每個查詢評分,再由人員判定隨機抽出的一部分查詢,並以這份樣本校正 LLM 的誤差。預測驅動推論(prediction-powered inference)能以人工樣本嚴謹地校正模型的預測,其信賴區間「對提供預測的機器學習演算法不做任何假設」即可成立(Angelopoulos et al., 2023)。以平均數為例,這個概念的簡單形式是:取 LLM 在所有查詢上的平均值,再加上人工樣本中「人工分數減 LLM 分數」的平均:
corrected = mean over all queries of the LLM score + mean over the human sample of (human score − LLM score)
在搜尋評估方面,Oosterhuis 等人以這種方式建立指標的信賴區間,並提出警告:「生成的相關性標註並非不會出現(系統性)錯誤」(Oosterhuis et al., 2024)。Takehi 等人則在預算限制下,讓 LLM 挑選要由人員判定的配對,再以這些判定來校準 LLM(Takehi et al., 2025)。
假設商店負擔得起 10 of 30 筆測試查詢的人工判定。本範例隨機抽取 2,000 組這種大小的集合,並在每一組中,針對每種提示詞,校正 LLM 對第 1 章各排序器之間兩項差距的估計:語意檢索減 BM25,以及混合式檢索減語意檢索;接著將每個校正後的估計值與單獨使用人工樣本所得的結果比較。離散程度(spread)指某個估計值在各次抽樣之間的標準差。模擬過程不做任何人為選擇:
1"""Draws random sets of human-judged test queries and corrects the LLM's estimate of two ranker gaps with each set. The prose's point rests on the 'smaller than the human sample's' lines, read with the correlation lines beside them."""2import numpy as np3import pandas as pd45from lib.agreement import PROMPTS, chapter_one_runs, llm_qrels6from lib.metrics import evaluate7from lib.store import load_products, load_queries, qrels89JUDGED, DRAWS = 10, 2000 # test queries people judge in each draw, and the number of draws10runs = chapter_one_runs(load_products(), load_queries("test"))11people = pd.DataFrame({ranker: evaluate(run, qrels("test"))["ndcg@10"] for ranker, run in runs.items()})12rng = np.random.default_rng(2026)13draws = [rng.choice(people.index, JUDGED, replace=False) for _ in range(DRAWS)] # which queries people judge14print(f"queries judged by people in each draw: {JUDGED} of {len(people)}\ndraws: {DRAWS:,}")1516for after, before in [("semantic", "bm25"), ("hybrid", "semantic")]:17 gap_people = people[after] - people[before] # per-query gaps under people's grades18 print(f"\n{after} minus {before}, people, all queries: {gap_people.mean():.3f}")19 # the spread: the standard deviation over the draws, with numpy's default divisor20 alone = np.std([gap_people[judged].mean() for judged in draws]) # the human sample used alone21 print(f"{after} minus {before}, human sample alone, spread: {alone:.3f}")22 for prompt in PROMPTS:23 llm = {ranker: evaluate(run, llm_qrels(prompt))["ndcg@10"] for ranker, run in runs.items()}24 gap_llm, name = llm[after] - llm[before], f"{after} minus {before}, {prompt} prompt"25 # the LLM's mean over all queries, plus the human sample's average of human minus LLM26 corrected = [gap_llm.mean() + (gap_people[judged] - gap_llm[judged]).mean() for judged in draws]27 print(f"{name}, LLM only: {gap_llm.mean():.3f}")28 print(f"{name}, corrected, mean over draws: {np.mean(corrected):.3f}")29 print(f"{name}, corrected, spread: {np.std(corrected):.3f}")30 print(f"{name}, corrected spread smaller than the human sample's: {'yes' if np.std(corrected) < alone else 'no'}")31 # corrcoef returns a 2×2 matrix; [0, 1] is the correlation between the two lists32 print(f"{name}, correlation of per-query gaps with people's: {np.corrcoef(gap_llm, gap_people)[0, 1]:.3f}")
python correct-llm-scores-with-humans.pyqueries judged by people in each draw: 10 of 30
draws: 2,000
semantic minus bm25, people, all queries: 0.105
semantic minus bm25, human sample alone, spread: 0.045
semantic minus bm25, short prompt, LLM only: 0.070
semantic minus bm25, short prompt, corrected, mean over draws: 0.104
semantic minus bm25, short prompt, corrected, spread: 0.036
semantic minus bm25, short prompt, corrected spread smaller than the human sample's: yes
semantic minus bm25, short prompt, correlation of per-query gaps with people's: 0.641
semantic minus bm25, guidelines prompt, LLM only: 0.073
semantic minus bm25, guidelines prompt, corrected, mean over draws: 0.104
semantic minus bm25, guidelines prompt, corrected, spread: 0.039
semantic minus bm25, guidelines prompt, corrected spread smaller than the human sample's: yes
semantic minus bm25, guidelines prompt, correlation of per-query gaps with people's: 0.692
hybrid minus semantic, people, all queries: -0.018
hybrid minus semantic, human sample alone, spread: 0.016
hybrid minus semantic, short prompt, LLM only: -0.002
hybrid minus semantic, short prompt, corrected, mean over draws: -0.017
hybrid minus semantic, short prompt, corrected, spread: 0.023
hybrid minus semantic, short prompt, corrected spread smaller than the human sample's: no
hybrid minus semantic, short prompt, correlation of per-query gaps with people's: 0.168
hybrid minus semantic, guidelines prompt, LLM only: -0.001
hybrid minus semantic, guidelines prompt, corrected, mean over draws: -0.017
hybrid minus semantic, guidelines prompt, corrected, spread: 0.024
hybrid minus semantic, guidelines prompt, corrected spread smaller than the human sample's: no
hybrid minus semantic, guidelines prompt, correlation of per-query gaps with people's: 0.342
就語意檢索減 BM25 而言,單靠 LLM 在 short 下估計為 0.070,而以人工評分計算則為 0.105。校正後的估計值在各次抽樣中平均為 0.104,因此校正消除了 LLM 的偏差。
其離散程度為 0.036,單獨使用人工樣本則為 0.045。校正後的離散程度是否小於人工樣本的?在 short 下為 yes,在 guidelines 下為 yes。LLM 對其他查詢的評分提供了額外資訊,因為它逐查詢的差距與人工的差距相關係數為 0.641。
就混合式檢索減語意檢索而言,校正後的離散程度在 short 下為 0.023,在 guidelines 下為 0.024,單獨使用人工樣本則為 0.016。是否小於人工樣本的?在 short 下為 no,在 guidelines 下為 no。在 short 下,逐查詢的差距相關係數為 0.168。
校正能修正分數的水準,但只有在 LLM 的逐查詢分數與人工分數相符時,才能縮小不確定性;而在這兩項差距中,較為接近的那一項正是兩者最不相符之處。
實用原則: 每當你根據 LLM 的評分回報分數時,應讓人員判定隨機抽出的一部分查詢,並回報校正後的估計值。在該樣本上,檢查 LLM 逐查詢的差距與人工的差距相關程度如何。若相關性偏弱,就把預算用在判定更多查詢上。
正式環境中會出什麼問題
**評判者會在沒有通知的情況下改變。**以 LLM 做評判的商店會一再執行它,因此不同的模型版本、量化(模型權重的儲存精度)或改寫過的提示詞,都等於另一個評判者。錄製資料的 manifest.json 會固定模型的摘要值(digest)以及兩份提示詞的逐字內容。回答結構(answer schema)與上下文長度則位於 recordings/llm.py 和 recordings/llm_judgments.py,因此這些也要一併記錄。只要其中任何一項改變,就要在同一份人工樣本上重新執行一致性檢查。
**針對評判者調校的系統,在該評判者下得分會偏高。**Clarke 與 Dietz 建立了一個利用 LLM 評判者的執行結果,它在自動評估下的排名(第 5 名)「明顯高於」人工評估下的排名(第 28 名)(Clarke and Dietz, 2024)。依據 LLM 評分調校的排序器,或以同一個 LLM 建構的排序器,其主要分數必須以人工評分為準。
**商品文字可能左右評判者。**Alaofi 等人發現,LLM 評判者可能被植入不相關文字中的查詢詞所誤導,也可能被寫入其中的指示所誤導(Alaofi et al., 2024)。賣家可以寫出一段對評判者說話的商品描述。每次評估都要保留一份人工樣本,如此一來,評判者被誤導時,就會表現為一致性下降。
**在測試集上校準,就等於在測試集上調校。**反覆改寫提示詞,直到它與測試查詢上的人工評分相符,會使該處的一致性失去意義。請在訓練查詢上選擇提示詞,並在測試查詢上報告一致性。
**人工評分同樣不是真相。**在 TREC Podcast 賽道中,針對 LLM 與 TREC 評估者分歧最大的項目,Mansour 等人發現,專家重新評判這些配對時,較常站在 LLM 這一邊(Mansour et al., 2026)。請把人工評分視為你所選定的參照標準,並在責怪任何一方之前,先閱讀那些分歧。
**實用原則:**固定評判者,每次評估都保留一份人工樣本,任何項目改變時都重新執行一致性檢查,並且絕不使用調校排序器時所用的評判者來為該排序器評分。
本章做出的決定
除非後續章節提出理由,否則資料庫沿用這些決定。
- 所有主要分數都採用人工評分。 LLM 評分只作為診斷工具:補足缺漏、篩選候選項目,以及檢查判定結果。第 8 章與第 9 章中以 LLM 為基礎的排序器,是以人工評分來計分。
- 關於評審的每項發現,都要註明所屬的資料池。 Kappa 與線性加權 Kappa,並一併列出觀察到的一致率、嚴重誤判,以及標籤組成,都是在人工評判過的訓練配對上測得,並同時回報測試資料池中已評判的配對。方向與最大誤差在各個資料池上分別判讀,而只有排序器的前 10 名才能說明排序器的分數。
- 兩個提示詞,逐字記錄。 訓練資料池無法區分兩者,因此資料庫兩者都保留,而且只有在兩個提示詞都得出同一結論時,才採信由 LLM 評分得出的結論。
- 以 LLM 評分填補未評判的配對,僅作為診斷, 並與僅含人工評分的分數並列回報。在測試查詢上,填補不會改變任何排序器的名次,也不會改變任何判定結果。
- 只在相同評分下比較分數。 在同一個資料池上,LLM 評分對各排序器 nDCG@10 的影響,幅度不一,方向也不同,因此以 LLM 評分得到的分數,不會與以人工評分得到的分數相互比較。
- 先看 Tau 及其交換配對,再依第 3 章的門檻進行規劃好的比較。 只有四個排序器時,Tau 相當粗略,最後由判定結果決定。
- 由 LLM 評分得出的分數,須附上人工抽樣的查詢 來修正,並附上相關係數,以說明 LLM 的評分是否縮小了不確定性。
- 本章所顯示的: LLM 評審可以補足缺漏並篩選候選項目,但它的評分對各排序器的影響並不均勻,因此重要的判定仍然需要人工評分。
本章,於本版次中
版次 2026.11.10範例於 2026年11 月 19 通過測試本章 11 個範例 個範例中有 11 個通過
譯自英文書,版次 2026.11.10。