把評估放進 CI,並在流量上比較各種變體

閱讀約需 47 分鐘 分鐘,5 節 中共有 4 個範例。

版次 2026.11.3 範例於 2026年11 月 21 通過測試 54 個範例中有 54 個通過 變更內容

由 AI 翻譯。經人工以英文驗證。 我們如何製作書籍

譯自英文書,版次 2026.11.3。 閱讀原文

讀完本章,你能把評估做成一道閘門,只在變更確實變差時才擋下該變更。你也能說明,對兩個變體做線上比較能顯示什麼,又不能顯示什麼。

sablefile 助理依據該工具的文件回答問題,而團隊每週都會調整它的提示詞、檢索方式或模型。團隊希望在變更合併之前就知道助理是否變差了。一道只要通過率下降就判定失敗的閘門,看起來是穩當的選擇,其實不然。在有 80 個開發項目的情況下,一個項目約使通過率變動 1.3 個百分點,因此以原始通過率為準的閘門,會因為一兩個項目的差異而擋下變更。團隊若常因雜訊而被擋下,往往會學會繞過這道閘門,到那時閘門就什麼也保護不了。

第 1 章首次完整執行了整個評估:測試集、檢查、評審,以及重播的回答。本章保留那次執行的設定不動,依序討論三個部分:尊重信賴區間的閘門、判定結果會在不同次取樣之間改變的項目,以及針對評估程式碼本身的測試。接著,本章把這三個部分對應到評估框架所用的詞彙,最後說明在線上流量上比較各種變體,能顯示什麼,又不能顯示什麼。

尊重區間的閘門

閘門是一條阻止變更合併的規則。在本書中,這條規則只依據一個數字:候選版本與基準版本在相同開發項目上的通過率配對差。這項比較是配對的,如第 8 章所定義:對每個項目,取候選版本的判定減去基準版本的判定,再對所有項目取平均,區間則以項目為單位做自助法(bootstrap)取得。由於兩次執行回答的是相同的問題,兩次執行結果一致的項目會在差值中相互抵消。

閘門只有一條規則:僅當區間上界低於負的門檻值時才失敗。上界是資料仍允許的最不負的讀數,因此閘門失敗,代表即使以最樂觀的方式解讀資料,降幅仍然大於門檻值。天真的閘門只要差值的點估計低於負的門檻值就會失敗,因此會被雜訊誤導而失敗。本書這條規則的代價,是它會放行某些資料尚無法證實的真實降幅,所以讀取通過結果時必須連同區間寬度一起看,後面幾段會說明。

門檻值是五個百分點。範例中的註解指出,這是本書的選擇,並非標準。以此開發集的規模,一個項目約為 1.3 個百分點,所以五個百分點相當於淨差四個項目。

第一次執行是一次大幅下降,以這個評審的衡量來看是如此。第二個模型 Qwen3-0.6B 比 4B 的基準版本小,並且回答了相同的開發項目。評審是 Qwen3-4B-Instruct-2507,也就是基準版本自身的模型,因此這個差距無法區分較弱的助理與自我偏好。自我偏好說明了本書為何無法檢驗這一點。最後再讀決定,因為它是由區間推得的。

gate-real-drop.py Python 3.12.15 · numpy 2.5.5 · pydantic 2.14.3 · typing-extensions 4.16.0 通過
"""Runs the gate on second-model against the baseline over the eighty dev items: both pass rates with their counts, the paireddifference, its interval, the threshold and the decision. The gate fails only when the upper bound of the interval is belowminus the threshold, so a drop that the interval cannot rule out below the threshold does not fail it."""import lib  # first: fixes the thread counts and seeds before anything else loadsfrom lib.calibration import readable_verdictsfrom lib.gate import FAIL, gatefrom lib.judging import passedfrom lib.prompts import REF_SHA256from lib.stats import interval_points, pctfrom lib.testset import load_itemsBASELINE = "judge-answers-baseline"CANDIDATE = "judge-answers-second-model"# Five points is this gate's threshold, chosen for the book, not a standard. With eighty dev items one item moves a rate# by 1.25 points, so five points is four items net: a smaller threshold would ask the gate to see drops the interval# cannot resolve at this size.THRESHOLD_POINTS = 5.0items = [item.id for item in load_items(split="dev")]  # eighty dev items, sorted by idbaseline_found, baseline_unreadable = readable_verdicts(BASELINE, items, REF_SHA256)candidate_found, candidate_unreadable = readable_verdicts(CANDIDATE, items, REF_SHA256)for label, unreadable in (("baseline", baseline_unreadable), ("second-model", candidate_unreadable)):    details = "".join(f' ({item_id}, "{first}")' for item_id, first in unreadable)    print(f"unreadable verdicts, {label}: {len(unreadable)}{details}")# An item the judge did not give a readable verdict on, under either run, is left out of both sides: a paired# difference needs the same items under both runs.excluded = {item_id for item_id, _ in baseline_unreadable + candidate_unreadable}kept = [item_id for item_id in items if item_id not in excluded]first = passed({item_id: baseline_found[item_id] for item_id in kept})  # 1.0 or 0.0 per item, in id ordersecond = passed({item_id: candidate_found[item_id] for item_id in kept})result = gate(first, second, threshold_points=THRESHOLD_POINTS)print(f"dev items compared (n): {result.n}")print(f"baseline pass rate: {pct(int(first.sum()), result.n)}")print(f"second-model pass rate: {pct(int(second.sum()), result.n)}")print(f"difference, second-model minus baseline: {100 * result.difference:+.1f} points")print(f"{100 * result.level:.0f}% interval of the difference: {interval_points(result.lo, result.hi)}")print(f"upper bound of the interval: {100 * result.hi:+.1f} points")print(f"threshold: {THRESHOLD_POINTS:.1f} points")if result.decision == FAIL:    print(f"decision: fail, since the upper bound {100 * result.hi:+.1f} is below {-THRESHOLD_POINTS:+.1f}")else:    print(f"decision: pass, since the upper bound {100 * result.hi:+.1f} is not below {-THRESHOLD_POINTS:+.1f}")
輸出python gate-real-drop.py
unreadable verdicts, baseline: 1 (q-0043, "VERDICT: PARTIAL")
unreadable verdicts, second-model: 0
dev items compared (n): 79
baseline pass rate: 78.5% (62 of 79)
second-model pass rate: 40.5% (32 of 79)
difference, second-model minus baseline: -38.0 points
95% interval of the difference: [-48.1, -26.6] points
upper bound of the interval: -26.6 points
threshold: 5.0 points
decision: fail, since the upper bound -26.6 is below -5.0

閘門失敗,原因在於區間,而不在於點估計。基準版本的通過率為 78.5% (62 of 79)。第二個模型的通過率為 40.5% (32 of 79)。

這項比較涵蓋 79 個項目,而非全部的開發項目,因為基準版本判定無法讀取的項目,會從兩次執行中一併剔除。配對差需要兩邊都有相同的項目。差值(第二個模型減去基準版本)為 -38.0 points,其 95% 區間為 [-48.1, -26.6] points。

上界為 -26.6 points,門檻值為 5.0 points,所以即使是上界,也遠低於門檻值。區間無法將這個降幅解釋為雜訊,因此閘門的設計正是要對這類降幅判定失敗,以這個評審的衡量來看是如此。決定為:fail, since the upper bound -26.6 is below -5.0。

較小的變更會得到不同的結果。假設變更使通過率下降較少,或資料集較小而區間變寬,以致上界高於負五個百分點,閘門便會通過。這表示它沒有發現大於門檻值的降幅,並不表示已經證明候選版本與基準版本一樣好。區間很寬的通過是微弱的通過,因此要讀區間寬度,而不只是讀決定。在給定規模下區間會有多寬,以及微小的配對差需要多少項目才能分辨,是第 8 章探討的內容。

閘門的限制在於規模。在 79 個項目上,上界必須遠低於負的門檻值,閘門才會失敗,因此小幅降幅會通過。第 8 章在其假設的不一致程度下估計,79 個項目足以在 80% 檢定力下顯示十個百分點的差異,而五個百分點的差異需要 316 個項目。閘門是比該檢定力計算更嚴格的檢驗,所以真實的十個百分點降幅仍可能通過。門檻值與資料集規模必須一併選定。

有些閘門會檢查多個指標。每多一個指標,就多一次讓雜訊看起來像降幅的機會,因此閘門要計入比較的次數。有 m 個指標時,每個區間都取在 1 減去 0.05 除以 m 的水準:兩個指標為 97.5%,三個指標約為 98.3%。這些檢查合起來,使誤判失敗的機率維持在百分之五以下。調整後的水準即第 8 章給出的水準,閘門使用同一個。重抽樣次數維持 2,000 次,種子維持固定,因此重新執行會得到相同的區間。

閘門在每次變更時都以開發項目執行,因為閘門每次都必須執行,而且團隊可能已在這些項目上調校過。保留集(held-out)每項發布決策只執行一次,並依據執行前就已固定的規則,如保留集,只用一次所說明。在那之後需要一份新的保留集。

要把閘門放進 CI,由一個工作對候選版本的錄製結果計算閘門,並在決定為失敗時讓建置失敗。閘門只讀取逐項目的值,因此該工作不需要其他狀態。無法執行的工作視為閘門失敗,因為被略過的閘門看起來會像通過。閘門自身的測試會檢查上述規則,它們與評估的其他測試一起在評估自身的測試中執行。

實用原則: 每一個配對差都要連同其區間與 n 一併報告,並讓決定由上界推得。不要僅憑點估計就說某項變更較好或較差,也不要把區間很寬的通過解讀為候選版本等同基準版本的證明。

此部分尚未翻譯,因此以英文呈現。

Flag flaky items, never edit them

最近一次變動於第 2026.11.3 版

An item flips when its sampled replies get different verdicts. Each dev item has five verdicts from the variance set, and an item is flaky when they are not all the same. The variance set and the flip rate, the share of items that flip, are introduced in Variance within one item. A team watches the flip rate to see how far the verdicts move when the assistant's replies do.

Each flagged item comes with its PASS count out of five, in id order, and the rule it is held to is printed beside the list.

flaky-items.py Python 3.12.15 · numpy 2.5.5 · pydantic 2.14.3 · typing-extensions 4.16.0 通過
"""The dev items flagged as flaky: the judge's verdict differs across their five variance samples. Each flagged item prints itsPASS count out of five, sorted by id, and the count matches the flip rate in flip-rate.py, since both read the same per-itemcounts. A flagged item is kept as it is, so the rule printed last is the whole policy: flagged, never edited or deleted."""import lib  # first: fixes the thread counts and seeds before anything else loadsfrom lib.testset import load_itemsfrom lib.variance import VARIANCE_SAMPLES, item_pass_countsVARIANCE = "judge-answers-variance"SAMPLES = len(VARIANCE_SAMPLES)items = [item.id for item in load_items(split="dev")]  # eighty dev items, sorted by idcounts = item_pass_counts(VARIANCE, items)for row in counts:    if not row.complete:  # an unreadable sample is not a FAIL: the item is named here and not flagged        samples = ", ".join(str(sample) for sample in row.unreadable)        print(f"left out, unreadable verdict, {row.item_id}: samples {samples}")flagged = [row for row in counts if row.flaky]for row in flagged:    print(f"{row.item_id}: {row.passes} of {SAMPLES} PASS")print(f"flagged items: {len(flagged)}")print(f"dev items compared (n): {sum(1 for row in counts if row.complete)}")print("rule: flagged, never edited or deleted")
輸出python flaky-items.py
left out, unreadable verdict, q-0043: samples 1, 4
q-0047: 4 of 5 PASS
q-0058: 1 of 5 PASS
flagged items: 2
dev items compared (n): 79
rule: flagged, never edited or deleted

Few items flip on this set. The flagged items come to 2 of the 79 dev items compared. Both flagged items are worth reading.

On q-0047, the verdicts across its five replies are 4 of 5 PASS, so most of its replies passed. On q-0058, the verdicts are 1 of 5 PASS, so most of its replies failed.

One item is set aside rather than used. The item q-0043 is left out, since its verdict was unreadable in samples 1, 4. An unreadable sample is not a fail. It is a sample the example cannot read, so the item is neither flagged nor counted on that basis.

The rule is that a flagged item stays flagged, never edited or deleted. It stays in the set with the verdicts the judge gave. Editing it until the judge agrees more often would hide the very signal the flag exists to show. Deleting it would shrink the set and make the pass rate look steadier than it is. The flag does its work when someone reads it. A flagged item with one PASS in five is a question about the item, the rubric, the judge or the assistant's replies, and it should be looked at before the next change is judged on it.

The count is small, and it describes this assistant, this judge and these dev items. It shows that few items flip in this set. It does not show that other assistants, sampling settings or judges flip as rarely. A team that changes any of them should run the variance set again before it trusts a low flip rate.

A flagged item can also move a paired difference from one run to the next. When a gate fails on a difference that depends on a flagged item, read the flag before the change.

Working rule: Flag an item whose verdicts differ across its samples, print its count beside the flag, and leave the item as it is. Report the flip rate with its denominator, and say that a low rate holds for this assistant, this judge and this set only.

評估本身的測試

評估是程式碼,而程式碼可能出錯,且錯誤看起來像是一個結果。bootstrap、判定結果讀取器或配對差異中的一個錯誤,使數字產生變動,就如同助理的改變所造成的一樣。因此評估有自己的測試,放在本書函式庫內的 tests 資料夾中,書中新增的每個模組都附帶各自的測試。這些測試檢查的是計分程式碼,而不是助理。測試通過,表示程式碼所計算的確實是它所宣稱的內容;至於助理回答得好不好,它並未說明。

這些測試在同一個行程中執行,結果按模組分別計數。範例接著印出 pytest 的結束碼,所用版本為 9.2.0。結束碼參考文件將 0 列為所有測試皆已收集且通過,將 5 列為未收集到任何測試。因此,沒有測試任何東西的執行會顯示為 5,而不是悄悄地通過。

own-tests-run.py Python 3.12.15 · bm25s 0.3.16 · numpy 2.5.5 · pydantic 2.14.3 · pytest 9.2.0 · typing-extensions 4.16.0 通過
"""Runs the book's own tests, lib/tests, in this process and prints how many passed and failed in each module, then pytest'sexit code. Exit code 0 means every test was collected and passed. Exit code 5 means nothing was collected, so the exampleprints the code it got rather than a silent pass. No paths and no timings are printed: they vary from run to run."""import lib  # first: fixes the thread counts and seeds before anything else loadsfrom collections import Counterfrom pathlib import Pathimport pytestTESTS = Path(lib.__file__).resolve().parent / "tests"  # read here, never printedclass Outcomes:    """Counts each test's outcome by module. A failed setup or teardown counts as a failure of its test."""    def __init__(self) -> None:        self.passed = Counter()        self.failed = Counter()    def pytest_runtest_logreport(self, report) -> None:        # The node id starts with the file's path; keep only the module's name, so no path is printed.        module = Path(report.nodeid.split("::")[0]).stem        if report.when == "call" and report.passed:            self.passed[module] += 1        elif report.failed:            self.failed[module] += 1collector = Outcomes()# -p no:terminal turns off pytest's own printing, which carries timings. That also removes the -q option, which belongs to# the terminal reporter, so it is not passed. -p no:cacheprovider keeps .pytest_cache out of the working copy.exit_code = pytest.main(    [str(TESTS), "-p", "no:terminal", "-p", "no:cacheprovider"],    plugins=[collector],)modules = sorted(path.stem for path in TESTS.glob("test_*.py"))  # every module, even one with no failuresprint("tests by module (passed, failed):")for module in modules:    print(f"  {module}: {collector.passed[module]} passed, {collector.failed[module]} failed")print(f"total: {sum(collector.passed.values())} passed, {sum(collector.failed.values())} failed")print(f"pytest exit code: {int(exit_code)}")
輸出python own-tests-run.py
tests by module (passed, failed):
  test_bench: 11 passed, 0 failed
  test_bias: 14 passed, 0 failed
  test_both_orders: 14 passed, 0 failed
  test_calibration: 9 passed, 0 failed
  test_calibration_bootstrap: 8 passed, 0 failed
  test_contract: 5 passed, 0 failed
  test_gate: 11 passed, 0 failed
  test_guard: 4 passed, 0 failed
  test_judging: 3 passed, 0 failed
  test_stats: 10 passed, 0 failed
  test_support_verdict: 5 passed, 0 failed
  test_testset: 5 passed, 0 failed
  test_variance: 15 passed, 0 failed
total: 114 passed, 0 failed
pytest exit code: 0

下表列出各模組的結果,並在旁邊註明該模組所測試的內容。

模組 測試內容 結果
test_bench 基準測試的載入器與計分器 11 passed, 0 failed
test_bias 長度區間、無法讀取的判定結果、翻轉方向與區間差距 14 passed, 0 failed
test_both_orders 成對比較的讀取器與兩種順序:對應回各個回覆的映射,以及四種類型 14 passed, 0 failed
test_calibration 評審與人工判定對照的計數、一致率、kappa、TPR、TNR 與校正後的通過率 9 passed, 0 failed
test_calibration_bootstrap 交叉點與校正後通過率的 bootstrap 區間 8 passed, 0 failed
test_contract 評審與助理必須遵守的回覆與判定格式 5 passed, 0 failed
test_gate 閘門的規則:僅當上界低於負的門檻值時才判定為未通過 11 passed, 0 failed
test_guard 拒答防護的規則及其門檻值,以手工製作的資料列測試 4 passed, 0 failed
test_judging 讀取評審的判定結果,以及配對差異的型別 3 passed, 0 failed
test_stats 輸出輔助函式與設定種子的 bootstrap 10 passed, 0 failed
test_support_verdict 忠實度判定結果的讀取器:第一行為 SUPPORTED 或 UNSUPPORTED 5 passed, 0 failed
test_testset 載入測試集,以及跨項目檢視的規則 5 passed, 0 failed
test_variance 變異集上每個項目的判定結果計數 15 passed, 0 failed

總計列在最後:114 passed, 0 failed,結束碼為 0,即所有收集到的測試皆通過的執行所對應的代碼。

請將此表視為對測量工具的檢查,而不是對助理的檢查。在閘門作業中應先執行這些測試,使損壞的計分器在讀取通過率之前就讓作業停止。輸出中不含任何在不同次執行之間會變動的內容:沒有耗時,也沒有路徑。讀者可以重新執行程式碼,並逐行比對輸出。

實用原則: 在任何要讀取數字的執行之前,先執行評估本身的測試,並將結束碼 5 視為失敗,而不是通過。計分程式碼中若有測試失敗,無論數字顯示什麼,都表示這些數字尚不可信。

此部分尚未翻譯,因此以英文呈現。

Running it as a framework task

最近一次變動於第 2026.11.3 版

Teams that already use an evaluation framework can map the parts of this book onto its words, and it helps to see the mapping once. In Inspect AI 0.3.291, dated 17 November 2026 in its changelog, a task is the recipe for an evaluation, made minimally of a dataset, a solver and a scorer, as the framework's documentation at that version describes. On that reading, the test set is the dataset, with one sample per item. The assistant's path of retrieving and then answering is the solver, and the checks and the judge make up the scorer. The book does not run that mapping, and it does not claim to. Instead, the book keeps one example environment: one set of files that every chapter's examples import, pinned to the same versions, so the replay model, the checks and the gate are the same code in every chapter. This chapter describes where each part would go, so that a reader on a framework can move the parts over while the book's examples stay as they are.

Working rule: When you map an evaluation onto a framework, name the framework's version and the date of its documentation. Keep the test set, the checks and the replay model as the files the book's examples run, and treat the map as a reading of the documentation until someone runs it.

此部分尚未翻譯,因此以英文呈現。

Variants on traffic

Offline, the test set and the replayed answers can say which variant is better on the items the team wrote. Once the assistant is live, a team may want to compare two variants on real questions, such as two prompts or two models, each sent to a share of users. That is a comparison, and it is only as good as the way users were assigned to the variants. If the people who see one variant are the ones who ask harder questions, the difference reflects them and not the variant. Random assignment, and an interval read the way the rest of this chapter reads one, are what turn a live comparison into a test. Kohavi and colleagues' guide, which chapter 10 cites, recommends deciding the power and the effect to detect, and estimating the variability, before the test starts. Without that, a live comparison is a list of who saw which variant.

The feedback table in this section is not traffic. Its rows are generated. Each dev item's vote is drawn from a stated model with a fixed seed. An answer is right with probability 0.85. A user who sees a wrong answer gives a thumbs-down with probability 0.5 and a thumbs-up with 0.05. A user who sees a right answer gives a thumbs-down with 0.03 and a thumbs-up with 0.15. Most items get no vote at all. Whether each answer was right is drawn and then dropped, as it would be in a real log, so the table holds only the votes. It is a teaching device, and its figures are what the stated model produces. They say nothing about this assistant.

feedback-table.py Python 3.12.15 · numpy 2.5.5 · pydantic 2.14.3 · typing-extensions 4.16.0 通過
"""A generated table of thumbs-up and thumbs-down rows over the eighty dev items, with what such a table can show and what itcan't. The rows are generated from a stated model with a fixed seed, not taken from users, so the table is a teachingdevice only.The model: an answer is right with probability 0.85. A user who sees a wrong answer gives thumbs-down with probability 0.5 andthumbs-up with 0.05; one who sees a right answer gives thumbs-down with 0.03 and thumbs-up with 0.15. Most items get no vote.The table keeps the votes only: whether each answer was right is drawn and then dropped, as it would be in a real log."""import lib  # first: fixes the thread counts and seeds before anything else loadsimport numpy as npfrom lib import SEEDfrom lib.stats import pctfrom lib.testset import load_itemsP_RIGHT = 0.85P_DOWN_WRONG, P_UP_WRONG = 0.50, 0.05P_DOWN_RIGHT, P_UP_RIGHT = 0.03, 0.15rng = np.random.default_rng(SEED)  # one generator for the whole table, so the same seed gives the same rowsitems = [item.id for item in load_items(split="dev")]rows = []  # (item id, vote) for each item that got a votefor item_id in items:    right = rng.random() < P_RIGHT  # hidden: the table has no column for it    draw = rng.random()    down, up = (P_DOWN_RIGHT, P_UP_RIGHT) if right else (P_DOWN_WRONG, P_UP_WRONG)    if draw < down:        rows.append((item_id, "thumbs-down"))    elif draw < down + up:        rows.append((item_id, "thumbs-up"))print("generated rows from a stated model, not user feedback (seed 0)")print("item    vote")for item_id, vote in rows:    print(f"{item_id}  {vote}")ups = sum(vote == "thumbs-up" for _, vote in rows)downs = sum(vote == "thumbs-down" for _, vote in rows)voted = len({item_id for item_id, _ in rows})print(f"dev items: {len(items)}")print(f"items with a vote: {voted}")print(f"items nobody voted on: {len(items) - voted}")print(f"thumbs-down among those who voted: {pct(downs, ups + downs)}")print(f"thumbs-up among those who voted: {pct(ups, ups + downs)}")print("what the table can show: the share of each kind of vote, among the votes cast, with its n.")print("what it can't show: anything about the items nobody voted on; whether a thumbs-up answer was right.")
輸出python feedback-table.py
generated rows from a stated model, not user feedback (seed 0)
item    vote
q-0002  thumbs-down
q-0006  thumbs-down
q-0007  thumbs-down
q-0008  thumbs-up
q-0009  thumbs-up
q-0011  thumbs-up
q-0020  thumbs-down
q-0027  thumbs-down
q-0028  thumbs-up
q-0030  thumbs-up
q-0035  thumbs-up
q-0040  thumbs-down
q-0056  thumbs-down
q-0057  thumbs-down
q-0059  thumbs-down
q-0060  thumbs-up
q-0064  thumbs-down
q-0070  thumbs-down
q-0072  thumbs-up
q-0079  thumbs-down
q-0080  thumbs-down
dev items: 80
items with a vote: 21
items nobody voted on: 59
thumbs-down among those who voted: 61.9% (13 of 21)
thumbs-up among those who voted: 38.1% (8 of 21)
what the table can show: the share of each kind of vote, among the votes cast, with its n.
what it can't show: anything about the items nobody voted on; whether a thumbs-up answer was right.

Under the stated model, only 21 of the 80 dev items get a vote, and 59 get none. So the table says little about the other items.

A vote is not a correctness label, and this table shows why. Under the stated model, answers are right 85% of the time, yet most votes are thumbs-down, because a wrong answer draws a vote far more often. Among the votes cast, thumbs-down is 61.9% (13 of 21). Thumbs-up is 38.1% (8 of 21).

A table of votes can show the share of each kind of vote, among the votes cast, with the count beside it, and the example's last lines say exactly that. It cannot show whether a thumbs-up answer was right, since the table has no column for correctness. It says nothing about the items nobody voted on, and a vote comes from someone who chose to vote on what they chose to read.

Joachims and colleagues studied clicks on search results in 2005 and found them informative but biased, so a click is not an absolute judgment of relevance. They report that relative preferences derived from clicks are reasonably accurate on average, in their paper on interpreting clickthrough data. This book uses that finding as an analogy only, and it is the book's own reading. A thumbs-up is a vote, not a label. A right answer can still draw a thumbs-down, and the table has no way to tell which votes those are.

Two groups of users are compared differently from the paired items above. Each user sees one variant, so the two groups are separate, and the interval for their difference is a different calculation from the paired one this chapter uses. Chapter 8 covers the paired comparison this chapter uses. It does not cover an interval for two separate groups.

A vote table also cannot compare variants unless each vote is tied to the variant the user saw. This table has no variant column, so it cannot compare anything. A live comparison would need that column, the random assignment above, and an interval. If few items get a vote, as in this generated table, the interval would be wide, and most items would have no vote to compare.

Before a real vote table could be trusted, it would need three things. Each vote would need to be tied to the variant the user saw. The share of items that get a vote would need to be reported with its count, since a table of votes speaks only to the items that were voted on. And a sample of its votes would need checking against correctness, so that a thumbs-up or a thumbs-down can be read against what it is meant to measure.

The book's parts add up to an evaluation for a feature of your own. A labelled test set comes first (chapter 3), then checks in code (chapter 4). A judge with a rubric and a measured error follows (chapter 5, chapter 6). A comparison reports its interval and n (chapter 8), and the gate in CI reads all of it. Chapter 7 says where the judge stops being reliable. Chapter 9 and chapter 10 apply the same counts to grounding and declining.

The book leaves three subjects to others. A Primer on Search Relevance covers ranking metrics. Agent Harnesses covers tasks that take several steps. Observability for LLM Applications covers tracing one request through retrieval and the model, and counting what each request costs. It is planned, so it is named here and not linked.

Working rule: Don't compare variants from traffic without random assignment and the variant recorded beside each vote or reply. Read a vote table as a share of the votes cast, with its count, and never as a measure of correctness.

本章,於本版次中

版次 2026.11.3範例於 2026年11 月 21 通過測試本章 4 個範例 個範例中有 4 個通過

譯自英文書,版次 2026.11.3。

引用本節

格式

各種格式皆引用英文原書,本譯本即依循該原書。

每種格式都會標示本版。其連結會開啟目前版次,節標題會說明該節內文最後一次變動的時間。存取日期與擷取日期即為你開啟此頁的當日。

本頁內容

第 11 章,共 11 章: 把評估放進 CI,並在流量上比較各種變體

  1. 尊重區間的閘門
  2. Flag flaky items, never edit them
  3. 評估本身的測試
  4. Running it as a framework task
  5. Variants on traffic

返回頂端所有章節