把评估放进 CI,并在线上流量上比较各变体
阅读约需 44分钟 分钟,5节中含4个示例。
读完本章,你可以把评估做成一道门禁,仅当某项变更在可测量的意义上确实更差时才拦下它。你还能说明,对两个变体做线上比较能够说明什么,不能说明什么。
sablefile 助手依据该工具的文档作答,团队每周都会修改它的提示词、检索方式或模型。团队希望在变更合并之前就知道,助手是否变差了。只要通过率有任何下降就判定失败的门禁,看上去是稳妥的选择,实则不然。开发集只有 80 条,一条样本就会使通过率变动约 1.3 个百分点,因此直接比较原始通过率的门禁,会因为一两条样本的差异而拦下变更。团队若总是被噪声拦住,往往会学会绕过门禁,于是门禁便什么也保护不了。
第 1 章首次完整运行了整套评估:测试集、各项检查、评判器和回放的答案。本章沿用那次运行的成果,依次讨论三个部分:考虑区间的门禁、各次采样之间判定结果会发生变化的样本,以及评估自身代码的测试。随后,本章把这三个部分对应到评估框架的术语上,最后讨论在线上流量中比较各变体能够说明什么、不能说明什么。
尊重区间的门禁
门禁是阻止某项变更合并的规则。在本书中,这条规则只依赖一个数:候选与基线在相同开发集条目上的通过率配对差。这种比较是配对的,如第 8 章所定义:对每个条目,取候选的判定减去基线的判定,再对各条目取平均,区间则按条目做自助法(bootstrap)得到。由于两次运行回答的是相同的问题,两次运行结果一致的条目会在差值中相互抵消。
门禁只有一条规则:仅当区间的上界低于负阈值时才判为失败。上界是数据仍然允许的最不负的读数,因此,门禁失败意味着即使对数据做最乐观的解读,下降幅度仍然超过阈值。朴素的门禁会在差值的点估计低于负阈值时就判为失败,这样的门禁会把噪声误判为失败。本书这条规则的代价是,它会放过一些数据尚无法证实的真实下降,所以读取通过结果时要结合区间宽度,下面几段会作说明。
阈值是五个百分点。示例中的注释说明,这是本书的一个选择,而不是标准。在这个规模的开发集上,一个条目约合 1.3 个百分点,所以五个百分点相当于净差四个条目。
第一次运行是一次大幅下降,这是该评判器的测量结果。第二个模型 Qwen3-0.6B 比 4B 基线更小,它回答的是相同的开发集条目。评判器是 Qwen3-4B-Instruct-2507,即基线自身所用的模型,因此这一差距无法区分是助手更弱还是自我偏好所致。自我偏好解释了本书为何无法检验这一点。最后再读决定,因为它由区间得出。
1"""Runs the gate on second-model against the baseline over the eighty dev items: both pass rates with their counts, the paired2difference, its interval, the threshold and the decision. The gate fails only when the upper bound of the interval is below3minus the threshold, so a drop that the interval cannot rule out below the threshold does not fail it."""4import lib # first: fixes the thread counts and seeds before anything else loads56from lib.calibration import readable_verdicts7from lib.gate import FAIL, gate8from lib.judging import passed9from lib.prompts import REF_SHA25610from lib.stats import interval_points, pct11from lib.testset import load_items1213BASELINE = "judge-answers-baseline"14CANDIDATE = "judge-answers-second-model"15# Five points is this gate's threshold, chosen for the book, not a standard. With eighty dev items one item moves a rate16# by 1.25 points, so five points is four items net: a smaller threshold would ask the gate to see drops the interval17# cannot resolve at this size.18THRESHOLD_POINTS = 5.01920items = [item.id for item in load_items(split="dev")] # eighty dev items, sorted by id21baseline_found, baseline_unreadable = readable_verdicts(BASELINE, items, REF_SHA256)22candidate_found, candidate_unreadable = readable_verdicts(CANDIDATE, items, REF_SHA256)23for label, unreadable in (("baseline", baseline_unreadable), ("second-model", candidate_unreadable)):24 details = "".join(f' ({item_id}, "{first}")' for item_id, first in unreadable)25 print(f"unreadable verdicts, {label}: {len(unreadable)}{details}")26# An item the judge did not give a readable verdict on, under either run, is left out of both sides: a paired27# difference needs the same items under both runs.28excluded = {item_id for item_id, _ in baseline_unreadable + candidate_unreadable}29kept = [item_id for item_id in items if item_id not in excluded]30first = passed({item_id: baseline_found[item_id] for item_id in kept}) # 1.0 or 0.0 per item, in id order31second = passed({item_id: candidate_found[item_id] for item_id in kept})3233result = gate(first, second, threshold_points=THRESHOLD_POINTS)34print(f"dev items compared (n): {result.n}")35print(f"baseline pass rate: {pct(int(first.sum()), result.n)}")36print(f"second-model pass rate: {pct(int(second.sum()), result.n)}")37print(f"difference, second-model minus baseline: {100 * result.difference:+.1f} points")38print(f"{100 * result.level:.0f}% interval of the difference: {interval_points(result.lo, result.hi)}")39print(f"upper bound of the interval: {100 * result.hi:+.1f} points")40print(f"threshold: {THRESHOLD_POINTS:.1f} points")41if result.decision == FAIL:42 print(f"decision: fail, since the upper bound {100 * result.hi:+.1f} is below {-THRESHOLD_POINTS:+.1f}")43else:44 print(f"decision: pass, since the upper bound {100 * result.hi:+.1f} is not below {-THRESHOLD_POINTS:+.1f}")
python gate-real-drop.pyunreadable verdicts, baseline: 1 (q-0043, "VERDICT: PARTIAL")
unreadable verdicts, second-model: 0
dev items compared (n): 79
baseline pass rate: 78.5% (62 of 79)
second-model pass rate: 40.5% (32 of 79)
difference, second-model minus baseline: -38.0 points
95% interval of the difference: [-48.1, -26.6] points
upper bound of the interval: -26.6 points
threshold: 5.0 points
decision: fail, since the upper bound -26.6 is below -5.0
门禁失败了,原因在于区间,而不在于点估计。基线的通过率为 78.5% (62 of 79)。第二个模型的通过率为 40.5% (32 of 79)。
比较涵盖 79 个条目,而不是全部开发集条目,因为基线判定无法读取的条目会同时从两次运行中剔除。配对差要求两次运行使用相同的条目。差值(第二个模型减去基线)为 -38.0 points,其 95% 区间为 [-48.1, -26.6] points。
上界为 -26.6 points,阈值为 5.0 points,所以即使是上界也远低于阈值。这样的下降无法用区间的不确定性来解释掉,因此门禁正是为在这类下降上失败而设计的,这是该评判器的测量结果。决定为:fail, since the upper bound -26.6 is below -5.0。
较小的变更会给出不同的结果。假设变更使通过率下降得更少,或者数据集更小、区间变宽,使上界升到负五个百分点以上。此时门禁通过。它没有发现超过阈值的下降,但这并未证明候选与基线一样好。区间很宽的通过是一次弱通过,所以应当阅读区间宽度,而不只是看决定。在给定规模下区间会有多宽,以及一个较小的配对差需要多少条目才能分辨出来,第 8 章对此有详细推演。
门禁的局限在于规模。在 79 个条目上,上界必须远低于负阈值,门禁才会失败,所以较小的下降会通过。第 8 章在其假设的不一致率下估计,79 个条目足以以 80% 的功效显示十个百分点的差异,而五个百分点的差异需要 316 个。门禁是比该功效计算更严格的检验,所以真实的十个百分点的下降仍可能通过。阈值和数据集规模必须一并选定。
有些门禁会检查不止一个指标。每增加一个指标,就多一次噪声看起来像下降的机会,因此门禁要对比较次数做校正。有 m 个指标时,每个区间取 1 减去 0.05 除以 m 的水平:两个指标为 97.5%,三个指标约为 98.3%。这些检查合在一起,使误判失败的概率不高于百分之五。校正后的水平即第 8 章给出的那个,门禁使用同一个。重采样次数保持为 2,000,种子保持固定,因此重新运行会得到相同的区间。
门禁对每次变更都在开发集条目上运行,因为门禁必须每次都运行,而团队可能已在这些条目上调过参。留出集在每次发布决策时只运行一次,所依据的规则在运行前就已固定,如留出集,仅一次所述。此后需要一份新的留出集。
要把门禁放进 CI,可由一个作业对候选的记录计算门禁,并在决定为失败时使构建失败。门禁只读取逐条目的值,所以该作业不需要其他状态。未能运行的作业按门禁失败计,因为被跳过的门禁看起来会像通过。门禁自身的测试检查上述规则,它们与评估的其他测试一起运行,见评估自身的测试。
工作准则: 报告每个配对差时都附上其区间和 n,并让决定由上界得出。不要凭点估计就说某项变更更好或更差,也不要把区间很宽的通过当作候选与基线相当的证明。
这部分尚未翻译,因此以英文显示。
Flag flaky items, never edit them
An item flips when its sampled replies get different verdicts. Each dev item has five verdicts from the variance set, and an item is flaky when they are not all the same. The variance set and the flip rate, the share of items that flip, are introduced in Variance within one item. A team watches the flip rate to see how far the verdicts move when the assistant's replies do.
Each flagged item comes with its PASS count out of five, in id order, and the rule it is held to is printed beside the list.
1"""The dev items flagged as flaky: the judge's verdict differs across their five variance samples. Each flagged item prints its2PASS count out of five, sorted by id, and the count matches the flip rate in flip-rate.py, since both read the same per-item3counts. A flagged item is kept as it is, so the rule printed last is the whole policy: flagged, never edited or deleted."""4import lib # first: fixes the thread counts and seeds before anything else loads56from lib.testset import load_items7from lib.variance import VARIANCE_SAMPLES, item_pass_counts89VARIANCE = "judge-answers-variance"10SAMPLES = len(VARIANCE_SAMPLES)1112items = [item.id for item in load_items(split="dev")] # eighty dev items, sorted by id13counts = item_pass_counts(VARIANCE, items)14for row in counts:15 if not row.complete: # an unreadable sample is not a FAIL: the item is named here and not flagged16 samples = ", ".join(str(sample) for sample in row.unreadable)17 print(f"left out, unreadable verdict, {row.item_id}: samples {samples}")18flagged = [row for row in counts if row.flaky]19for row in flagged:20 print(f"{row.item_id}: {row.passes} of {SAMPLES} PASS")21print(f"flagged items: {len(flagged)}")22print(f"dev items compared (n): {sum(1 for row in counts if row.complete)}")23print("rule: flagged, never edited or deleted")
python flaky-items.pyleft out, unreadable verdict, q-0043: samples 1, 4
q-0047: 4 of 5 PASS
q-0058: 1 of 5 PASS
flagged items: 2
dev items compared (n): 79
rule: flagged, never edited or deleted
Few items flip on this set. The flagged items come to 2 of the 79 dev items compared. Both flagged items are worth reading.
On q-0047, the verdicts across its five replies are 4 of 5 PASS, so most of its replies passed. On q-0058, the verdicts are 1 of 5 PASS, so most of its replies failed.
One item is set aside rather than used. The item q-0043 is left out, since its verdict was unreadable in samples 1, 4. An unreadable sample is not a fail. It is a sample the example cannot read, so the item is neither flagged nor counted on that basis.
The rule is that a flagged item stays flagged, never edited or deleted. It stays in the set with the verdicts the judge gave. Editing it until the judge agrees more often would hide the very signal the flag exists to show. Deleting it would shrink the set and make the pass rate look steadier than it is. The flag does its work when someone reads it. A flagged item with one PASS in five is a question about the item, the rubric, the judge or the assistant's replies, and it should be looked at before the next change is judged on it.
The count is small, and it describes this assistant, this judge and these dev items. It shows that few items flip in this set. It does not show that other assistants, sampling settings or judges flip as rarely. A team that changes any of them should run the variance set again before it trusts a low flip rate.
A flagged item can also move a paired difference from one run to the next. When a gate fails on a difference that depends on a flagged item, read the flag before the change.
Working rule: Flag an item whose verdicts differ across its samples, print its count beside the flag, and leave the item as it is. Report the flip rate with its denominator, and say that a low rate holds for this assistant, this judge and this set only.
评测自身的测试
评测本身就是代码,而代码的错误可能以结果的面貌出现。自助法(bootstrap)、判定读取器或配对差值中的一个缺陷,会像助手发生变化一样改变某个数字。因此,评测有自己的测试,放在本书代码库内的 tests 文件夹中,本书新增的每个模块都带有各自的测试。这些测试检查的是打分代码,而不是助手。测试通过,只说明代码统计的正是它所声称统计的内容,并不说明助手回答得好不好。
这些测试在同一个进程中运行,结果按模块逐个统计。随后,示例打印 pytest 的退出码,版本为 9.2.0。退出码参考将 0 列为所有测试均已收集并通过,将 5 列为未收集到任何测试。因此,没有测试任何内容的一次运行会显示为 5,而不是悄无声息地通过。
1"""Runs the book's own tests, lib/tests, in this process and prints how many passed and failed in each module, then pytest's2exit code. Exit code 0 means every test was collected and passed. Exit code 5 means nothing was collected, so the example3prints the code it got rather than a silent pass. No paths and no timings are printed: they vary from run to run."""4import lib # first: fixes the thread counts and seeds before anything else loads56from collections import Counter7from pathlib import Path89import pytest1011TESTS = Path(lib.__file__).resolve().parent / "tests" # read here, never printed121314class Outcomes:15 """Counts each test's outcome by module. A failed setup or teardown counts as a failure of its test."""1617 def __init__(self) -> None:18 self.passed = Counter()19 self.failed = Counter()2021 def pytest_runtest_logreport(self, report) -> None:22 # The node id starts with the file's path; keep only the module's name, so no path is printed.23 module = Path(report.nodeid.split("::")[0]).stem24 if report.when == "call" and report.passed:25 self.passed[module] += 126 elif report.failed:27 self.failed[module] += 1282930collector = Outcomes()31# -p no:terminal turns off pytest's own printing, which carries timings. That also removes the -q option, which belongs to32# the terminal reporter, so it is not passed. -p no:cacheprovider keeps .pytest_cache out of the working copy.33exit_code = pytest.main(34 [str(TESTS), "-p", "no:terminal", "-p", "no:cacheprovider"],35 plugins=[collector],36)3738modules = sorted(path.stem for path in TESTS.glob("test_*.py")) # every module, even one with no failures39print("tests by module (passed, failed):")40for module in modules:41 print(f" {module}: {collector.passed[module]} passed, {collector.failed[module]} failed")42print(f"total: {sum(collector.passed.values())} passed, {sum(collector.failed.values())} failed")43print(f"pytest exit code: {int(exit_code)}")
python own-tests-run.pytests by module (passed, failed):
test_bench: 11 passed, 0 failed
test_bias: 14 passed, 0 failed
test_both_orders: 14 passed, 0 failed
test_calibration: 9 passed, 0 failed
test_calibration_bootstrap: 8 passed, 0 failed
test_contract: 5 passed, 0 failed
test_gate: 11 passed, 0 failed
test_guard: 4 passed, 0 failed
test_judging: 3 passed, 0 failed
test_stats: 10 passed, 0 failed
test_support_verdict: 5 passed, 0 failed
test_testset: 5 passed, 0 failed
test_variance: 15 passed, 0 failed
total: 114 passed, 0 failed
pytest exit code: 0
下表给出每个模块的结果,并在旁边列出该模块所测试的内容。
| 模块 | 测试内容 | 结果 |
|---|---|---|
| test_bench | 基准测试的加载器与打分器 | 11 passed, 0 failed |
| test_bias | 长度分段、无法读取的判定、翻转方向与分段差距 | 14 passed, 0 failed |
| test_both_orders | 成对比较读取器与两种顺序:映射回各个回复,以及四种类型 | 14 passed, 0 failed |
| test_calibration | 评判者与人工的计数、一致率、kappa、TPR、TNR 及校正后的通过率 | 9 passed, 0 failed |
| test_calibration_bootstrap | 交叉点以及校正后通过率的自助法区间 | 8 passed, 0 failed |
| test_contract | 评判者与助手必须遵守的回复和判定格式 | 5 passed, 0 failed |
| test_gate | 闸门的规则:仅当上界低于负的阈值时才判为失败 | 11 passed, 0 failed |
| test_guard | 拒答守卫的规则及其阈值,基于手工构造的行 | 4 passed, 0 failed |
| test_judging | 读取评判者的判定,以及配对差值的类型 | 3 passed, 0 failed |
| test_stats | 打印辅助函数与带种子的自助法 | 10 passed, 0 failed |
| test_support_verdict | 忠实性判定读取器:首行为 SUPPORTED 或 UNSUPPORTED | 5 passed, 0 failed |
| test_testset | 加载测试集,以及跨条目检查的规则 | 5 passed, 0 failed |
| test_variance | 方差集上每个条目的判定计数 | 15 passed, 0 failed |
总计列在最后:114 passed, 0 failed,退出码为 0,即所有已收集的测试均通过的运行所对应的退出码。
应把这张表理解为对测量工具的检查,而不是对助手的检查。在闸门作业中先运行这些测试,这样,有缺陷的打分器会在读取通过率之前就让作业停止。输出中不打印任何随运行而变化的内容:没有耗时,也没有路径。读者可以重新运行代码,并逐行比对输出。
工作准则: 在任何读取数字的运行之前,先运行评测自身的测试,并将退出码 5 视为失败,而不是通过。打分代码中只要有一个测试失败,无论数字显示什么,这些数字都还不可信。
作为框架任务运行
已经在使用评估框架的团队,可以把本书的各个部分对应到框架的术语上,先看一遍这种对应会有帮助。在 Inspect AI 0.3.291 中(其更新日志标注的日期为 2026 年 11 月 17 日),任务是一次评估的配方,最少由数据集、求解器和评分器三部分构成,该版本的框架文档对此有说明。按这种读法,测试集就是数据集,每个条目对应一个样本;助手先检索、再回答的路径就是求解器;各项检查与评判器共同构成评分器。本书并不运行这一对应,也不声称运行过。本书只保留一个示例环境:一组文件,每一章的示例都导入它,并固定在相同的版本上,因此重放模型、检查和关卡在每一章中都是同一份代码。本章说明各部分在框架中的位置,使使用框架的读者能够把这些部分迁移过去,而本书的示例保持原样。
**工作准则:**把一项评估对应到框架上时,要写明框架的版本及其文档的日期。测试集、检查和重放模型仍以本书示例所运行的文件为准;在有人实际运行之前,这份对应只能视为对文档的一种解读。
这部分尚未翻译,因此以英文显示。
Variants on traffic
Offline, the test set and the replayed answers can say which variant is better on the items the team wrote. Once the assistant is live, a team may want to compare two variants on real questions, such as two prompts or two models, each sent to a share of users. That is a comparison, and it is only as good as the way users were assigned to the variants. If the people who see one variant are the ones who ask harder questions, the difference reflects them and not the variant. Random assignment, and an interval read the way the rest of this chapter reads one, are what turn a live comparison into a test. Kohavi and colleagues' guide, which chapter 10 cites, recommends deciding the power and the effect to detect, and estimating the variability, before the test starts. Without that, a live comparison is a list of who saw which variant.
The feedback table in this section is not traffic. Its rows are generated. Each dev item's vote is drawn from a stated model with a fixed seed. An answer is right with probability 0.85. A user who sees a wrong answer gives a thumbs-down with probability 0.5 and a thumbs-up with 0.05. A user who sees a right answer gives a thumbs-down with 0.03 and a thumbs-up with 0.15. Most items get no vote at all. Whether each answer was right is drawn and then dropped, as it would be in a real log, so the table holds only the votes. It is a teaching device, and its figures are what the stated model produces. They say nothing about this assistant.
1"""A generated table of thumbs-up and thumbs-down rows over the eighty dev items, with what such a table can show and what it2can't. The rows are generated from a stated model with a fixed seed, not taken from users, so the table is a teaching3device only.45The model: an answer is right with probability 0.85. A user who sees a wrong answer gives thumbs-down with probability 0.5 and6thumbs-up with 0.05; one who sees a right answer gives thumbs-down with 0.03 and thumbs-up with 0.15. Most items get no vote.7The table keeps the votes only: whether each answer was right is drawn and then dropped, as it would be in a real log."""8import lib # first: fixes the thread counts and seeds before anything else loads910import numpy as np1112from lib import SEED13from lib.stats import pct14from lib.testset import load_items1516P_RIGHT = 0.8517P_DOWN_WRONG, P_UP_WRONG = 0.50, 0.0518P_DOWN_RIGHT, P_UP_RIGHT = 0.03, 0.151920rng = np.random.default_rng(SEED) # one generator for the whole table, so the same seed gives the same rows21items = [item.id for item in load_items(split="dev")]22rows = [] # (item id, vote) for each item that got a vote23for item_id in items:24 right = rng.random() < P_RIGHT # hidden: the table has no column for it25 draw = rng.random()26 down, up = (P_DOWN_RIGHT, P_UP_RIGHT) if right else (P_DOWN_WRONG, P_UP_WRONG)27 if draw < down:28 rows.append((item_id, "thumbs-down"))29 elif draw < down + up:30 rows.append((item_id, "thumbs-up"))3132print("generated rows from a stated model, not user feedback (seed 0)")33print("item vote")34for item_id, vote in rows:35 print(f"{item_id} {vote}")3637ups = sum(vote == "thumbs-up" for _, vote in rows)38downs = sum(vote == "thumbs-down" for _, vote in rows)39voted = len({item_id for item_id, _ in rows})40print(f"dev items: {len(items)}")41print(f"items with a vote: {voted}")42print(f"items nobody voted on: {len(items) - voted}")43print(f"thumbs-down among those who voted: {pct(downs, ups + downs)}")44print(f"thumbs-up among those who voted: {pct(ups, ups + downs)}")45print("what the table can show: the share of each kind of vote, among the votes cast, with its n.")46print("what it can't show: anything about the items nobody voted on; whether a thumbs-up answer was right.")
python feedback-table.pygenerated rows from a stated model, not user feedback (seed 0)
item vote
q-0002 thumbs-down
q-0006 thumbs-down
q-0007 thumbs-down
q-0008 thumbs-up
q-0009 thumbs-up
q-0011 thumbs-up
q-0020 thumbs-down
q-0027 thumbs-down
q-0028 thumbs-up
q-0030 thumbs-up
q-0035 thumbs-up
q-0040 thumbs-down
q-0056 thumbs-down
q-0057 thumbs-down
q-0059 thumbs-down
q-0060 thumbs-up
q-0064 thumbs-down
q-0070 thumbs-down
q-0072 thumbs-up
q-0079 thumbs-down
q-0080 thumbs-down
dev items: 80
items with a vote: 21
items nobody voted on: 59
thumbs-down among those who voted: 61.9% (13 of 21)
thumbs-up among those who voted: 38.1% (8 of 21)
what the table can show: the share of each kind of vote, among the votes cast, with its n.
what it can't show: anything about the items nobody voted on; whether a thumbs-up answer was right.
Under the stated model, only 21 of the 80 dev items get a vote, and 59 get none. So the table says little about the other items.
A vote is not a correctness label, and this table shows why. Under the stated model, answers are right 85% of the time, yet most votes are thumbs-down, because a wrong answer draws a vote far more often. Among the votes cast, thumbs-down is 61.9% (13 of 21). Thumbs-up is 38.1% (8 of 21).
A table of votes can show the share of each kind of vote, among the votes cast, with the count beside it, and the example's last lines say exactly that. It cannot show whether a thumbs-up answer was right, since the table has no column for correctness. It says nothing about the items nobody voted on, and a vote comes from someone who chose to vote on what they chose to read.
Joachims and colleagues studied clicks on search results in 2005 and found them informative but biased, so a click is not an absolute judgment of relevance. They report that relative preferences derived from clicks are reasonably accurate on average, in their paper on interpreting clickthrough data. This book uses that finding as an analogy only, and it is the book's own reading. A thumbs-up is a vote, not a label. A right answer can still draw a thumbs-down, and the table has no way to tell which votes those are.
Two groups of users are compared differently from the paired items above. Each user sees one variant, so the two groups are separate, and the interval for their difference is a different calculation from the paired one this chapter uses. Chapter 8 covers the paired comparison this chapter uses. It does not cover an interval for two separate groups.
A vote table also cannot compare variants unless each vote is tied to the variant the user saw. This table has no variant column, so it cannot compare anything. A live comparison would need that column, the random assignment above, and an interval. If few items get a vote, as in this generated table, the interval would be wide, and most items would have no vote to compare.
Before a real vote table could be trusted, it would need three things. Each vote would need to be tied to the variant the user saw. The share of items that get a vote would need to be reported with its count, since a table of votes speaks only to the items that were voted on. And a sample of its votes would need checking against correctness, so that a thumbs-up or a thumbs-down can be read against what it is meant to measure.
The book's parts add up to an evaluation for a feature of your own. A labelled test set comes first (chapter 3), then checks in code (chapter 4). A judge with a rubric and a measured error follows (chapter 5, chapter 6). A comparison reports its interval and n (chapter 8), and the gate in CI reads all of it. Chapter 7 says where the judge stops being reliable. Chapter 9 and chapter 10 apply the same counts to grounding and declining.
The book leaves three subjects to others. A Primer on Search Relevance covers ranking metrics. Agent Harnesses covers tasks that take several steps. Observability for LLM Applications covers tracing one request through retrieval and the model, and counting what each request costs. It is planned, so it is named here and not linked.
Working rule: Don't compare variants from traffic without random assignment and the variant recorded beside each vote or reply. Read a vote table as a share of the votes cast, with its count, and never as a measure of correctness.
本章,在本版次中
版次 2026.11.3示例于 2026年11月21日 通过本章 4个示例 个示例中有 4 个通过
译自英文版图书,版次 2026.11.3。