评判回答并发现回归
阅读约需 97分钟 分钟,8节中含6个示例。
学完本章,你将能够:同时用多项指标为智能体的一次运行打分;只在结果无法直接比较时才使用模型作为评判者;对照任何人都能重新计算的参考标注来校准这个评判者;看出分数在两次运行之间的波动幅度;并在 CI 中基于已记录的运行,对提示词、上下文或指标的变更设置门禁。
几乎每一章都改动过商店的智能体。第 4 章为提示词增加了上下文,第 6 章把智能体迁移到语义层的指标上。随后,第 8 章重写了提示词,使智能体在拒绝时给出理由。每次改动都在黄金问题的开发集上评分。黄金问题是商店的 48 个带有已验证答案的问题,每个问题只运行一次,这样的评分能显示大的变化,却显示不出小的变化。第 8 章的新提示词修正了一个开发集问题,又弄坏了另一个,即 gq-008(2025 年的总收入),而第 8 章只能说,像这样的单次翻转“可能只是偶然”。
本章构建用于解决这类问题的度量方法。它报告一次运行的多项指标,而不只是准确率,并附上区间,说明这个数字在多大程度上可以信任。它加入了一个评判者,也就是一次模型调用,用来评分黄金比对无法处理的内容,并且在信任它之前先对它进行度量。它把每个问题运行五次以观察方差,并把以上所有内容整合为一道门禁,让 CI 作业在每次变更时都能依据录制的记录,离线运行。
这部分尚未翻译,因此以英文显示。
Score a run on more than accuracy
A single accuracy figure answers one question, how often the agent was right, and hides three others that matter as much: what it did when it was wrong, what each answer cost, and how far the figure would move on another sample of questions. An agent that gives a wrong number and one that declines both lose a point of accuracy, but only one of them puts a wrong number in a deck. So a scorecard reports wrong numbers, declines and failures apart, the decline measures from chapter 8, and the cost in model calls and prompt tokens, beside accuracy.
The figure also needs an interval. On a set the size of a golden split, each question is several points, and the interval that most people learn first, the Wald interval, behaves badly for small sets. Brown, Cai and DasGupta recommend "the Wilson interval or the equal-tailed Jeffreys prior interval for small n" (Brown et al.). In words, Wilson's interval inverts the test of a proportion: it holds every true accuracy that the observed one wouldn't reject at the chosen level, so it never runs below 0 or above 1. For k correct of n, with p = k / n and z = 1.96 for about 95%:
centre = (p + z² / 2n) / (1 + z² / n)
half = z × √( p(1 − p) / n + z² / 4n² ) / (1 + z² / n)
interval = centre − half to centre + half
wilson() in lib/regressions.py computes it in a few lines. statsmodels implements the same interval as proportion_confint(method="wilson") (statsmodels); the book's pins don't include statsmodels, so the example doesn't call it. replay() runs one of the book's agents over golden questions from its recordings, and Scorecard scores the runs. The scorecard that follows scores three agents on the 21 answer and decline questions of each split: chapter 3's schema-only SQL, chapter 6's metrics-first agent and chapter 8's declining agent. Forbidden questions are left out, because chapter 9 decides them below the model.
The test split is held out, and the book scores it only for a number it reports. Chapters 3 and 6 recorded their agents on test once. This chapter recorded the declining agent on test once, in the recording decline-runs-test, with chapter 8's prompt unchanged, and changed nothing after it. These are the book's final test numbers for its agents.
1"""Score the book's agents on more than accuracy, on dev to compare them and on the held-out test split once.2The chapter's points rest on the dev and test lines for correct answers, wrong numbers and decline recall,3and on the fixed and broken lines for the declining prompt."""4from lib.golden import load_golden5from lib.regressions import Scorecard, replay6from lib.semantic import load7from lib.warehouse import connect89con, layer = connect(), load()10# Answer and decline questions; forbidden questions are decided below the model, by chapter 9's policies.11questions = [q for q in load_golden() if q.kind in ("answer", "decline")]12for split in ("dev", "test"):13 in_split = [q for q in questions if q.split == split]14 print(f"{split} questions: {len(in_split)}")15 runs = {agent: replay(con, layer, agent, in_split) for agent in ("schema-only", "metrics-first", "declining")}16 for agent, agent_runs in runs.items():17 print("\n".join(Scorecard(agent_runs).lines(f"{split} {agent}")))18 for label, was, now in (("fixed", False, True), ("broken", True, False)):19 # both agents' runs follow the same question order, so zip pairs one question's two outcomes20 ids = [a.question.question_id for a, b in zip(runs["metrics-first"], runs["declining"])21 if (a.outcome == "correct") == was and (b.outcome == "correct") == now]22 print(f"{split} {label} by the declining prompt: {', '.join(ids) or 'none'}")
python scorecard.pydev questions: 21
dev schema-only correct: 2 of 21
dev schema-only accuracy: 0.10 (95% interval 0.03 to 0.29)
dev schema-only wrong numbers: 11
dev schema-only errors or step limits: 8
dev schema-only decline recall: 0.00
dev schema-only decline precision: no declines
dev schema-only coverage: 0.62
dev schema-only selective risk: 0.85
dev schema-only model calls per question: 1.0
dev schema-only prompt tokens per question: 827
dev metrics-first correct: 15 of 21
dev metrics-first accuracy: 0.71 (95% interval 0.50 to 0.86)
dev metrics-first wrong numbers: 5
dev metrics-first errors or step limits: 1
dev metrics-first decline recall: 0.75
dev metrics-first decline precision: 1.00
dev metrics-first coverage: 0.81
dev metrics-first selective risk: 0.29
dev metrics-first model calls per question: 1.8
dev metrics-first prompt tokens per question: 2,196
dev declining correct: 15 of 21
dev declining accuracy: 0.71 (95% interval 0.50 to 0.86)
dev declining wrong numbers: 4
dev declining errors or step limits: 1
dev declining decline recall: 1.00
dev declining decline precision: 0.80
dev declining coverage: 0.71
dev declining selective risk: 0.27
dev declining model calls per question: 1.9
dev declining prompt tokens per question: 2,602
dev fixed by the declining prompt: gq-020
dev broken by the declining prompt: gq-008
test questions: 21
test schema-only correct: 3 of 21
test schema-only accuracy: 0.14 (95% interval 0.05 to 0.35)
test schema-only wrong numbers: 10
test schema-only errors or step limits: 8
test schema-only decline recall: 0.00
test schema-only decline precision: no declines
test schema-only coverage: 0.62
test schema-only selective risk: 0.77
test schema-only model calls per question: 1.0
test schema-only prompt tokens per question: 829
test metrics-first correct: 9 of 21
test metrics-first accuracy: 0.43 (95% interval 0.24 to 0.63)
test metrics-first wrong numbers: 5
test metrics-first errors or step limits: 1
test metrics-first decline recall: 1.00
test metrics-first decline precision: 0.40
test metrics-first coverage: 0.48
test metrics-first selective risk: 0.50
test metrics-first model calls per question: 1.5
test metrics-first prompt tokens per question: 1,854
test declining correct: 8 of 21
test declining accuracy: 0.38 (95% interval 0.21 to 0.59)
test declining wrong numbers: 4
test declining errors or step limits: 3
test declining decline recall: 0.75
test declining decline precision: 0.33
test declining coverage: 0.43
test declining selective risk: 0.44
test declining model calls per question: 1.9
test declining prompt tokens per question: 2,622
test fixed by the declining prompt: none
test broken by the declining prompt: gq-043
On dev, both agents get 15 of 21 right, an accuracy of 0.71 (95% interval 0.50 to 0.86), but not on the same questions. The declining prompt fixed gq-020 and broke gq-008, the two flips chapter 8 found.
The difference is in the misses. The declining agent gave 4 wrong numbers against metrics-first's 5, and its decline recall is 1.00 against 0.75.
It pays for that in coverage, 0.71 against 0.81, and in prompt tokens, 2,602 against 2,196 a question.
Schema-only SQL gets 2 of 21 right, and of the numbers it gives, the share that is wrong, its selective risk, is 0.85.
On test, the declining prompt fixed no question and broke one, gq-043, "What was revenue in Q2?", which chapter 6's agent met with a question back and the declining agent ran out of model calls on. The declining agent gets 8 of 21 against 9 of 21 for metrics-first, and schema-only SQL gets 3 of 21.
Its wrong numbers fell from 5 to 4, because one wrong answer became a run that hit the step limit, not a right one. The intervals of the two agents overlap almost entirely on both splits. Two totals with overlapping intervals can't tell you whether a change helped, which is why the gate compares the same questions in pairs.
Working rule: Report every run as a scorecard: correct with a Wilson interval, wrong numbers, declines and their recall and precision, failures by kind, and calls and tokens per question. Compare agents on the same questions in the same table. Score the test split once, when you report a final number, and say that you did; when two agents' intervals overlap, don't call either better from the totals.
这部分尚未翻译,因此以英文显示。
Judge only what can't be compared
results_match(), the comparison chapter 3 built, grades a number against a golden result. Much of what the agent says isn't a number. A decline's reason, the sentence that tells the person why no number came, has no golden result to match: two good reasons can share no words, and a fluent reason can be false. For text like that, the common tool is a judge, a model call that grades an answer against a rubric and returns a verdict. A judge is cheap and tireless, and it has known biases. Zheng and colleagues, who found that strong judges matched human preferences about as often as humans matched each other, also name "position, verbosity, and self-enhancement biases" (Zheng et al.): a judge can favour the answer shown first, the longer answer, and its own. A later study that catalogued twelve potential biases found that strong judges still carry significant bias on particular tasks (Ye et al.).
So use a judge only where a result can't be compared directly, and design each call so that a bias would show. The shop's judge is the book's own model, the same one the agent uses, called with a separate prompt, JUDGE_PAIR in lib/prompts.py. It sees a question and two reasons for declining it, labelled A and B, and names the better one, or a tie, as JSON. The rubric asks for a reason that says accurately why the shop's data can't answer and what the person could ask instead. The cases are every golden question, on either split, that both chapter 6's metrics-first agent and chapter 8's declining agent declined: 8 pairs. Only 3 of them, gq-021, gq-042 and gq-045, are decline questions. The other 5, gq-032, gq-033, gq-034, gq-037 and gq-041, are answer questions the warehouse can answer, the gaps in the layer chapter 6 found, and both agents declined them wrongly. On those, the right reply was a number, so the judge compares two wrong declines. Each pair is judged twice, once with each agent's reason first, so a verdict that follows the position rather than the text changes with the order. parse_verdict() in lib/judge.py reads each reply, and the recording judge-verdicts holds every call.
1"""Judge only what can't be compared: two reasons for declining the same question, shown in both orders.2The points about position, length and which reasons won rest on the same-verdict, consistent-winner and3won-in-both-orders lines, and the answerable lines show which reasons say the data lacks what was asked."""4import re56from lib.judge import judge_pairs, recorded_cases7from lib.model import RecordedModel8from lib.semantic import load9from lib.warehouse import connect1011con, layer = connect(), load()12_, pairs = recorded_cases(con, layer)13results = judge_pairs(RecordedModel("judge-verdicts"), pairs)14GROUPS = {"decline": "decline questions", "answer": "answerable questions both agents declined"}15AGENTS = ("metrics-first", "declining")16# a reason that says the data does not include or contain what was asked17BLAMES_DATA = re.compile(r"\bdata (does not|doesn't) (include|contain)")18# words a reply uses to offer a path forward or reframe the question19ALTERNATIVE = ("alternative", "instead", "reframe", "path forward")20for r in results:21 print(f"{r.case.question.question_id} ({r.case.question.kind}): metrics-first first -> {r.first.label}; "22 f"declining first -> {r.second.label}")23n = len(results)24print(f"pairs: {n}")25for group, name in GROUPS.items():26 print(f"pairs on {name}: {sum(r.case.question.kind == group for r in results)}")27# second_as_first maps the second order's label back to the first order's labels28print(f"same verdict in both orders: {sum(r.first.label == r.second_as_first for r in results)} of {n}")29print(f"first reason won in both orders: {sum(r.first.label == r.second.label == 'A' for r in results)} of {n}")30print(f"second reason won in both orders: {sum(r.first.label == r.second.label == 'B' for r in results)} of {n}")31for agent in AGENTS:32 print(f"{agent} reason won in both orders: {sum(r.winner == agent for r in results)}")33for group, name in GROUPS.items():34 for agent in AGENTS:35 print(f"{name}, {agent} reason won in both orders: "36 f"{sum(r.winner == agent and r.case.question.kind == group for r in results)}")37# longer_won compares the winning reason's length with the losing reason's, in characters38print(f"consistent winner was the longer reason: {sum(r.longer_won for r in results)}")39answerable = [r.case for r in results if r.case.question.kind == "answer"]40for agent, reasons in (("metrics-first", [c.first for c in answerable]), ("declining", [c.second for c in answerable])):41 said = sum(bool(BLAMES_DATA.search(x)) for x in reasons)42 print(f"answerable questions, {agent} reasons that say the data lacks it: {said} of {len(answerable)}")43replies = [v for r in results for v in (r.first, r.second)]44suggests = sum(any(w in v.reason.lower() for w in ALTERNATIVE) for v in replies)45print(f"judge replies that suggest an alternative or a reframe: {suggests} of {len(replies)}")
python judge-an-explanation.pygq-021 (decline): metrics-first first -> A; declining first -> A
gq-032 (answer): metrics-first first -> B; declining first -> A
gq-033 (answer): metrics-first first -> A; declining first -> B
gq-034 (answer): metrics-first first -> A; declining first -> B
gq-037 (answer): metrics-first first -> A; declining first -> B
gq-041 (answer): metrics-first first -> A; declining first -> B
gq-042 (decline): metrics-first first -> B; declining first -> A
gq-045 (decline): metrics-first first -> A; declining first -> B
pairs: 8
pairs on decline questions: 3
pairs on answerable questions both agents declined: 5
same verdict in both orders: 7 of 8
first reason won in both orders: 1 of 8
second reason won in both orders: 0 of 8
metrics-first reason won in both orders: 5
declining reason won in both orders: 2
decline questions, metrics-first reason won in both orders: 1
decline questions, declining reason won in both orders: 1
answerable questions both agents declined, metrics-first reason won in both orders: 4
answerable questions both agents declined, declining reason won in both orders: 1
consistent winner was the longer reason: 6
answerable questions, metrics-first reasons that say the data lacks it: 1 of 5
answerable questions, declining reasons that say the data lacks it: 5 of 5
judge replies that suggest an alternative or a reframe: 9 of 16
The judge gave the same verdict in both orders on 7 of 8 pairs, and the one that changed, gq-021, the cost of shipping, followed whichever reason came first. That is position bias in the one place a pair was close, since the two reasons say nearly the same thing.
Of the consistent verdicts, chapter 6's reason won 5 and chapter 8's 2, and the longer reason won 6 of them. That is what verbosity bias predicts, though 8 pairs can't separate it from longer reasons being better.
The two groups of pairs tell different stories. On the answerable questions, the judge preferred chapter 6's reason in 4 of the 5 pairs, and chapter 8's in 1.
Chapter 8's reasons say the data doesn't include what was asked in 5 of 5 answerable pairs, which is false: the warehouse holds raw.customers.email for gq-037 and fct_orders.discount_eur for gq-041. Chapter 6's reasons say so in 1 of 5; the others say the metrics don't include it, which is true of the semantic layer. So on these pairs the judge may have preferred the more accurate reason, chapter 6's, and accuracy and length point the same way.
On the decline questions, the case the rubric was written for, chapter 6's reason won 1 of the 3 pairs, chapter 8's 1, and the third changed with the order. That is too few pairs to say whose reasons are better.
Reading the verdicts shows a bigger problem than order. Of the judge's replies, 9 of 16 praise a reason for "offering a constructive path", an alternative, or a way to reframe the question, or fault one for lacking it, and neither agent's reason does that in any of these pairs. For gq-032, gross margin, the judge preferred chapter 8's reason for explaining the calculation, and the calculation that reason gives, gross revenue minus net revenue over gross revenue, is not a gross margin: the golden SQL subtracts the products' unit cost from net revenue. The judge rewarded a confident wrong definition. A verdict on text is only as good as the judge's grip on the facts behind it, which is why the next section measures the judge where the facts are known.
Working rule: Use a judge only for what can't be compared with a result, such as reasons, explanations and questions back, and keep results_match() for every number. Judge each pair in both orders and count the verdicts that change, record every verdict, and read a sample of the judge's reasons, not only its labels. Don't let a judge's preference between two texts decide a release until you have measured that judge on cases with known answers.
这部分尚未翻译,因此以英文显示。
Calibrate the judge against reference labels
A judge's verdicts on reasons can't be checked against a golden result, so measure the judge first on a task where they can. The shop's golden questions offer one. Ask the same model to grade the agent's answers to answer questions as correct or incorrect, without showing it the golden result, and compare its verdicts with labels that results_match() computes from the golden result. Reference labels computed by results_match() are nobody's opinion: anyone can recompute them from questions.toml and the recordings, and when one looks wrong, the fix is to the golden question, where it's open to correction like the rest of the book. Calibrating a judge against reference labels measures how often the judge agrees with a check you trust, and in which direction it errs.
Agreement needs a chance correction. If most answers are wrong, a judge that calls everything wrong agrees most of the time and has learned nothing; Artstein and Poesio describe the case where raw agreement is high and chance-corrected agreement can still be low (Artstein and Poesio). Cohen's kappa and Scott's pi both subtract the agreement that chance would give, and differ only in how they estimate it: Cohen's kappa estimates chance agreement from each rater's own label shares, and Scott's pi estimates chance agreement from the two raters' shares pooled. Thakur and colleagues dropped kappa for pi in a study of judges, because with kappa "the chance agreement adjusts for the observed average differences between raters, which is in fact part of what we intend to measure" (Thakur et al.). A judge that says "correct" more often than the reference is exactly what you want the statistic to catch, so we report both, and read the lower. In formulas, with p_o the observed agreement:
p_o = cases where the judge and the reference agree / cases
chance, κ = Σ over labels (reference's share of the label × judge's share of the label)
chance, π = Σ over labels (the label's share of both raters' labels pooled)²
kappa, pi = (p_o − chance) / (1 − chance)
agreement() in lib/judge.py computes all three; scikit-learn's cohen_kappa_score uses the same definition of kappa (scikit-learn). The prompt, JUDGE_ANSWER, gives the judge the shop's glossary, the question, the role, the metric and SQL behind the answer and its result. The cases are every distinct answer with a number that schema-only SQL, the metrics-first agent and the declining agent gave to an answer question, on both splits, from their recordings: 56 in all. The judge prompts were written once, and no verdict changed them. What the judge is shown changed once. Schema-only SQL's query for gq-009 has no ORDER BY, so its rows came back in another order on replay, and the prompt built from them no longer matched its recording. shown() in lib/judge.py now sorts the rows of any query without ORDER BY before the judge sees them. The recording made before that fix was set aside, and every judge call was recorded again from scratch with the fix in place, as the manifest of judge-verdicts says. Every reply matched the set-aside recording's, so no number in this chapter moved.
1"""Calibrate the judge: its verdicts on the book's recorded answers against labels from results_match().2The overall and per-agent agreement lines, and the one-verdict kappa ranges, are the figures the chapter's points rest on."""3from lib.judge import agreement, answer_messages, kappa_one_verdict, parse_verdict, recorded_cases4from lib.model import RecordedModel5from lib.semantic import load6from lib.warehouse import connect78con, layer = connect(), load()9judge = RecordedModel("judge-verdicts")10cases, _ = recorded_cases(con, layer) # every distinct answer with a number, with its reference label11judged = {c.case_id: parse_verdict(judge.complete(answer_messages(c.question, c.answer))) for c in cases}12verdicts = {case_id: v.label for case_id, v in judged.items()}13print(f"cases: {len(cases)}")14print(f"reference correct: {sum(c.reference == 'correct' for c in cases)}")15print(f"judge correct: {list(verdicts.values()).count('correct')}")16print(f"judge unparsed: {list(verdicts.values()).count('unparsed')}")17# a case ID ends with the agent that gave the answer, so the metric agents' cases are all the rest18subsets = {"all": cases, "schema-only": [c for c in cases if c.case_id.endswith("schema-only")],19 "metric agents": [c for c in cases if not c.case_id.endswith("schema-only")]}20for name, subset in subsets.items():21 a = agreement([c.reference for c in subset], [verdicts[c.case_id] for c in subset])22 print(f"{name} cases: {a.n}")23 print(f"{name} cases the reference labels correct: {sum(c.reference == 'correct' for c in subset)}")24 print(f"{name} observed agreement: {a.observed:.2f}")25 print(f"{name} Cohen's kappa: {a.kappa:.2f}")26 print(f"{name} Scott's pi: {a.pi:.2f}")27 # kappa with each single verdict flipped in turn, to show how far one case moves it28 low, high = kappa_one_verdict([c.reference for c in subset], [verdicts[c.case_id] for c in subset])29 print(f"{name} kappa with any one verdict changed: {low:.2f} to {high:.2f}")30for reference, verdict, label in (("incorrect", "correct", "wrong answers the judge called correct"),31 ("correct", "incorrect", "right answers the judge called incorrect")):32 ids = [c.case_id for c in cases if c.reference == reference and verdicts[c.case_id] == verdict]33 print(f"{label}: {len(ids)} of {sum(c.reference == reference for c in cases)}")34 print(f"{label}, cases: {', '.join(ids)}")35# Read the reasons: how many rejections of right answers say a year's filter keeps only its first day?36first_day = [c.case_id for c in cases if c.reference == "correct" and judged[c.case_id].label == "incorrect"37 and any(w in judged[c.case_id].reason for w in ("January 1", "first day"))]38print(f"right answers rejected for reading a year's filter as January 1: {len(first_day)}")
python judge-agreement.pycases: 56
reference correct: 26
judge correct: 20
judge unparsed: 0
all cases: 56
all cases the reference labels correct: 26
all observed agreement: 0.75
all Cohen's kappa: 0.49
all Scott's pi: 0.48
all kappa with any one verdict changed: 0.45 to 0.53
schema-only cases: 23
schema-only cases the reference labels correct: 5
schema-only observed agreement: 0.74
schema-only Cohen's kappa: 0.10
schema-only Scott's pi: 0.09
schema-only kappa with any one verdict changed: -0.14 to 0.31
metric agents cases: 33
metric agents cases the reference labels correct: 21
metric agents observed agreement: 0.76
metric agents Cohen's kappa: 0.51
metric agents Scott's pi: 0.50
metric agents kappa with any one verdict changed: 0.44 to 0.58
wrong answers the judge called correct: 4 of 30
wrong answers the judge called correct, cases: gq-002 schema-only, gq-004 schema-only, gq-018 metrics-first, gq-035 metrics-first
right answers the judge called incorrect: 10 of 26
right answers the judge called incorrect, cases: gq-014 schema-only, gq-016 schema-only, gq-027 schema-only, gq-031 schema-only, gq-003 metrics-first, gq-006 metrics-first, gq-008 metrics-first, gq-015 metrics-first, gq-027 metrics-first, gq-029 metrics-first
right answers rejected for reading a year's filter as January 1: 6
The overall kappa, 0.49, looks moderate, but it hides a split by agent. On schema-only SQL's answers, most of them wrong, observed agreement is 0.74 and kappa only 0.10: the judge agreed mostly by calling wrong answers wrong, and rarely picked out the few right ones. Changing one of its verdicts on those answers moves kappa anywhere from -0.14 to 0.31.
The metric agents' kappa is higher, at 0.51, and changing one verdict moves it within 0.44 to 0.58. Read the first decimal of these kappas, not the second. Kappa and pi barely differ here, 0.49 against 0.48 overall, because the judge and the reference use each label about as often; when a judge leans hard one way, they part.
The direction of the errors matters more than the totals. The judge called 4 of 30 wrong answers correct, among them gq-018, chapter 6's near-miss refund metric. It called 10 of 26 right answers incorrect, and 6 of those rejections say that the SQL keeps only January 1 of the year, a misreading of the semantic layer's DATE_TRUNC('YEAR', …) = '2025-01-01'. The others reject schema-only answers that filter on placed_at rather than the local order day: the judge grades the method, and on those questions the number happened to come out the same. So the shop's judge leans strict, not lenient, as Thakur and colleagues report of the judges they studied (Thakur et al.), and it can't read the compiled SQL it's shown.
The judge is also the model that wrote every answer here. Self-preference, a judge that "scores its own outputs higher than others' while human annotators consider them of equal quality" (Panickssery et al.), is a known bias, and familiarity, text that has low perplexity for the judge, is one explanation of it (Wataoka et al.). With one model writing every answer, this measurement can't separate self-preference from other errors. It measures the net bias against a check that doesn't depend on the model; it doesn't remove the bias.
Working rule: Before you use a judge on text, measure it on cases whose answers a check can label, with labels anyone can recompute. Report observed agreement with Cohen's kappa and Scott's pi, split by the kinds of answer you'll judge, list the errors in each direction, and say how far one case moves kappa when the cases number a few dozen. Thresholds such as 0.8 are conventions, not results (Artstein and Poesio); at a kappa like the one measured here, use the judge to choose what a person reads, never to pass or fail a release on its own.
这部分尚未翻译,因此以英文显示。
Repeat runs to see the variance
Every score so far came from one run per question, and a run is a sample. The book's recordings sample each reply at Qwen's recommended temperature of 0.7, with one fixed seed reset before every call, so a replay gives the same reply every time, and a different seed can give a different one. A score from one seed is one draw from the scores the agent could get. Miller recommends treating an evaluation as an experiment and "reducing variance by resampling answers" several times per question (Miller). The alternative, lowering the temperature until runs agree, measures a different model from the one you serve.
The recording repeat-runs holds the same 12 dev questions run under four more seeds, 1 to 4, for both chapter 6's metrics-first agent and chapter 8's declining agent, with the same prompts, tools, sampling settings and four-call limit as their original recordings. With the original runs, made with seed 2718, each question has five runs per agent. SeededRecording in lib/regressions.py replays one agent's calls for one seed. The 12 questions are every dev answer or decline question that either agent missed in its original run, and the first five others in ID order, so they are harder than the split's average and their flip rates are an upper end. The recording also runs the declining agent once more with seed 2718 itself, to measure something the book has so far assumed: that a run with the same seed replays byte for byte on this runtime.
1"""Five runs of 12 dev questions per agent, one per seed: how much a score moves, and which questions flip.2The flipping, across-seeds and always-or-never lines and the seed check are the printed lines the chapter's points rest on."""3from lib.golden import load_golden4from lib.model import RecordedModel5from lib.regressions import ORIGINAL_SEED, SeededRecording, replay6from lib.semantic import load7from lib.warehouse import connect89con, layer = connect(), load()10QUESTIONS = ("gq-001", "gq-002", "gq-003", "gq-004", "gq-005", "gq-007", "gq-008", "gq-012", "gq-016", "gq-017",11 "gq-018", "gq-020") # the 12 questions repeat-runs holds12questions = [q for q in load_golden() if q.question_id in QUESTIONS]13seeds = (ORIGINAL_SEED, 1, 2, 3, 4) # 2718 replays the original recordings14for agent in ("metrics-first", "declining"):15 per_question = {q.question_id: 0 for q in questions}16 scores = []17 for seed in seeds:18 runs = replay(con, layer, agent, questions, seed=seed)19 scores.append(sum(r.outcome == "correct" for r in runs))20 for r in runs:21 per_question[r.question.question_id] += r.outcome == "correct"22 print(f"{agent} seed {seed}: {scores[-1]} of {len(questions)} correct")23 spread = f"{min(scores)} to {max(scores)}" if min(scores) < max(scores) else f"{min(scores)} at every seed"24 print(f"{agent} correct across seeds: {spread}")25 # a question flips when some seeds get it right and others wrong26 flips = [f"{q} ({n} of 5)" for q, n in per_question.items() if 0 < n < 5]27 print(f"{agent} questions that flip: {len(flips)}")28 print(f"{agent} flipping: {', '.join(flips)}")29 print(f"{agent} always correct: {sum(n == 5 for n in per_question.values())}")30 print(f"{agent} never correct: {sum(n == 0 for n in per_question.values())}")3132# Does a run replay byte for byte? The declining agent ran again with seed 2718; compare it with decline-runs.33again, original = SeededRecording("declining", ORIGINAL_SEED).calls, RecordedModel("decline-runs").calls34same = sum(key in original and call["reply"] == original[key]["reply"] for key, call in again.items())35print(f"seed {ORIGINAL_SEED} again: {same} of {len(again)} calls identical to decline-runs")
python seed-variance.pymetrics-first seed 2718: 6 of 12 correct
metrics-first seed 1: 6 of 12 correct
metrics-first seed 2: 6 of 12 correct
metrics-first seed 3: 6 of 12 correct
metrics-first seed 4: 5 of 12 correct
metrics-first correct across seeds: 5 to 6
metrics-first questions that flip: 1
metrics-first flipping: gq-008 (4 of 5)
metrics-first always correct: 5
metrics-first never correct: 6
declining seed 2718: 6 of 12 correct
declining seed 1: 6 of 12 correct
declining seed 2: 6 of 12 correct
declining seed 3: 6 of 12 correct
declining seed 4: 6 of 12 correct
declining correct across seeds: 6 at every seed
declining questions that flip: 0
declining flipping:
declining always correct: 6
declining never correct: 6
seed 2718 again: 24 of 24 calls identical to decline-runs
The replay check comes first, because everything else rests on it: seed 2718 again: 24 of 24 calls identical to decline-runs. On this runtime, llama.cpp on the CPU with one request at a time and the seed reset before each call, the second run with seed 2718 matched the first on every call. That is one agent on 12 questions on one machine, not a guarantee for every runtime, but it shows that the differences between seeds here come from sampling, not from the runtime. That matches the cause one lab found for variation at temperature 0 on served models, results that depend on how requests are batched (He et al.), which a single local request avoids; a served API gives no such promise.
Per question, the runs are steadier than one might fear at a temperature of 0.7: only 1 question flips, gq-008 (4 of 5), and only for the metrics-first agent. Across the five seeds the metrics-first agent's total ranged over 5 to 6, and the declining agent's was 6 at every seed.
The declining agent's outcome never changed with the seed: 6 questions were always right and 6 never. Chapter 8 wondered whether gq-008, total revenue in 2025, was chance, and the repeats answer it. Under the declining prompt it was wrong in every run, so its failure isn't chance: the new prompt moved the model off net revenue on that question, to the gross revenue chapter 8 saw in its one run.
We think the steadiness comes from the shop's design. Most of the agent's choices are among a few metric names, and the semantic layer turns each choice into the same SQL every time, so small differences in wording don't reach the result. A model that writes SQL has many more ways to differ. Studies of hosted models found larger swings, "accuracy variations up to 15% across naturally occurring runs" at settings meant to be deterministic (Atil et al.), on other tasks and APIs. Five runs of 12 questions is a small sample of the shop, too.
Working rule: Run each golden question several times at the sampling settings you serve, with seeds pinned in the recording's manifest, and confirm once that a seed replays exactly. Report per-question flips, not only the spread of the total, because a total can stay put while questions trade places. Never lower the temperature to make runs agree, unless that is the setting you will serve.
这部分尚未翻译,因此以英文显示。
Gate a change in CI
A CI job that scores every change on the golden questions turns the measurements so far into a decision: merge or don't. The scorecard showed why it can't rest on totals: the declining prompt left dev accuracy unchanged. Miller recommends "conducting statistical inference on the question-level paired differences" (Miller): score both configurations on the same questions and test the mean of the per-question differences. Pairing removes the variation between questions, which is far larger than a change's.
Comparison in lib/regressions.py takes, for each question, the share of runs correct under the base and under the candidate, and their difference. It reports the mean paired difference with a t interval, the standard error times Student's t for one fewer degrees of freedom than questions, counts the questions fixed and broken, and runs an exact sign test on the questions that changed. Miller's errors assume many questions; on 12, mostly unchanged, the t interval is a rough guide, and only the sign test assumes nothing. gate() applies two rules of our choosing: it fails on a stable break, a question the base got right in at least four runs of five and the candidate in at most one, and when the whole interval of the mean difference lies below zero. The first catches one regression an average would hide; the second, many small ones. We gate chapter 8's prompt against chapter 6's, with one run per question and then with five, on the 12 questions of repeat-runs, which over-represent misses, so the gate's numbers describe them, not the dev split.
1"""Gate a prompt change on paired, per-question differences over five recorded runs of each question.2The one-run sign test, the five-run lines and the gate's reasons are the printed lines the chapter's points rest on."""3from lib.golden import load_golden4from lib.regressions import ORIGINAL_SEED, Comparison, gate, replay, smallest_detectable, t_quantile5from lib.semantic import load6from lib.warehouse import connect78con, layer = connect(), load()9QUESTIONS = ("gq-001", "gq-002", "gq-003", "gq-004", "gq-005", "gq-007", "gq-008", "gq-012", "gq-016", "gq-017",10 "gq-018", "gq-020") # the 12 questions repeat-runs holds11questions = [q for q in load_golden() if q.question_id in QUESTIONS]12seeds = (ORIGINAL_SEED, 1, 2, 3, 4)13correct = {agent: {q.question_id: [] for q in questions} for agent in ("metrics-first", "declining")}14for agent, runs in correct.items():15 for seed in seeds:16 for run in replay(con, layer, agent, questions, seed=seed):17 runs[run.question.question_id].append(run.outcome == "correct")1819# One run each, as a CI job that runs every question once would see it.20once = Comparison({q: r[:1] for q, r in correct["metrics-first"].items()},21 {q: r[:1] for q, r in correct["declining"].items()})22fixed, broken = once.moved()23print(f"one run: fixed {len(fixed)} ({', '.join(fixed)}), broken {len(broken)} ({', '.join(broken)})")24print(f"one run sign test p: {once.sign_test():.2f}")25print(f"one run stable breaks: {', '.join(once.stable_breaks()) or 'none'}") # with one run, every flip is "stable"2627five = Comparison(correct["metrics-first"], correct["declining"])28for q in five.questions:29 print(f"{q}: metrics-first {sum(five.base[q])} of 5, declining {sum(five.candidate[q])} of 5")30fixed, broken = five.moved()31print(f"five runs: fixed {len(fixed)} ({', '.join(fixed) or 'none'}), broken {len(broken)} ({', '.join(broken) or 'none'})")32low, high = five.interval()33print(f"five runs mean paired difference: {five.mean_difference:+.2f}")34print(f"five runs 95% interval: {low:+.2f} to {high:+.2f}")35# one degree of freedom goes to estimating the spread of the differences from the same questions36print(f"t multiplier, {len(five.questions) - 1} degrees of freedom: {t_quantile(0.975, len(five.questions) - 1):.2f}")37print(f"five runs interval half-width: {five.half_width():.2f}")38k = smallest_detectable(len(five.questions))39print(f"fewest clean flips the mean rule detects: {k} of {len(five.questions)}")40print(f"smallest mean change the mean rule detects: {k / len(five.questions):.2f}")41print(f"five runs sign test p: {five.sign_test():.2f}")42passed, reasons = gate(five)43print(f"gate: {'pass' if passed else 'fail'}")44print("\n".join(f"gate reason: {r}" for r in reasons))
python regression-gate.pyone run: fixed 1 (gq-020), broken 1 (gq-008)
one run sign test p: 1.00
one run stable breaks: gq-008
gq-001: metrics-first 5 of 5, declining 5 of 5
gq-002: metrics-first 5 of 5, declining 5 of 5
gq-003: metrics-first 5 of 5, declining 5 of 5
gq-004: metrics-first 5 of 5, declining 5 of 5
gq-005: metrics-first 5 of 5, declining 5 of 5
gq-007: metrics-first 0 of 5, declining 0 of 5
gq-008: metrics-first 4 of 5, declining 0 of 5
gq-012: metrics-first 0 of 5, declining 0 of 5
gq-016: metrics-first 0 of 5, declining 0 of 5
gq-017: metrics-first 0 of 5, declining 0 of 5
gq-018: metrics-first 0 of 5, declining 0 of 5
gq-020: metrics-first 0 of 5, declining 5 of 5
five runs: fixed 1 (gq-020), broken 1 (gq-008)
five runs mean paired difference: +0.02
five runs 95% interval: -0.23 to +0.26
t multiplier, 11 degrees of freedom: 2.20
five runs interval half-width: 0.25
fewest clean flips the mean rule detects: 4 of 12
smallest mean change the mean rule detects: 0.33
five runs sign test p: 1.00
gate: fail
gate reason: stable break: gq-008
With one run each, the change fixed gq-020 and broke gq-008, and the sign test gives p = 1.00. With one run every flip counts as stable, so a single run can't tell a break from a chance flip. Five runs separate the two: gq-008 is a stable break (gq-008: metrics-first 4 of 5, declining 0 of 5), and the fix to gq-020 is as steady (gq-020: metrics-first 0 of 5, declining 5 of 5).
The mean paired difference is +0.02, with a t multiplier of 2.20 and an interval of -0.23 to +0.26, so on average the change did nothing measurable.
The gate says fail, for the reason stable break: gq-008. The prompt fixed one question and reliably broke another, and someone has to choose, or fix it, before it merges.
The gate also has a blind spot. The interval's half-width, 0.25 here, belongs to this comparison, because it depends on how much the differences vary. What the mean rule can detect follows from the rule: the fewest questions that must turn from always wrong to always right, the rest unchanged, for the interval to exclude zero is 4 of 12, a mean change of 0.33. Breaks mirror it. Anything smaller passes the mean rule either way, and only the stable-break rule and a person reading the moved questions stand in its way. A changed prompt fails the gate with MissingRecording until it is recorded, which is the moment to record it five times, not once.
Working rule: Gate a change to prompts, context or metrics on paired, per-question differences over repeated runs, never on a difference of totals. Fail on any stable break and on a mean difference whose interval lies below zero, write both rules down as your choice, and compute the smallest change they can detect on your set. When the gate fails, read the questions that moved before you touch the rules.
这部分尚未翻译,因此以英文显示。
Catch a schema change before users do
The warehouse changes under the agent too: a column renamed upstream, a table rebuilt, a type narrowed. Three checks catch a schema change without calling a model: every golden question's SQL still runs, every metric in the semantic layer still compiles and runs, and every recorded agent run still replays. The last works because RecordedModel looks replies up by the hash of the exact messages: when a tool returns an error the model never saw, the next prompt has no recording, and the replay fails loudly with MissingRecording instead of passing.
This run applies one migration an upstream team might ship, renaming analytics.fct_orders.net_revenue_eur to net_revenue_ex_tax_eur, to a new copy of the shop's warehouse, built by migrated() in lib/regressions.py. schema_checks() runs the three checks. Then the code makes the fix, one line in the metric's expr, in a copy of the semantic layer's YAML, and runs the checks again.
1"""An upstream rename of one column, caught by three checks that need no model, and the one-line fix.2The golden, metric and replay lines before the fix are the printed lines the chapter's points rest on."""3import shutil4import tempfile5from pathlib import Path67from lib.golden import load_golden8from lib.regressions import migrated, schema_checks9from lib.semantic import SEMANTIC_DIR, load1011RENAME = "ALTER TABLE analytics.fct_orders RENAME COLUMN net_revenue_eur TO net_revenue_ex_tax_eur"12con = migrated(RENAME)13golden = load_golden()14dev = [q for q in golden if q.split == "dev"]1516failed = schema_checks(con, load(), golden)17print(f"golden SQL that fails: {len(failed['golden SQL'])} of {sum(bool(q.golden_sql) for q in golden)}")18print(f"golden questions affected: {', '.join(failed['golden SQL'])}")19print(f"metrics that fail: {', '.join(failed['metrics'])}")20# forbidden questions are decided below the model, so they have no recording to replay21replays = schema_checks(con, load(), dev)["replays"]22print(f"recorded dev runs that no longer replay: {len(replays)} of {sum(q.kind != 'forbidden' for q in dev)}")23print(f"runs that stop: {', '.join(replays)}")2425# The fix: the metric's expr names the new column. A copy of the YAML stands in for the commit that changes it.26fixed = Path(tempfile.mkdtemp()) / "semantic"27shutil.copytree(SEMANTIC_DIR, fixed)28metrics = fixed / "metrics.yml"29metrics.write_text(metrics.read_text().replace("expr: net_revenue_eur", "expr: net_revenue_ex_tax_eur"))30after = schema_checks(con, load(fixed), dev)31print(f"after the fix, metrics that fail: {', '.join(after['metrics']) or 'none'}")32print(f"after the fix, recorded dev runs that no longer replay: {len(after['replays'])}")33shutil.rmtree(fixed.parent)
python renamed-column-caught.pygolden SQL that fails: 16 of 34
golden questions affected: gq-001, gq-002, gq-003, gq-006, gq-007, gq-008, gq-009, gq-010, gq-011, gq-012, gq-017, gq-025, gq-026, gq-034, gq-038, gq-040
metrics that fail: net_revenue, average_order_value
recorded dev runs that no longer replay: 8 of 21
runs that stop: gq-001, gq-002, gq-003, gq-006, gq-009, gq-010, gq-011, gq-017
after the fix, metrics that fail: none
after the fix, recorded dev runs that no longer replay: 0
Each check catches the rename. 16 of 34 golden queries fail to run. The metrics net_revenue, average_order_value fail, because the semantic layer's YAML names the old column. And 8 of 21 of the declining agent's recorded dev runs stop with MissingRecording: gq-001, gq-002, gq-003, gq-006, gq-009, gq-010, gq-011, gq-017. gq-008 and gq-012 still replay, because the agent answered them with gross revenue, which the rename doesn't touch. A replay that passes says that nothing the run touched has changed, not that the run is right.
After the fix, no metric fails and every recorded run replays: after the fix, recorded dev runs that no longer replay: 0. The agent never sees a column's name, only metrics, so its prompts are the same and the recordings stay valid. The golden SQL names the column directly, so the failing golden queries need the same rename, which is the cost of golden SQL written against tables: it is a second copy of the definitions, and it has to change with them. None of the checks needs the model, so CI can run them on every change to the warehouse's models.
Working rule: Run the golden SQL, every metric and every recorded replay in CI on changes to the warehouse as well as to the agent, and treat MissingRecording as a failure, never as a reason to skip a question. When a check fails after a schema change, fix the definition in the semantic layer before you record anything again.
生产环境中会出什么问题
这家商店的各项度量,建立在一个小模型的录制结果和 48 个问题之上。一旦用真实的智能体取代录制结果,评判器、关卡和各项得分都会出现录制结果从未显示过的变化,而每一种变化都有可以观察的迹象。
评判器的模型一变,评判器就变了。 新的模型版本、新的评判器提示词或新的评分细则,都是一个新的评判器,它与参考标签的一致程度必须重新度量。把校准用例及其标签与评判器提示词一起放在代码仓库中,两者任何一个变动时,都要重新运行一致性度量。智能体没有任何改动而 kappa 发生变化,就说明是评判器在变。
评判器读不懂它所评判的内容。 商店的评判器读错了语义层编译出的年份过滤条件,因此否决了正确的答案。如果给评判器看它无法解析的 SQL,它评的是 SQL 的外观,而不是其含义。应当给评判器它能够核对的内容,例如指标的名称和过滤条件,而不是编译后的 SQL;并且在看评判器的得分之前,先阅读它给出否决的理由。
实时运行的差异是回放所没有的。 回放的运行每次都相同,因此基于录制结果的 CI 是确定性的。实时模型则不然,即便在托管 API 上将温度设为 0 也是如此:一项研究观察到“自然出现的多次运行之间,准确率差异最高可达 15%”(Atil et al.),另有一个实验室将其中很大一部分追溯到某个服务器,其结果取决于有多少个请求共用同一批次(He et al.)。录制时要记录多次重复运行;在重复运行证明并非如此之前,把与录制得分不同的实时得分视为方差。
降低温度只是掩盖方差,而不是消除方差。 把温度设为 0 以使各次运行一致,这很有诱惑力。Miller 建议不要这样做,“除非目的是研究模型在新温度下的表现”(Miller):此时所度量的模型已不是你所提供服务的模型。应当在部署时的设置下度量,并进行重采样。
总是失败的关卡会被关掉。 只有十几个问题时,严格的规则会因噪声而触发,人们便学会绕过它合并代码。应当与对这些数字负责的人一起设定关卡的规则,写明关卡能检测什么、不能检测什么,并每月复查它的误报。
重新录制会变成调优。 每次修改提示词都需要新的录制结果,而在调优时一次又一次重新录制开发集,会使开发集的结果偏乐观,正如第 6 章所记录的。应在每份清单中说明某个数据划分被录制了多少次,并在调优结束后只录制一次测试集。
工作准则: 在关卡守护生产环境的智能体之前,把其评判器的校准用例与评判器提示词放在一起,并在两者任一变动时重新度量;向评判器展示它能够核对的内容;在部署时的设置下记录多次重复运行;与对这些数字负责的人商定关卡的规则;并在每份清单中记录某个数据划分被录制了多少次。
本章做出的决定
- 一次运行以记分卡的形式报告, 而不是只报告准确率:包括带 Wilson 区间的正确数、错误的数值、失败数、第 8 章的拒答指标,以及每个问题的模型调用次数和提示词 token 数,并且对每个智能体都使用同一批问题。
- 测试集只评分一次,用于得出最终数字。 第 8 章的拒答智能体在测试集上只记录了一次,记录于
decline-runs-test,其提示词保持不变,与第 3 章和第 6 章记录的测试运行并列。 - 评审器只评定
results_match()无法评定的内容: 即拒答的理由,采用两两比较的方式,每一对都按两种顺序各比较一次。数值始终与黄金结果比较。 - 评审器使用本书自己的模型和自己的提示词, 每一次裁决都记录在
judge-verdicts中。在shown()被修正为对无序的行进行排序之后,整份记录从头重新生成。 - 评审器的参考标签来自
results_match(), 而不是来自某个人的评分,因此任何人都可以重新计算这些标签。它们衡量的是数值是否正确,而不是方法是否正确。 - 一致性以观测一致率、Cohen 的 kappa 和 Scott 的 pi 报告, 既给出总体结果,也按智能体分别给出,并列出每个方向上的错误。每个 kappa 都建立在几十个案例之上,所以该示例展示了一次裁决会使它变动多少。评审器在 kappa 处于此处所测水平时,决定的是人要阅读什么,而绝不能单独放行一次发布。
- 方差来自每个问题的五次运行, 即原始记录再加四个种子,采用每次记录所用的采样设置,运行于
repeat-runs的 12 个开发集问题上,这些问题经过挑选,涵盖了每一次未命中。 - 门禁按问题逐个、成对地比较各配置, 并基于重复运行:出现稳定的失败,或者平均成对差值的 t 区间位于零以下时,门禁即判定失败。这两条规则都是本书的选择,在 12 个问题上,均值规则检测不到小于 4 of 12 个问题彻底改变的差异。
- CI 还会检查数据仓库: 运行黄金 SQL,每个指标都能编译并运行,每一次记录的运行都能重放,并且将
MissingRecording视为失败。
本章,在本版次中
版次 2026.11.15示例于 2026年11月19日 通过本章 6个示例 个示例中有 6 个通过
译自英文版图书,版次 2026.11.15。