評判答案並找出退化
閱讀約需 91 分鐘 分鐘,8 節 中共有 6 個範例。
讀完本章後,你將能夠:同時以多項指標為代理程式的一次執行評分;只在結果無法直接比較時,才以模型擔任評審;用任何人都能重新計算的參考標籤來校準該評審;觀察分數在不同次執行之間的變動幅度;並在 CI 中依據已錄製的執行結果,為提示詞、情境或指標的變更設下關卡。
幾乎每一章都改動過商店的代理程式。第 4 章在提示詞中加入了情境,第 6 章則讓代理程式改用語意層的指標。第 8 章接著改寫提示詞,使代理程式在拒絕回答時會說明理由。每次改動都是在黃金問題(golden questions)的開發切分上評分,也就是商店那 48 個附有已驗證答案的問題,而且每題只執行一次,這樣的做法能看出大幅變動,卻看不出小幅變動。第 8 章的新提示詞修正了一個開發問題,卻弄壞了另一個,也就是 gq-008(2025 年的總營收),而第 8 章只能說,像這樣的單一翻轉「可能只是偶然」。
本章要建立的測量方法,就是用來解決這類疑問。它除了準確率之外,還會從更多面向回報一次執行的結果,並附上區間,說明這個數字有多少可信度。它加入了評審,也就是一次用來評分的模型呼叫,負責評判黃金比對無法處理的部分,並且在信任評審之前先測量評審本身。它將每個問題執行五次以觀察變異,最後把這一切化為一道關卡,讓 CI 工作能在每次變更時,根據錄製的結果離線執行。
此部分尚未翻譯,因此以英文呈現。
Score a run on more than accuracy
A single accuracy figure answers one question, how often the agent was right, and hides three others that matter as much: what it did when it was wrong, what each answer cost, and how far the figure would move on another sample of questions. An agent that gives a wrong number and one that declines both lose a point of accuracy, but only one of them puts a wrong number in a deck. So a scorecard reports wrong numbers, declines and failures apart, the decline measures from chapter 8, and the cost in model calls and prompt tokens, beside accuracy.
The figure also needs an interval. On a set the size of a golden split, each question is several points, and the interval that most people learn first, the Wald interval, behaves badly for small sets. Brown, Cai and DasGupta recommend "the Wilson interval or the equal-tailed Jeffreys prior interval for small n" (Brown et al.). In words, Wilson's interval inverts the test of a proportion: it holds every true accuracy that the observed one wouldn't reject at the chosen level, so it never runs below 0 or above 1. For k correct of n, with p = k / n and z = 1.96 for about 95%:
centre = (p + z² / 2n) / (1 + z² / n)
half = z × √( p(1 − p) / n + z² / 4n² ) / (1 + z² / n)
interval = centre − half to centre + half
wilson() in lib/regressions.py computes it in a few lines. statsmodels implements the same interval as proportion_confint(method="wilson") (statsmodels); the book's pins don't include statsmodels, so the example doesn't call it. replay() runs one of the book's agents over golden questions from its recordings, and Scorecard scores the runs. The scorecard that follows scores three agents on the 21 answer and decline questions of each split: chapter 3's schema-only SQL, chapter 6's metrics-first agent and chapter 8's declining agent. Forbidden questions are left out, because chapter 9 decides them below the model.
The test split is held out, and the book scores it only for a number it reports. Chapters 3 and 6 recorded their agents on test once. This chapter recorded the declining agent on test once, in the recording decline-runs-test, with chapter 8's prompt unchanged, and changed nothing after it. These are the book's final test numbers for its agents.
1"""Score the book's agents on more than accuracy, on dev to compare them and on the held-out test split once.2The chapter's points rest on the dev and test lines for correct answers, wrong numbers and decline recall,3and on the fixed and broken lines for the declining prompt."""4from lib.golden import load_golden5from lib.regressions import Scorecard, replay6from lib.semantic import load7from lib.warehouse import connect89con, layer = connect(), load()10# Answer and decline questions; forbidden questions are decided below the model, by chapter 9's policies.11questions = [q for q in load_golden() if q.kind in ("answer", "decline")]12for split in ("dev", "test"):13 in_split = [q for q in questions if q.split == split]14 print(f"{split} questions: {len(in_split)}")15 runs = {agent: replay(con, layer, agent, in_split) for agent in ("schema-only", "metrics-first", "declining")}16 for agent, agent_runs in runs.items():17 print("\n".join(Scorecard(agent_runs).lines(f"{split} {agent}")))18 for label, was, now in (("fixed", False, True), ("broken", True, False)):19 # both agents' runs follow the same question order, so zip pairs one question's two outcomes20 ids = [a.question.question_id for a, b in zip(runs["metrics-first"], runs["declining"])21 if (a.outcome == "correct") == was and (b.outcome == "correct") == now]22 print(f"{split} {label} by the declining prompt: {', '.join(ids) or 'none'}")
python scorecard.pydev questions: 21
dev schema-only correct: 2 of 21
dev schema-only accuracy: 0.10 (95% interval 0.03 to 0.29)
dev schema-only wrong numbers: 11
dev schema-only errors or step limits: 8
dev schema-only decline recall: 0.00
dev schema-only decline precision: no declines
dev schema-only coverage: 0.62
dev schema-only selective risk: 0.85
dev schema-only model calls per question: 1.0
dev schema-only prompt tokens per question: 827
dev metrics-first correct: 15 of 21
dev metrics-first accuracy: 0.71 (95% interval 0.50 to 0.86)
dev metrics-first wrong numbers: 5
dev metrics-first errors or step limits: 1
dev metrics-first decline recall: 0.75
dev metrics-first decline precision: 1.00
dev metrics-first coverage: 0.81
dev metrics-first selective risk: 0.29
dev metrics-first model calls per question: 1.8
dev metrics-first prompt tokens per question: 2,196
dev declining correct: 15 of 21
dev declining accuracy: 0.71 (95% interval 0.50 to 0.86)
dev declining wrong numbers: 4
dev declining errors or step limits: 1
dev declining decline recall: 1.00
dev declining decline precision: 0.80
dev declining coverage: 0.71
dev declining selective risk: 0.27
dev declining model calls per question: 1.9
dev declining prompt tokens per question: 2,602
dev fixed by the declining prompt: gq-020
dev broken by the declining prompt: gq-008
test questions: 21
test schema-only correct: 3 of 21
test schema-only accuracy: 0.14 (95% interval 0.05 to 0.35)
test schema-only wrong numbers: 10
test schema-only errors or step limits: 8
test schema-only decline recall: 0.00
test schema-only decline precision: no declines
test schema-only coverage: 0.62
test schema-only selective risk: 0.77
test schema-only model calls per question: 1.0
test schema-only prompt tokens per question: 829
test metrics-first correct: 9 of 21
test metrics-first accuracy: 0.43 (95% interval 0.24 to 0.63)
test metrics-first wrong numbers: 5
test metrics-first errors or step limits: 1
test metrics-first decline recall: 1.00
test metrics-first decline precision: 0.40
test metrics-first coverage: 0.48
test metrics-first selective risk: 0.50
test metrics-first model calls per question: 1.5
test metrics-first prompt tokens per question: 1,854
test declining correct: 8 of 21
test declining accuracy: 0.38 (95% interval 0.21 to 0.59)
test declining wrong numbers: 4
test declining errors or step limits: 3
test declining decline recall: 0.75
test declining decline precision: 0.33
test declining coverage: 0.43
test declining selective risk: 0.44
test declining model calls per question: 1.9
test declining prompt tokens per question: 2,622
test fixed by the declining prompt: none
test broken by the declining prompt: gq-043
On dev, both agents get 15 of 21 right, an accuracy of 0.71 (95% interval 0.50 to 0.86), but not on the same questions. The declining prompt fixed gq-020 and broke gq-008, the two flips chapter 8 found.
The difference is in the misses. The declining agent gave 4 wrong numbers against metrics-first's 5, and its decline recall is 1.00 against 0.75.
It pays for that in coverage, 0.71 against 0.81, and in prompt tokens, 2,602 against 2,196 a question.
Schema-only SQL gets 2 of 21 right, and of the numbers it gives, the share that is wrong, its selective risk, is 0.85.
On test, the declining prompt fixed no question and broke one, gq-043, "What was revenue in Q2?", which chapter 6's agent met with a question back and the declining agent ran out of model calls on. The declining agent gets 8 of 21 against 9 of 21 for metrics-first, and schema-only SQL gets 3 of 21.
Its wrong numbers fell from 5 to 4, because one wrong answer became a run that hit the step limit, not a right one. The intervals of the two agents overlap almost entirely on both splits. Two totals with overlapping intervals can't tell you whether a change helped, which is why the gate compares the same questions in pairs.
Working rule: Report every run as a scorecard: correct with a Wilson interval, wrong numbers, declines and their recall and precision, failures by kind, and calls and tokens per question. Compare agents on the same questions in the same table. Score the test split once, when you report a final number, and say that you did; when two agents' intervals overlap, don't call either better from the totals.
此部分尚未翻譯,因此以英文呈現。
Judge only what can't be compared
results_match(), the comparison chapter 3 built, grades a number against a golden result. Much of what the agent says isn't a number. A decline's reason, the sentence that tells the person why no number came, has no golden result to match: two good reasons can share no words, and a fluent reason can be false. For text like that, the common tool is a judge, a model call that grades an answer against a rubric and returns a verdict. A judge is cheap and tireless, and it has known biases. Zheng and colleagues, who found that strong judges matched human preferences about as often as humans matched each other, also name "position, verbosity, and self-enhancement biases" (Zheng et al.): a judge can favour the answer shown first, the longer answer, and its own. A later study that catalogued twelve potential biases found that strong judges still carry significant bias on particular tasks (Ye et al.).
So use a judge only where a result can't be compared directly, and design each call so that a bias would show. The shop's judge is the book's own model, the same one the agent uses, called with a separate prompt, JUDGE_PAIR in lib/prompts.py. It sees a question and two reasons for declining it, labelled A and B, and names the better one, or a tie, as JSON. The rubric asks for a reason that says accurately why the shop's data can't answer and what the person could ask instead. The cases are every golden question, on either split, that both chapter 6's metrics-first agent and chapter 8's declining agent declined: 8 pairs. Only 3 of them, gq-021, gq-042 and gq-045, are decline questions. The other 5, gq-032, gq-033, gq-034, gq-037 and gq-041, are answer questions the warehouse can answer, the gaps in the layer chapter 6 found, and both agents declined them wrongly. On those, the right reply was a number, so the judge compares two wrong declines. Each pair is judged twice, once with each agent's reason first, so a verdict that follows the position rather than the text changes with the order. parse_verdict() in lib/judge.py reads each reply, and the recording judge-verdicts holds every call.
1"""Judge only what can't be compared: two reasons for declining the same question, shown in both orders.2The points about position, length and which reasons won rest on the same-verdict, consistent-winner and3won-in-both-orders lines, and the answerable lines show which reasons say the data lacks what was asked."""4import re56from lib.judge import judge_pairs, recorded_cases7from lib.model import RecordedModel8from lib.semantic import load9from lib.warehouse import connect1011con, layer = connect(), load()12_, pairs = recorded_cases(con, layer)13results = judge_pairs(RecordedModel("judge-verdicts"), pairs)14GROUPS = {"decline": "decline questions", "answer": "answerable questions both agents declined"}15AGENTS = ("metrics-first", "declining")16# a reason that says the data does not include or contain what was asked17BLAMES_DATA = re.compile(r"\bdata (does not|doesn't) (include|contain)")18# words a reply uses to offer a path forward or reframe the question19ALTERNATIVE = ("alternative", "instead", "reframe", "path forward")20for r in results:21 print(f"{r.case.question.question_id} ({r.case.question.kind}): metrics-first first -> {r.first.label}; "22 f"declining first -> {r.second.label}")23n = len(results)24print(f"pairs: {n}")25for group, name in GROUPS.items():26 print(f"pairs on {name}: {sum(r.case.question.kind == group for r in results)}")27# second_as_first maps the second order's label back to the first order's labels28print(f"same verdict in both orders: {sum(r.first.label == r.second_as_first for r in results)} of {n}")29print(f"first reason won in both orders: {sum(r.first.label == r.second.label == 'A' for r in results)} of {n}")30print(f"second reason won in both orders: {sum(r.first.label == r.second.label == 'B' for r in results)} of {n}")31for agent in AGENTS:32 print(f"{agent} reason won in both orders: {sum(r.winner == agent for r in results)}")33for group, name in GROUPS.items():34 for agent in AGENTS:35 print(f"{name}, {agent} reason won in both orders: "36 f"{sum(r.winner == agent and r.case.question.kind == group for r in results)}")37# longer_won compares the winning reason's length with the losing reason's, in characters38print(f"consistent winner was the longer reason: {sum(r.longer_won for r in results)}")39answerable = [r.case for r in results if r.case.question.kind == "answer"]40for agent, reasons in (("metrics-first", [c.first for c in answerable]), ("declining", [c.second for c in answerable])):41 said = sum(bool(BLAMES_DATA.search(x)) for x in reasons)42 print(f"answerable questions, {agent} reasons that say the data lacks it: {said} of {len(answerable)}")43replies = [v for r in results for v in (r.first, r.second)]44suggests = sum(any(w in v.reason.lower() for w in ALTERNATIVE) for v in replies)45print(f"judge replies that suggest an alternative or a reframe: {suggests} of {len(replies)}")
python judge-an-explanation.pygq-021 (decline): metrics-first first -> A; declining first -> A
gq-032 (answer): metrics-first first -> B; declining first -> A
gq-033 (answer): metrics-first first -> A; declining first -> B
gq-034 (answer): metrics-first first -> A; declining first -> B
gq-037 (answer): metrics-first first -> A; declining first -> B
gq-041 (answer): metrics-first first -> A; declining first -> B
gq-042 (decline): metrics-first first -> B; declining first -> A
gq-045 (decline): metrics-first first -> A; declining first -> B
pairs: 8
pairs on decline questions: 3
pairs on answerable questions both agents declined: 5
same verdict in both orders: 7 of 8
first reason won in both orders: 1 of 8
second reason won in both orders: 0 of 8
metrics-first reason won in both orders: 5
declining reason won in both orders: 2
decline questions, metrics-first reason won in both orders: 1
decline questions, declining reason won in both orders: 1
answerable questions both agents declined, metrics-first reason won in both orders: 4
answerable questions both agents declined, declining reason won in both orders: 1
consistent winner was the longer reason: 6
answerable questions, metrics-first reasons that say the data lacks it: 1 of 5
answerable questions, declining reasons that say the data lacks it: 5 of 5
judge replies that suggest an alternative or a reframe: 9 of 16
The judge gave the same verdict in both orders on 7 of 8 pairs, and the one that changed, gq-021, the cost of shipping, followed whichever reason came first. That is position bias in the one place a pair was close, since the two reasons say nearly the same thing.
Of the consistent verdicts, chapter 6's reason won 5 and chapter 8's 2, and the longer reason won 6 of them. That is what verbosity bias predicts, though 8 pairs can't separate it from longer reasons being better.
The two groups of pairs tell different stories. On the answerable questions, the judge preferred chapter 6's reason in 4 of the 5 pairs, and chapter 8's in 1.
Chapter 8's reasons say the data doesn't include what was asked in 5 of 5 answerable pairs, which is false: the warehouse holds raw.customers.email for gq-037 and fct_orders.discount_eur for gq-041. Chapter 6's reasons say so in 1 of 5; the others say the metrics don't include it, which is true of the semantic layer. So on these pairs the judge may have preferred the more accurate reason, chapter 6's, and accuracy and length point the same way.
On the decline questions, the case the rubric was written for, chapter 6's reason won 1 of the 3 pairs, chapter 8's 1, and the third changed with the order. That is too few pairs to say whose reasons are better.
Reading the verdicts shows a bigger problem than order. Of the judge's replies, 9 of 16 praise a reason for "offering a constructive path", an alternative, or a way to reframe the question, or fault one for lacking it, and neither agent's reason does that in any of these pairs. For gq-032, gross margin, the judge preferred chapter 8's reason for explaining the calculation, and the calculation that reason gives, gross revenue minus net revenue over gross revenue, is not a gross margin: the golden SQL subtracts the products' unit cost from net revenue. The judge rewarded a confident wrong definition. A verdict on text is only as good as the judge's grip on the facts behind it, which is why the next section measures the judge where the facts are known.
Working rule: Use a judge only for what can't be compared with a result, such as reasons, explanations and questions back, and keep results_match() for every number. Judge each pair in both orders and count the verdicts that change, record every verdict, and read a sample of the judge's reasons, not only its labels. Don't let a judge's preference between two texts decide a release until you have measured that judge on cases with known answers.
此部分尚未翻譯,因此以英文呈現。
Calibrate the judge against reference labels
A judge's verdicts on reasons can't be checked against a golden result, so measure the judge first on a task where they can. The shop's golden questions offer one. Ask the same model to grade the agent's answers to answer questions as correct or incorrect, without showing it the golden result, and compare its verdicts with labels that results_match() computes from the golden result. Reference labels computed by results_match() are nobody's opinion: anyone can recompute them from questions.toml and the recordings, and when one looks wrong, the fix is to the golden question, where it's open to correction like the rest of the book. Calibrating a judge against reference labels measures how often the judge agrees with a check you trust, and in which direction it errs.
Agreement needs a chance correction. If most answers are wrong, a judge that calls everything wrong agrees most of the time and has learned nothing; Artstein and Poesio describe the case where raw agreement is high and chance-corrected agreement can still be low (Artstein and Poesio). Cohen's kappa and Scott's pi both subtract the agreement that chance would give, and differ only in how they estimate it: Cohen's kappa estimates chance agreement from each rater's own label shares, and Scott's pi estimates chance agreement from the two raters' shares pooled. Thakur and colleagues dropped kappa for pi in a study of judges, because with kappa "the chance agreement adjusts for the observed average differences between raters, which is in fact part of what we intend to measure" (Thakur et al.). A judge that says "correct" more often than the reference is exactly what you want the statistic to catch, so we report both, and read the lower. In formulas, with p_o the observed agreement:
p_o = cases where the judge and the reference agree / cases
chance, κ = Σ over labels (reference's share of the label × judge's share of the label)
chance, π = Σ over labels (the label's share of both raters' labels pooled)²
kappa, pi = (p_o − chance) / (1 − chance)
agreement() in lib/judge.py computes all three; scikit-learn's cohen_kappa_score uses the same definition of kappa (scikit-learn). The prompt, JUDGE_ANSWER, gives the judge the shop's glossary, the question, the role, the metric and SQL behind the answer and its result. The cases are every distinct answer with a number that schema-only SQL, the metrics-first agent and the declining agent gave to an answer question, on both splits, from their recordings: 56 in all. The judge prompts were written once, and no verdict changed them. What the judge is shown changed once. Schema-only SQL's query for gq-009 has no ORDER BY, so its rows came back in another order on replay, and the prompt built from them no longer matched its recording. shown() in lib/judge.py now sorts the rows of any query without ORDER BY before the judge sees them. The recording made before that fix was set aside, and every judge call was recorded again from scratch with the fix in place, as the manifest of judge-verdicts says. Every reply matched the set-aside recording's, so no number in this chapter moved.
1"""Calibrate the judge: its verdicts on the book's recorded answers against labels from results_match().2The overall and per-agent agreement lines, and the one-verdict kappa ranges, are the figures the chapter's points rest on."""3from lib.judge import agreement, answer_messages, kappa_one_verdict, parse_verdict, recorded_cases4from lib.model import RecordedModel5from lib.semantic import load6from lib.warehouse import connect78con, layer = connect(), load()9judge = RecordedModel("judge-verdicts")10cases, _ = recorded_cases(con, layer) # every distinct answer with a number, with its reference label11judged = {c.case_id: parse_verdict(judge.complete(answer_messages(c.question, c.answer))) for c in cases}12verdicts = {case_id: v.label for case_id, v in judged.items()}13print(f"cases: {len(cases)}")14print(f"reference correct: {sum(c.reference == 'correct' for c in cases)}")15print(f"judge correct: {list(verdicts.values()).count('correct')}")16print(f"judge unparsed: {list(verdicts.values()).count('unparsed')}")17# a case ID ends with the agent that gave the answer, so the metric agents' cases are all the rest18subsets = {"all": cases, "schema-only": [c for c in cases if c.case_id.endswith("schema-only")],19 "metric agents": [c for c in cases if not c.case_id.endswith("schema-only")]}20for name, subset in subsets.items():21 a = agreement([c.reference for c in subset], [verdicts[c.case_id] for c in subset])22 print(f"{name} cases: {a.n}")23 print(f"{name} cases the reference labels correct: {sum(c.reference == 'correct' for c in subset)}")24 print(f"{name} observed agreement: {a.observed:.2f}")25 print(f"{name} Cohen's kappa: {a.kappa:.2f}")26 print(f"{name} Scott's pi: {a.pi:.2f}")27 # kappa with each single verdict flipped in turn, to show how far one case moves it28 low, high = kappa_one_verdict([c.reference for c in subset], [verdicts[c.case_id] for c in subset])29 print(f"{name} kappa with any one verdict changed: {low:.2f} to {high:.2f}")30for reference, verdict, label in (("incorrect", "correct", "wrong answers the judge called correct"),31 ("correct", "incorrect", "right answers the judge called incorrect")):32 ids = [c.case_id for c in cases if c.reference == reference and verdicts[c.case_id] == verdict]33 print(f"{label}: {len(ids)} of {sum(c.reference == reference for c in cases)}")34 print(f"{label}, cases: {', '.join(ids)}")35# Read the reasons: how many rejections of right answers say a year's filter keeps only its first day?36first_day = [c.case_id for c in cases if c.reference == "correct" and judged[c.case_id].label == "incorrect"37 and any(w in judged[c.case_id].reason for w in ("January 1", "first day"))]38print(f"right answers rejected for reading a year's filter as January 1: {len(first_day)}")
python judge-agreement.pycases: 56
reference correct: 26
judge correct: 20
judge unparsed: 0
all cases: 56
all cases the reference labels correct: 26
all observed agreement: 0.75
all Cohen's kappa: 0.49
all Scott's pi: 0.48
all kappa with any one verdict changed: 0.45 to 0.53
schema-only cases: 23
schema-only cases the reference labels correct: 5
schema-only observed agreement: 0.74
schema-only Cohen's kappa: 0.10
schema-only Scott's pi: 0.09
schema-only kappa with any one verdict changed: -0.14 to 0.31
metric agents cases: 33
metric agents cases the reference labels correct: 21
metric agents observed agreement: 0.76
metric agents Cohen's kappa: 0.51
metric agents Scott's pi: 0.50
metric agents kappa with any one verdict changed: 0.44 to 0.58
wrong answers the judge called correct: 4 of 30
wrong answers the judge called correct, cases: gq-002 schema-only, gq-004 schema-only, gq-018 metrics-first, gq-035 metrics-first
right answers the judge called incorrect: 10 of 26
right answers the judge called incorrect, cases: gq-014 schema-only, gq-016 schema-only, gq-027 schema-only, gq-031 schema-only, gq-003 metrics-first, gq-006 metrics-first, gq-008 metrics-first, gq-015 metrics-first, gq-027 metrics-first, gq-029 metrics-first
right answers rejected for reading a year's filter as January 1: 6
The overall kappa, 0.49, looks moderate, but it hides a split by agent. On schema-only SQL's answers, most of them wrong, observed agreement is 0.74 and kappa only 0.10: the judge agreed mostly by calling wrong answers wrong, and rarely picked out the few right ones. Changing one of its verdicts on those answers moves kappa anywhere from -0.14 to 0.31.
The metric agents' kappa is higher, at 0.51, and changing one verdict moves it within 0.44 to 0.58. Read the first decimal of these kappas, not the second. Kappa and pi barely differ here, 0.49 against 0.48 overall, because the judge and the reference use each label about as often; when a judge leans hard one way, they part.
The direction of the errors matters more than the totals. The judge called 4 of 30 wrong answers correct, among them gq-018, chapter 6's near-miss refund metric. It called 10 of 26 right answers incorrect, and 6 of those rejections say that the SQL keeps only January 1 of the year, a misreading of the semantic layer's DATE_TRUNC('YEAR', …) = '2025-01-01'. The others reject schema-only answers that filter on placed_at rather than the local order day: the judge grades the method, and on those questions the number happened to come out the same. So the shop's judge leans strict, not lenient, as Thakur and colleagues report of the judges they studied (Thakur et al.), and it can't read the compiled SQL it's shown.
The judge is also the model that wrote every answer here. Self-preference, a judge that "scores its own outputs higher than others' while human annotators consider them of equal quality" (Panickssery et al.), is a known bias, and familiarity, text that has low perplexity for the judge, is one explanation of it (Wataoka et al.). With one model writing every answer, this measurement can't separate self-preference from other errors. It measures the net bias against a check that doesn't depend on the model; it doesn't remove the bias.
Working rule: Before you use a judge on text, measure it on cases whose answers a check can label, with labels anyone can recompute. Report observed agreement with Cohen's kappa and Scott's pi, split by the kinds of answer you'll judge, list the errors in each direction, and say how far one case moves kappa when the cases number a few dozen. Thresholds such as 0.8 are conventions, not results (Artstein and Poesio); at a kappa like the one measured here, use the judge to choose what a person reads, never to pass or fail a release on its own.
此部分尚未翻譯,因此以英文呈現。
Repeat runs to see the variance
Every score so far came from one run per question, and a run is a sample. The book's recordings sample each reply at Qwen's recommended temperature of 0.7, with one fixed seed reset before every call, so a replay gives the same reply every time, and a different seed can give a different one. A score from one seed is one draw from the scores the agent could get. Miller recommends treating an evaluation as an experiment and "reducing variance by resampling answers" several times per question (Miller). The alternative, lowering the temperature until runs agree, measures a different model from the one you serve.
The recording repeat-runs holds the same 12 dev questions run under four more seeds, 1 to 4, for both chapter 6's metrics-first agent and chapter 8's declining agent, with the same prompts, tools, sampling settings and four-call limit as their original recordings. With the original runs, made with seed 2718, each question has five runs per agent. SeededRecording in lib/regressions.py replays one agent's calls for one seed. The 12 questions are every dev answer or decline question that either agent missed in its original run, and the first five others in ID order, so they are harder than the split's average and their flip rates are an upper end. The recording also runs the declining agent once more with seed 2718 itself, to measure something the book has so far assumed: that a run with the same seed replays byte for byte on this runtime.
1"""Five runs of 12 dev questions per agent, one per seed: how much a score moves, and which questions flip.2The flipping, across-seeds and always-or-never lines and the seed check are the printed lines the chapter's points rest on."""3from lib.golden import load_golden4from lib.model import RecordedModel5from lib.regressions import ORIGINAL_SEED, SeededRecording, replay6from lib.semantic import load7from lib.warehouse import connect89con, layer = connect(), load()10QUESTIONS = ("gq-001", "gq-002", "gq-003", "gq-004", "gq-005", "gq-007", "gq-008", "gq-012", "gq-016", "gq-017",11 "gq-018", "gq-020") # the 12 questions repeat-runs holds12questions = [q for q in load_golden() if q.question_id in QUESTIONS]13seeds = (ORIGINAL_SEED, 1, 2, 3, 4) # 2718 replays the original recordings14for agent in ("metrics-first", "declining"):15 per_question = {q.question_id: 0 for q in questions}16 scores = []17 for seed in seeds:18 runs = replay(con, layer, agent, questions, seed=seed)19 scores.append(sum(r.outcome == "correct" for r in runs))20 for r in runs:21 per_question[r.question.question_id] += r.outcome == "correct"22 print(f"{agent} seed {seed}: {scores[-1]} of {len(questions)} correct")23 spread = f"{min(scores)} to {max(scores)}" if min(scores) < max(scores) else f"{min(scores)} at every seed"24 print(f"{agent} correct across seeds: {spread}")25 # a question flips when some seeds get it right and others wrong26 flips = [f"{q} ({n} of 5)" for q, n in per_question.items() if 0 < n < 5]27 print(f"{agent} questions that flip: {len(flips)}")28 print(f"{agent} flipping: {', '.join(flips)}")29 print(f"{agent} always correct: {sum(n == 5 for n in per_question.values())}")30 print(f"{agent} never correct: {sum(n == 0 for n in per_question.values())}")3132# Does a run replay byte for byte? The declining agent ran again with seed 2718; compare it with decline-runs.33again, original = SeededRecording("declining", ORIGINAL_SEED).calls, RecordedModel("decline-runs").calls34same = sum(key in original and call["reply"] == original[key]["reply"] for key, call in again.items())35print(f"seed {ORIGINAL_SEED} again: {same} of {len(again)} calls identical to decline-runs")
python seed-variance.pymetrics-first seed 2718: 6 of 12 correct
metrics-first seed 1: 6 of 12 correct
metrics-first seed 2: 6 of 12 correct
metrics-first seed 3: 6 of 12 correct
metrics-first seed 4: 5 of 12 correct
metrics-first correct across seeds: 5 to 6
metrics-first questions that flip: 1
metrics-first flipping: gq-008 (4 of 5)
metrics-first always correct: 5
metrics-first never correct: 6
declining seed 2718: 6 of 12 correct
declining seed 1: 6 of 12 correct
declining seed 2: 6 of 12 correct
declining seed 3: 6 of 12 correct
declining seed 4: 6 of 12 correct
declining correct across seeds: 6 at every seed
declining questions that flip: 0
declining flipping:
declining always correct: 6
declining never correct: 6
seed 2718 again: 24 of 24 calls identical to decline-runs
The replay check comes first, because everything else rests on it: seed 2718 again: 24 of 24 calls identical to decline-runs. On this runtime, llama.cpp on the CPU with one request at a time and the seed reset before each call, the second run with seed 2718 matched the first on every call. That is one agent on 12 questions on one machine, not a guarantee for every runtime, but it shows that the differences between seeds here come from sampling, not from the runtime. That matches the cause one lab found for variation at temperature 0 on served models, results that depend on how requests are batched (He et al.), which a single local request avoids; a served API gives no such promise.
Per question, the runs are steadier than one might fear at a temperature of 0.7: only 1 question flips, gq-008 (4 of 5), and only for the metrics-first agent. Across the five seeds the metrics-first agent's total ranged over 5 to 6, and the declining agent's was 6 at every seed.
The declining agent's outcome never changed with the seed: 6 questions were always right and 6 never. Chapter 8 wondered whether gq-008, total revenue in 2025, was chance, and the repeats answer it. Under the declining prompt it was wrong in every run, so its failure isn't chance: the new prompt moved the model off net revenue on that question, to the gross revenue chapter 8 saw in its one run.
We think the steadiness comes from the shop's design. Most of the agent's choices are among a few metric names, and the semantic layer turns each choice into the same SQL every time, so small differences in wording don't reach the result. A model that writes SQL has many more ways to differ. Studies of hosted models found larger swings, "accuracy variations up to 15% across naturally occurring runs" at settings meant to be deterministic (Atil et al.), on other tasks and APIs. Five runs of 12 questions is a small sample of the shop, too.
Working rule: Run each golden question several times at the sampling settings you serve, with seeds pinned in the recording's manifest, and confirm once that a seed replays exactly. Report per-question flips, not only the spread of the total, because a total can stay put while questions trade places. Never lower the temperature to make runs agree, unless that is the setting you will serve.
在 CI 中把關變更
CI 工作會在每次變更時以黃金問題集評分,把前面的各項量測轉成一個決定:合併或不合併。計分卡說明了為什麼這個決定不能只憑總分:拒答型提示詞讓開發集的準確率毫無變化。Miller 建議「對問題層級的成對差異進行統計推論」(Miller):以同一批問題為兩種組態評分,再檢定每題差異的平均值。成對比較消除了問題之間的變異,而這種變異遠大於一次變更所造成的變異。
Comparison(位於 lib/regressions.py)會針對每一題,取出基準版本與候選版本各自答對的執行次數比例,以及兩者的差。它回報平均成對差異與 t 區間(標準誤乘以 Student t 值,自由度為問題數減一),計算被修正與被破壞的問題數,並對有變動的問題進行精確符號檢定。Miller 的誤差公式假設問題數很多;在 12 題且多數未變動的情況下,t 區間只是粗略的參考,只有符號檢定不作任何假設。gate() 套用我們自行選定的兩條規則:一是出現穩定破壞時判定失敗,也就是基準版本在五次執行中至少答對四次、候選版本至多答對一次的問題;二是平均差異的整個區間都低於零時判定失敗。前者能抓出被平均值掩蓋的單一退步,後者能抓出許多細微的退步。我們以第 8 章的提示詞對照第 6 章的提示詞進行把關,先每題執行一次,再每題執行五次,使用 repeat-runs 的 12 道問題;這些問題過度代表了答錯的情形,因此把關所得的數字描述的是這些問題,而不是開發集。
1"""Gate a prompt change on paired, per-question differences over five recorded runs of each question.2The one-run sign test, the five-run lines and the gate's reasons are the printed lines the chapter's points rest on."""3from lib.golden import load_golden4from lib.regressions import ORIGINAL_SEED, Comparison, gate, replay, smallest_detectable, t_quantile5from lib.semantic import load6from lib.warehouse import connect78con, layer = connect(), load()9QUESTIONS = ("gq-001", "gq-002", "gq-003", "gq-004", "gq-005", "gq-007", "gq-008", "gq-012", "gq-016", "gq-017",10 "gq-018", "gq-020") # the 12 questions repeat-runs holds11questions = [q for q in load_golden() if q.question_id in QUESTIONS]12seeds = (ORIGINAL_SEED, 1, 2, 3, 4)13correct = {agent: {q.question_id: [] for q in questions} for agent in ("metrics-first", "declining")}14for agent, runs in correct.items():15 for seed in seeds:16 for run in replay(con, layer, agent, questions, seed=seed):17 runs[run.question.question_id].append(run.outcome == "correct")1819# One run each, as a CI job that runs every question once would see it.20once = Comparison({q: r[:1] for q, r in correct["metrics-first"].items()},21 {q: r[:1] for q, r in correct["declining"].items()})22fixed, broken = once.moved()23print(f"one run: fixed {len(fixed)} ({', '.join(fixed)}), broken {len(broken)} ({', '.join(broken)})")24print(f"one run sign test p: {once.sign_test():.2f}")25print(f"one run stable breaks: {', '.join(once.stable_breaks()) or 'none'}") # with one run, every flip is "stable"2627five = Comparison(correct["metrics-first"], correct["declining"])28for q in five.questions:29 print(f"{q}: metrics-first {sum(five.base[q])} of 5, declining {sum(five.candidate[q])} of 5")30fixed, broken = five.moved()31print(f"five runs: fixed {len(fixed)} ({', '.join(fixed) or 'none'}), broken {len(broken)} ({', '.join(broken) or 'none'})")32low, high = five.interval()33print(f"five runs mean paired difference: {five.mean_difference:+.2f}")34print(f"five runs 95% interval: {low:+.2f} to {high:+.2f}")35# one degree of freedom goes to estimating the spread of the differences from the same questions36print(f"t multiplier, {len(five.questions) - 1} degrees of freedom: {t_quantile(0.975, len(five.questions) - 1):.2f}")37print(f"five runs interval half-width: {five.half_width():.2f}")38k = smallest_detectable(len(five.questions))39print(f"fewest clean flips the mean rule detects: {k} of {len(five.questions)}")40print(f"smallest mean change the mean rule detects: {k / len(five.questions):.2f}")41print(f"five runs sign test p: {five.sign_test():.2f}")42passed, reasons = gate(five)43print(f"gate: {'pass' if passed else 'fail'}")44print("\n".join(f"gate reason: {r}" for r in reasons))
python regression-gate.pyone run: fixed 1 (gq-020), broken 1 (gq-008)
one run sign test p: 1.00
one run stable breaks: gq-008
gq-001: metrics-first 5 of 5, declining 5 of 5
gq-002: metrics-first 5 of 5, declining 5 of 5
gq-003: metrics-first 5 of 5, declining 5 of 5
gq-004: metrics-first 5 of 5, declining 5 of 5
gq-005: metrics-first 5 of 5, declining 5 of 5
gq-007: metrics-first 0 of 5, declining 0 of 5
gq-008: metrics-first 4 of 5, declining 0 of 5
gq-012: metrics-first 0 of 5, declining 0 of 5
gq-016: metrics-first 0 of 5, declining 0 of 5
gq-017: metrics-first 0 of 5, declining 0 of 5
gq-018: metrics-first 0 of 5, declining 0 of 5
gq-020: metrics-first 0 of 5, declining 5 of 5
five runs: fixed 1 (gq-020), broken 1 (gq-008)
five runs mean paired difference: +0.02
five runs 95% interval: -0.23 to +0.26
t multiplier, 11 degrees of freedom: 2.20
five runs interval half-width: 0.25
fewest clean flips the mean rule detects: 4 of 12
smallest mean change the mean rule detects: 0.33
five runs sign test p: 1.00
gate: fail
gate reason: stable break: gq-008
每題只執行一次時,這項變更修正了 gq-020,破壞了 gq-008,符號檢定給出 p = 1.00。只執行一次時,每一次翻轉都被視為穩定,因此單次執行無法區分真正的破壞與偶然的翻轉。執行五次就能區分兩者:gq-008 是穩定破壞(gq-008: metrics-first 4 of 5, declining 0 of 5),而對 gq-020 的修正同樣穩定(gq-020: metrics-first 0 of 5, declining 5 of 5)。
平均成對差異為 +0.02,t 乘數為 2.20,區間為 -0.23 to +0.26,因此平均而言,這項變更沒有產生可測得的影響。
把關結果是 fail,原因為 stable break: gq-008。這個提示詞修正了一題,卻穩定地破壞了另一題,必須有人在合併前做出取捨,或加以修復。
這個把關也有盲點。區間的半寬度(此處為 0.25)屬於這次比較本身,因為它取決於差異的變動程度。平均規則能偵測到什麼,由規則本身決定:要使區間不含零,在其餘問題不變的情況下,最少必須有幾題從全錯變成全對,這個數目是 4 of 12,對應的平均變化為 0.33。破壞的情形與此對稱。比這更小的變化,無論是進步或退步,都會通過平均規則,能擋住它們的只有穩定破壞規則,以及一位檢視這些變動問題的人。變更過的提示詞在錄製之前,會以 MissingRecording 讓把關失敗,這正是該錄製五次而非一次的時機。
實用原則: 對提示詞、上下文或指標的變更,應依據重複執行下的成對、逐題差異來把關,絕不依據總分之差。遇到任何穩定破壞,以及區間低於零的平均差異,都判定失敗;把這兩條規則記錄為你自己的選擇,並計算它們在你的資料集上能偵測到的最小變化。把關失敗時,先閱讀有變動的問題,再去動規則。
此部分尚未翻譯,因此以英文呈現。
Catch a schema change before users do
The warehouse changes under the agent too: a column renamed upstream, a table rebuilt, a type narrowed. Three checks catch a schema change without calling a model: every golden question's SQL still runs, every metric in the semantic layer still compiles and runs, and every recorded agent run still replays. The last works because RecordedModel looks replies up by the hash of the exact messages: when a tool returns an error the model never saw, the next prompt has no recording, and the replay fails loudly with MissingRecording instead of passing.
This run applies one migration an upstream team might ship, renaming analytics.fct_orders.net_revenue_eur to net_revenue_ex_tax_eur, to a new copy of the shop's warehouse, built by migrated() in lib/regressions.py. schema_checks() runs the three checks. Then the code makes the fix, one line in the metric's expr, in a copy of the semantic layer's YAML, and runs the checks again.
1"""An upstream rename of one column, caught by three checks that need no model, and the one-line fix.2The golden, metric and replay lines before the fix are the printed lines the chapter's points rest on."""3import shutil4import tempfile5from pathlib import Path67from lib.golden import load_golden8from lib.regressions import migrated, schema_checks9from lib.semantic import SEMANTIC_DIR, load1011RENAME = "ALTER TABLE analytics.fct_orders RENAME COLUMN net_revenue_eur TO net_revenue_ex_tax_eur"12con = migrated(RENAME)13golden = load_golden()14dev = [q for q in golden if q.split == "dev"]1516failed = schema_checks(con, load(), golden)17print(f"golden SQL that fails: {len(failed['golden SQL'])} of {sum(bool(q.golden_sql) for q in golden)}")18print(f"golden questions affected: {', '.join(failed['golden SQL'])}")19print(f"metrics that fail: {', '.join(failed['metrics'])}")20# forbidden questions are decided below the model, so they have no recording to replay21replays = schema_checks(con, load(), dev)["replays"]22print(f"recorded dev runs that no longer replay: {len(replays)} of {sum(q.kind != 'forbidden' for q in dev)}")23print(f"runs that stop: {', '.join(replays)}")2425# The fix: the metric's expr names the new column. A copy of the YAML stands in for the commit that changes it.26fixed = Path(tempfile.mkdtemp()) / "semantic"27shutil.copytree(SEMANTIC_DIR, fixed)28metrics = fixed / "metrics.yml"29metrics.write_text(metrics.read_text().replace("expr: net_revenue_eur", "expr: net_revenue_ex_tax_eur"))30after = schema_checks(con, load(fixed), dev)31print(f"after the fix, metrics that fail: {', '.join(after['metrics']) or 'none'}")32print(f"after the fix, recorded dev runs that no longer replay: {len(after['replays'])}")33shutil.rmtree(fixed.parent)
python renamed-column-caught.pygolden SQL that fails: 16 of 34
golden questions affected: gq-001, gq-002, gq-003, gq-006, gq-007, gq-008, gq-009, gq-010, gq-011, gq-012, gq-017, gq-025, gq-026, gq-034, gq-038, gq-040
metrics that fail: net_revenue, average_order_value
recorded dev runs that no longer replay: 8 of 21
runs that stop: gq-001, gq-002, gq-003, gq-006, gq-009, gq-010, gq-011, gq-017
after the fix, metrics that fail: none
after the fix, recorded dev runs that no longer replay: 0
Each check catches the rename. 16 of 34 golden queries fail to run. The metrics net_revenue, average_order_value fail, because the semantic layer's YAML names the old column. And 8 of 21 of the declining agent's recorded dev runs stop with MissingRecording: gq-001, gq-002, gq-003, gq-006, gq-009, gq-010, gq-011, gq-017. gq-008 and gq-012 still replay, because the agent answered them with gross revenue, which the rename doesn't touch. A replay that passes says that nothing the run touched has changed, not that the run is right.
After the fix, no metric fails and every recorded run replays: after the fix, recorded dev runs that no longer replay: 0. The agent never sees a column's name, only metrics, so its prompts are the same and the recordings stay valid. The golden SQL names the column directly, so the failing golden queries need the same rename, which is the cost of golden SQL written against tables: it is a second copy of the definitions, and it has to change with them. None of the checks needs the model, so CI can run them on every change to the warehouse's models.
Working rule: Run the golden SQL, every metric and every recorded replay in CI on changes to the warehouse as well as to the agent, and treat MissingRecording as a failure, never as a reason to skip a question. When a check fails after a schema change, fix the definition in the semantic layer before you record anything again.
正式環境中會出什麼問題
這家商店的各項量測,建立在一個小型模型的錄製紀錄與 48 個問題之上。一旦以真實的代理取代錄製紀錄,評審、閘門與分數就會各自出現錄製紀錄從未顯示過的變動,而每一種變動都有可供觀察的徵兆。
評審會隨其模型而改變。 新的模型版本、新的評審提示或新的評分準則,都等於一個新的評審,其與參考標籤的一致性必須重新量測。請將校準案例及其標籤與評審提示一起放在儲存庫中,並在任一者變動時重新執行一致性量測。若 kappa 在代理沒有任何變動的情況下發生變化,就表示是評審在變動。
評審讀不懂它所評判的內容。 這家商店的評審誤讀了語意層編譯出的年份篩選條件,因而駁回了正確的答案。評審若看到自己無法解析的 SQL,評的是 SQL 的外觀,而不是它的語意。請提供評審能夠檢查的內容,例如指標的名稱與篩選條件,而不是編譯後的 SQL;並且在查看分數之前,先讀它駁回答案的理由。
即時執行的變異是重播所沒有的。 重播的執行每次都相同,因此以錄製紀錄進行的 CI 是確定性的。即時模型則不然,即使在提供服務的 API 上將溫度設為 0 也一樣:一項研究就觀察到「自然發生的多次執行之間,準確率變動最高可達 15%」(Atil et al.),而一個實驗室將其中大部分歸因於伺服器:其結果取決於有多少個 requests 共用同一個批次(He et al.)。錄製時請記錄重複執行的結果;在重複執行證明並非如此之前,請將即時分數與錄製分數之間的差異視為變異。
降低溫度只是隱藏變異,而不是消除變異。 為了讓各次執行一致而將溫度設為 0,是很誘人的做法。Miller 建議不要這樣做,除非「目的是研究模型在新溫度下的表現」(Miller):因為這時所量測的模型並不是實際提供服務的模型。請以實際提供服務的設定進行量測,並重複取樣。
總是失敗的閘門終究會被關掉。 在只有十幾個問題的情況下,嚴格的規則會因雜訊而觸發,人們便學會繞過它直接合併。請與為這些數字負責的人共同訂定閘門的規則,寫下閘門能偵測與無法偵測的事項,並每月檢討其誤報。
重新錄製會演變成調校。 每次修改提示都需要新的錄製紀錄,而在調校期間一再重新錄製開發分割,會使開發分割的結果偏於樂觀,如第 6 章所記錄的那樣。請在每份清單中說明某個分割被錄製了幾次,並在調校結束後,只錄製一次測試分割。
實用原則: 在閘門守護正式環境的代理之前,請將其評審的校準案例與評審提示放在一起,並在任一者變動時重新量測;向評審提供它能夠檢查的內容;以實際提供服務的設定錄製重複的執行;與為這些數字負責的人共同議定閘門的規則;並在每份清單中計入某個分割被錄製的次數。
本章做出的決定
- 一次執行以計分卡呈報, 而不是單一的準確率:包含附 Wilson 區間的答對數、錯誤的數字、失敗次數、第 8 章的拒答指標,以及每個問題的模型呼叫次數與提示詞 token 數,並且讓每個代理程式都在相同的問題上比較。
- 測試切分只計分一次,用於最終數字。 第 8 章的拒答代理程式在測試切分上只記錄一次,記錄於
decline-runs-test,其提示詞維持不變,並與第 3 章和第 6 章記錄的測試執行並列。 - 評審只評分
results_match()無法判斷的部分: 即拒答的理由,以成對方式比較,每一對都以兩種順序各比較一次。數字則一律與標準答案的結果比對。 - 評審使用本書自己的模型與自己的提示詞, 每一項判定都記錄在
judge-verdicts中。在shown()修正為對無序的資料列排序之後,整份紀錄已從頭重新製作。 - 評審的參考標籤來自
results_match(), 因此任何人都能重新計算,而不必依賴人工評分。這些標籤衡量的是數字是否正確,而不是方法是否正確。 - 一致性以觀察到的一致率、Cohen's kappa 與 Scott's pi 呈報, 並整體與依代理程式分別呈現,同時列出各方向的錯誤。每個 kappa 都只建立在幾十個案例上,因此範例會顯示單一判定能使它變動多少。在與此處所測得相近的 kappa 下,評審是用來決定人要閱讀哪些內容,而絕不能單獨決定是否發布。
- 變異來自每個問題五次執行, 即原始紀錄再加上另外四個種子,採用每次紀錄都使用的取樣設定,對象是
repeat-runs中的 12 個開發問題,並且特意納入每一個答錯的問題。 - 閘門以成對方式逐題比較各種設定, 並跨多次重複執行:遇到穩定的破壞,或平均成對差異的 t 區間落在零以下時,閘門即判定失敗。這兩條規則都是本書的選擇,而在 12 個問題上,平均規則偵測不到小於 4 of 12 個問題完全翻轉的變化。
- CI 也會檢查資料倉儲: 執行標準答案 SQL,每個指標都能編譯並執行,每一次已記錄的執行都能重播,而出現
MissingRecording則視為失敗。
本章,於本版次中
版次 2026.11.15範例於 2026年11 月 19 通過測試本章 6 個範例 個範例中有 6 個通過
譯自英文書,版次 2026.11.15。