답을 판정하고 회귀를 잡아내기

읽는 데 약 63분분 걸린다, 8절에 6개 예제이(가) 있음.

2026.11.15판 예제 통과 2026년 11월 19 78개 예제 중 78개 통과 변경 사항

AI로 번역함. 영어 원문을 사람이 검증함. 우리가 책을 만드는 방식

영어판 2026.11.15판을 번역한 것이다. 원문 읽기

이 장을 마치면 여러 척도로 에이전트의 한 번의 실행을 동시에 채점하고, 결과를 직접 비교할 수 없는 경우에만 모델을 판정자로 사용하며, 누구나 다시 계산할 수 있는 기준 레이블로 그 판정자를 보정하고, 실행마다 점수가 얼마나 움직이는지 확인하며, 기록된 실행을 바탕으로 프롬프트, 컨텍스트, 지표의 변경을 CI에서 통과시키거나 막을 수 있다.

거의 모든 장이 상점의 에이전트를 바꾸었다. 4장은 프롬프트에 컨텍스트를 추가했고, 6장은 에이전트가 시맨틱 레이어의 지표를 쓰도록 옮겼다. 이어서 8장은 이유를 밝히며 답변을 거절하도록 프롬프트를 고쳐 썼다. 각 변경은 검증된 답이 있는 상점의 48개 질문인 골든 질문의 dev 분할에서 질문당 한 번씩 실행해 채점했는데, 이 방식은 큰 변화는 보여 줄 수 있어도 작은 변화는 보여 주지 못한다. 8장의 새 프롬프트는 dev 질문 하나를 바로잡고 다른 하나, 곧 2025년 총매출에 해당하는 gq-008를 망가뜨렸으며, 8장은 이런 단일 뒤집힘이 "우연일 수 있다"고만 말할 수 있었다.

이 장은 그런 의문을 가려 줄 측정을 만든다. 정확도뿐 아니라 그 수치를 어디까지 믿을 수 있는지 알려 주는 구간과 함께 실행 결과를 보고한다. 골든 비교로는 채점할 수 없는 부분을 평가하는 모델 호출인 판정자를 추가하고, 신뢰하기 전에 판정자를 측정한다. 분산을 보기 위해 질문을 다섯 번씩 실행하고, 이 모든 것을 CI 작업이 변경마다 녹화본만으로 오프라인에서 실행할 수 있는 게이트로 만든다.

이 부분은 아직 번역되지 않았으므로 영어로 표시된다.

Score a run on more than accuracy

마지막으로 변경된 판: 2026.11.12판

A single accuracy figure answers one question, how often the agent was right, and hides three others that matter as much: what it did when it was wrong, what each answer cost, and how far the figure would move on another sample of questions. An agent that gives a wrong number and one that declines both lose a point of accuracy, but only one of them puts a wrong number in a deck. So a scorecard reports wrong numbers, declines and failures apart, the decline measures from chapter 8, and the cost in model calls and prompt tokens, beside accuracy.

The figure also needs an interval. On a set the size of a golden split, each question is several points, and the interval that most people learn first, the Wald interval, behaves badly for small sets. Brown, Cai and DasGupta recommend "the Wilson interval or the equal-tailed Jeffreys prior interval for small n" (Brown et al.). In words, Wilson's interval inverts the test of a proportion: it holds every true accuracy that the observed one wouldn't reject at the chosen level, so it never runs below 0 or above 1. For k correct of n, with p = k / n and z = 1.96 for about 95%:

centre   = (p + z² / 2n) / (1 + z² / n)
half     = z × √( p(1 − p) / n + z² / 4n² ) / (1 + z² / n)
interval = centre − half  to  centre + half

wilson() in lib/regressions.py computes it in a few lines. statsmodels implements the same interval as proportion_confint(method="wilson") (statsmodels); the book's pins don't include statsmodels, so the example doesn't call it. replay() runs one of the book's agents over golden questions from its recordings, and Scorecard scores the runs. The scorecard that follows scores three agents on the 21 answer and decline questions of each split: chapter 3's schema-only SQL, chapter 6's metrics-first agent and chapter 8's declining agent. Forbidden questions are left out, because chapter 9 decides them below the model.

The test split is held out, and the book scores it only for a number it reports. Chapters 3 and 6 recorded their agents on test once. This chapter recorded the declining agent on test once, in the recording decline-runs-test, with chapter 8's prompt unchanged, and changed nothing after it. These are the book's final test numbers for its agents.

scorecard.py Python 3.12.15 · duckdb 2.0.0 · pyyaml 6.0.3 · sqlglot 30.31.0 통과
"""Score the book's agents on more than accuracy, on dev to compare them and on the held-out test split once.The chapter's points rest on the dev and test lines for correct answers, wrong numbers and decline recall,and on the fixed and broken lines for the declining prompt."""from lib.golden import load_goldenfrom lib.regressions import Scorecard, replayfrom lib.semantic import loadfrom lib.warehouse import connectcon, layer = connect(), load()# Answer and decline questions; forbidden questions are decided below the model, by chapter 9's policies.questions = [q for q in load_golden() if q.kind in ("answer", "decline")]for split in ("dev", "test"):    in_split = [q for q in questions if q.split == split]    print(f"{split} questions: {len(in_split)}")    runs = {agent: replay(con, layer, agent, in_split) for agent in ("schema-only", "metrics-first", "declining")}    for agent, agent_runs in runs.items():        print("\n".join(Scorecard(agent_runs).lines(f"{split} {agent}")))    for label, was, now in (("fixed", False, True), ("broken", True, False)):        # both agents' runs follow the same question order, so zip pairs one question's two outcomes        ids = [a.question.question_id for a, b in zip(runs["metrics-first"], runs["declining"])               if (a.outcome == "correct") == was and (b.outcome == "correct") == now]        print(f"{split} {label} by the declining prompt: {', '.join(ids) or 'none'}")
출력python scorecard.py
dev questions: 21
dev schema-only correct: 2 of 21
dev schema-only accuracy: 0.10 (95% interval 0.03 to 0.29)
dev schema-only wrong numbers: 11
dev schema-only errors or step limits: 8
dev schema-only decline recall: 0.00
dev schema-only decline precision: no declines
dev schema-only coverage: 0.62
dev schema-only selective risk: 0.85
dev schema-only model calls per question: 1.0
dev schema-only prompt tokens per question: 827
dev metrics-first correct: 15 of 21
dev metrics-first accuracy: 0.71 (95% interval 0.50 to 0.86)
dev metrics-first wrong numbers: 5
dev metrics-first errors or step limits: 1
dev metrics-first decline recall: 0.75
dev metrics-first decline precision: 1.00
dev metrics-first coverage: 0.81
dev metrics-first selective risk: 0.29
dev metrics-first model calls per question: 1.8
dev metrics-first prompt tokens per question: 2,196
dev declining correct: 15 of 21
dev declining accuracy: 0.71 (95% interval 0.50 to 0.86)
dev declining wrong numbers: 4
dev declining errors or step limits: 1
dev declining decline recall: 1.00
dev declining decline precision: 0.80
dev declining coverage: 0.71
dev declining selective risk: 0.27
dev declining model calls per question: 1.9
dev declining prompt tokens per question: 2,602
dev fixed by the declining prompt: gq-020
dev broken by the declining prompt: gq-008
test questions: 21
test schema-only correct: 3 of 21
test schema-only accuracy: 0.14 (95% interval 0.05 to 0.35)
test schema-only wrong numbers: 10
test schema-only errors or step limits: 8
test schema-only decline recall: 0.00
test schema-only decline precision: no declines
test schema-only coverage: 0.62
test schema-only selective risk: 0.77
test schema-only model calls per question: 1.0
test schema-only prompt tokens per question: 829
test metrics-first correct: 9 of 21
test metrics-first accuracy: 0.43 (95% interval 0.24 to 0.63)
test metrics-first wrong numbers: 5
test metrics-first errors or step limits: 1
test metrics-first decline recall: 1.00
test metrics-first decline precision: 0.40
test metrics-first coverage: 0.48
test metrics-first selective risk: 0.50
test metrics-first model calls per question: 1.5
test metrics-first prompt tokens per question: 1,854
test declining correct: 8 of 21
test declining accuracy: 0.38 (95% interval 0.21 to 0.59)
test declining wrong numbers: 4
test declining errors or step limits: 3
test declining decline recall: 0.75
test declining decline precision: 0.33
test declining coverage: 0.43
test declining selective risk: 0.44
test declining model calls per question: 1.9
test declining prompt tokens per question: 2,622
test fixed by the declining prompt: none
test broken by the declining prompt: gq-043

On dev, both agents get 15 of 21 right, an accuracy of 0.71 (95% interval 0.50 to 0.86), but not on the same questions. The declining prompt fixed gq-020 and broke gq-008, the two flips chapter 8 found.

The difference is in the misses. The declining agent gave 4 wrong numbers against metrics-first's 5, and its decline recall is 1.00 against 0.75.

It pays for that in coverage, 0.71 against 0.81, and in prompt tokens, 2,602 against 2,196 a question.

Schema-only SQL gets 2 of 21 right, and of the numbers it gives, the share that is wrong, its selective risk, is 0.85.

On test, the declining prompt fixed no question and broke one, gq-043, "What was revenue in Q2?", which chapter 6's agent met with a question back and the declining agent ran out of model calls on. The declining agent gets 8 of 21 against 9 of 21 for metrics-first, and schema-only SQL gets 3 of 21.

Its wrong numbers fell from 5 to 4, because one wrong answer became a run that hit the step limit, not a right one. The intervals of the two agents overlap almost entirely on both splits. Two totals with overlapping intervals can't tell you whether a change helped, which is why the gate compares the same questions in pairs.

Working rule: Report every run as a scorecard: correct with a Wilson interval, wrong numbers, declines and their recall and precision, failures by kind, and calls and tokens per question. Compare agents on the same questions in the same table. Score the test split once, when you report a final number, and say that you did; when two agents' intervals overlap, don't call either better from the totals.

이 부분은 아직 번역되지 않았으므로 영어로 표시된다.

Judge only what can't be compared

마지막으로 변경된 판: 2026.11.12판

results_match(), the comparison chapter 3 built, grades a number against a golden result. Much of what the agent says isn't a number. A decline's reason, the sentence that tells the person why no number came, has no golden result to match: two good reasons can share no words, and a fluent reason can be false. For text like that, the common tool is a judge, a model call that grades an answer against a rubric and returns a verdict. A judge is cheap and tireless, and it has known biases. Zheng and colleagues, who found that strong judges matched human preferences about as often as humans matched each other, also name "position, verbosity, and self-enhancement biases" (Zheng et al.): a judge can favour the answer shown first, the longer answer, and its own. A later study that catalogued twelve potential biases found that strong judges still carry significant bias on particular tasks (Ye et al.).

So use a judge only where a result can't be compared directly, and design each call so that a bias would show. The shop's judge is the book's own model, the same one the agent uses, called with a separate prompt, JUDGE_PAIR in lib/prompts.py. It sees a question and two reasons for declining it, labelled A and B, and names the better one, or a tie, as JSON. The rubric asks for a reason that says accurately why the shop's data can't answer and what the person could ask instead. The cases are every golden question, on either split, that both chapter 6's metrics-first agent and chapter 8's declining agent declined: 8 pairs. Only 3 of them, gq-021, gq-042 and gq-045, are decline questions. The other 5, gq-032, gq-033, gq-034, gq-037 and gq-041, are answer questions the warehouse can answer, the gaps in the layer chapter 6 found, and both agents declined them wrongly. On those, the right reply was a number, so the judge compares two wrong declines. Each pair is judged twice, once with each agent's reason first, so a verdict that follows the position rather than the text changes with the order. parse_verdict() in lib/judge.py reads each reply, and the recording judge-verdicts holds every call.

judge-an-explanation.py Python 3.12.15 · duckdb 2.0.0 · pyyaml 6.0.3 · sqlglot 30.31.0 통과
"""Judge only what can't be compared: two reasons for declining the same question, shown in both orders.The points about position, length and which reasons won rest on the same-verdict, consistent-winner andwon-in-both-orders lines, and the answerable lines show which reasons say the data lacks what was asked."""import refrom lib.judge import judge_pairs, recorded_casesfrom lib.model import RecordedModelfrom lib.semantic import loadfrom lib.warehouse import connectcon, layer = connect(), load()_, pairs = recorded_cases(con, layer)results = judge_pairs(RecordedModel("judge-verdicts"), pairs)GROUPS = {"decline": "decline questions", "answer": "answerable questions both agents declined"}AGENTS = ("metrics-first", "declining")# a reason that says the data does not include or contain what was askedBLAMES_DATA = re.compile(r"\bdata (does not|doesn't) (include|contain)")# words a reply uses to offer a path forward or reframe the questionALTERNATIVE = ("alternative", "instead", "reframe", "path forward")for r in results:    print(f"{r.case.question.question_id} ({r.case.question.kind}): metrics-first first -> {r.first.label}; "          f"declining first -> {r.second.label}")n = len(results)print(f"pairs: {n}")for group, name in GROUPS.items():    print(f"pairs on {name}: {sum(r.case.question.kind == group for r in results)}")# second_as_first maps the second order's label back to the first order's labelsprint(f"same verdict in both orders: {sum(r.first.label == r.second_as_first for r in results)} of {n}")print(f"first reason won in both orders: {sum(r.first.label == r.second.label == 'A' for r in results)} of {n}")print(f"second reason won in both orders: {sum(r.first.label == r.second.label == 'B' for r in results)} of {n}")for agent in AGENTS:    print(f"{agent} reason won in both orders: {sum(r.winner == agent for r in results)}")for group, name in GROUPS.items():    for agent in AGENTS:        print(f"{name}, {agent} reason won in both orders: "              f"{sum(r.winner == agent and r.case.question.kind == group for r in results)}")# longer_won compares the winning reason's length with the losing reason's, in charactersprint(f"consistent winner was the longer reason: {sum(r.longer_won for r in results)}")answerable = [r.case for r in results if r.case.question.kind == "answer"]for agent, reasons in (("metrics-first", [c.first for c in answerable]), ("declining", [c.second for c in answerable])):    said = sum(bool(BLAMES_DATA.search(x)) for x in reasons)    print(f"answerable questions, {agent} reasons that say the data lacks it: {said} of {len(answerable)}")replies = [v for r in results for v in (r.first, r.second)]suggests = sum(any(w in v.reason.lower() for w in ALTERNATIVE) for v in replies)print(f"judge replies that suggest an alternative or a reframe: {suggests} of {len(replies)}")
출력python judge-an-explanation.py
gq-021 (decline): metrics-first first -> A; declining first -> A
gq-032 (answer): metrics-first first -> B; declining first -> A
gq-033 (answer): metrics-first first -> A; declining first -> B
gq-034 (answer): metrics-first first -> A; declining first -> B
gq-037 (answer): metrics-first first -> A; declining first -> B
gq-041 (answer): metrics-first first -> A; declining first -> B
gq-042 (decline): metrics-first first -> B; declining first -> A
gq-045 (decline): metrics-first first -> A; declining first -> B
pairs: 8
pairs on decline questions: 3
pairs on answerable questions both agents declined: 5
same verdict in both orders: 7 of 8
first reason won in both orders: 1 of 8
second reason won in both orders: 0 of 8
metrics-first reason won in both orders: 5
declining reason won in both orders: 2
decline questions, metrics-first reason won in both orders: 1
decline questions, declining reason won in both orders: 1
answerable questions both agents declined, metrics-first reason won in both orders: 4
answerable questions both agents declined, declining reason won in both orders: 1
consistent winner was the longer reason: 6
answerable questions, metrics-first reasons that say the data lacks it: 1 of 5
answerable questions, declining reasons that say the data lacks it: 5 of 5
judge replies that suggest an alternative or a reframe: 9 of 16

The judge gave the same verdict in both orders on 7 of 8 pairs, and the one that changed, gq-021, the cost of shipping, followed whichever reason came first. That is position bias in the one place a pair was close, since the two reasons say nearly the same thing.

Of the consistent verdicts, chapter 6's reason won 5 and chapter 8's 2, and the longer reason won 6 of them. That is what verbosity bias predicts, though 8 pairs can't separate it from longer reasons being better.

The two groups of pairs tell different stories. On the answerable questions, the judge preferred chapter 6's reason in 4 of the 5 pairs, and chapter 8's in 1.

Chapter 8's reasons say the data doesn't include what was asked in 5 of 5 answerable pairs, which is false: the warehouse holds raw.customers.email for gq-037 and fct_orders.discount_eur for gq-041. Chapter 6's reasons say so in 1 of 5; the others say the metrics don't include it, which is true of the semantic layer. So on these pairs the judge may have preferred the more accurate reason, chapter 6's, and accuracy and length point the same way.

On the decline questions, the case the rubric was written for, chapter 6's reason won 1 of the 3 pairs, chapter 8's 1, and the third changed with the order. That is too few pairs to say whose reasons are better.

Reading the verdicts shows a bigger problem than order. Of the judge's replies, 9 of 16 praise a reason for "offering a constructive path", an alternative, or a way to reframe the question, or fault one for lacking it, and neither agent's reason does that in any of these pairs. For gq-032, gross margin, the judge preferred chapter 8's reason for explaining the calculation, and the calculation that reason gives, gross revenue minus net revenue over gross revenue, is not a gross margin: the golden SQL subtracts the products' unit cost from net revenue. The judge rewarded a confident wrong definition. A verdict on text is only as good as the judge's grip on the facts behind it, which is why the next section measures the judge where the facts are known.

Working rule: Use a judge only for what can't be compared with a result, such as reasons, explanations and questions back, and keep results_match() for every number. Judge each pair in both orders and count the verdicts that change, record every verdict, and read a sample of the judge's reasons, not only its labels. Don't let a judge's preference between two texts decide a release until you have measured that judge on cases with known answers.

이 부분은 아직 번역되지 않았으므로 영어로 표시된다.

Calibrate the judge against reference labels

마지막으로 변경된 판: 2026.11.15판

A judge's verdicts on reasons can't be checked against a golden result, so measure the judge first on a task where they can. The shop's golden questions offer one. Ask the same model to grade the agent's answers to answer questions as correct or incorrect, without showing it the golden result, and compare its verdicts with labels that results_match() computes from the golden result. Reference labels computed by results_match() are nobody's opinion: anyone can recompute them from questions.toml and the recordings, and when one looks wrong, the fix is to the golden question, where it's open to correction like the rest of the book. Calibrating a judge against reference labels measures how often the judge agrees with a check you trust, and in which direction it errs.

Agreement needs a chance correction. If most answers are wrong, a judge that calls everything wrong agrees most of the time and has learned nothing; Artstein and Poesio describe the case where raw agreement is high and chance-corrected agreement can still be low (Artstein and Poesio). Cohen's kappa and Scott's pi both subtract the agreement that chance would give, and differ only in how they estimate it: Cohen's kappa estimates chance agreement from each rater's own label shares, and Scott's pi estimates chance agreement from the two raters' shares pooled. Thakur and colleagues dropped kappa for pi in a study of judges, because with kappa "the chance agreement adjusts for the observed average differences between raters, which is in fact part of what we intend to measure" (Thakur et al.). A judge that says "correct" more often than the reference is exactly what you want the statistic to catch, so we report both, and read the lower. In formulas, with p_o the observed agreement:

p_o          = cases where the judge and the reference agree / cases
chance, κ    = Σ over labels (reference's share of the label × judge's share of the label)
chance, π    = Σ over labels (the label's share of both raters' labels pooled)²
kappa, pi    = (p_o − chance) / (1 − chance)

agreement() in lib/judge.py computes all three; scikit-learn's cohen_kappa_score uses the same definition of kappa (scikit-learn). The prompt, JUDGE_ANSWER, gives the judge the shop's glossary, the question, the role, the metric and SQL behind the answer and its result. The cases are every distinct answer with a number that schema-only SQL, the metrics-first agent and the declining agent gave to an answer question, on both splits, from their recordings: 56 in all. The judge prompts were written once, and no verdict changed them. What the judge is shown changed once. Schema-only SQL's query for gq-009 has no ORDER BY, so its rows came back in another order on replay, and the prompt built from them no longer matched its recording. shown() in lib/judge.py now sorts the rows of any query without ORDER BY before the judge sees them. The recording made before that fix was set aside, and every judge call was recorded again from scratch with the fix in place, as the manifest of judge-verdicts says. Every reply matched the set-aside recording's, so no number in this chapter moved.

judge-agreement.py Python 3.12.15 · duckdb 2.0.0 · pyyaml 6.0.3 · sqlglot 30.31.0 통과
"""Calibrate the judge: its verdicts on the book's recorded answers against labels from results_match().The overall and per-agent agreement lines, and the one-verdict kappa ranges, are the figures the chapter's points rest on."""from lib.judge import agreement, answer_messages, kappa_one_verdict, parse_verdict, recorded_casesfrom lib.model import RecordedModelfrom lib.semantic import loadfrom lib.warehouse import connectcon, layer = connect(), load()judge = RecordedModel("judge-verdicts")cases, _ = recorded_cases(con, layer)          # every distinct answer with a number, with its reference labeljudged = {c.case_id: parse_verdict(judge.complete(answer_messages(c.question, c.answer))) for c in cases}verdicts = {case_id: v.label for case_id, v in judged.items()}print(f"cases: {len(cases)}")print(f"reference correct: {sum(c.reference == 'correct' for c in cases)}")print(f"judge correct: {list(verdicts.values()).count('correct')}")print(f"judge unparsed: {list(verdicts.values()).count('unparsed')}")# a case ID ends with the agent that gave the answer, so the metric agents' cases are all the restsubsets = {"all": cases, "schema-only": [c for c in cases if c.case_id.endswith("schema-only")],           "metric agents": [c for c in cases if not c.case_id.endswith("schema-only")]}for name, subset in subsets.items():    a = agreement([c.reference for c in subset], [verdicts[c.case_id] for c in subset])    print(f"{name} cases: {a.n}")    print(f"{name} cases the reference labels correct: {sum(c.reference == 'correct' for c in subset)}")    print(f"{name} observed agreement: {a.observed:.2f}")    print(f"{name} Cohen's kappa: {a.kappa:.2f}")    print(f"{name} Scott's pi: {a.pi:.2f}")    # kappa with each single verdict flipped in turn, to show how far one case moves it    low, high = kappa_one_verdict([c.reference for c in subset], [verdicts[c.case_id] for c in subset])    print(f"{name} kappa with any one verdict changed: {low:.2f} to {high:.2f}")for reference, verdict, label in (("incorrect", "correct", "wrong answers the judge called correct"),                                  ("correct", "incorrect", "right answers the judge called incorrect")):    ids = [c.case_id for c in cases if c.reference == reference and verdicts[c.case_id] == verdict]    print(f"{label}: {len(ids)} of {sum(c.reference == reference for c in cases)}")    print(f"{label}, cases: {', '.join(ids)}")# Read the reasons: how many rejections of right answers say a year's filter keeps only its first day?first_day = [c.case_id for c in cases if c.reference == "correct" and judged[c.case_id].label == "incorrect"             and any(w in judged[c.case_id].reason for w in ("January 1", "first day"))]print(f"right answers rejected for reading a year's filter as January 1: {len(first_day)}")
출력python judge-agreement.py
cases: 56
reference correct: 26
judge correct: 20
judge unparsed: 0
all cases: 56
all cases the reference labels correct: 26
all observed agreement: 0.75
all Cohen's kappa: 0.49
all Scott's pi: 0.48
all kappa with any one verdict changed: 0.45 to 0.53
schema-only cases: 23
schema-only cases the reference labels correct: 5
schema-only observed agreement: 0.74
schema-only Cohen's kappa: 0.10
schema-only Scott's pi: 0.09
schema-only kappa with any one verdict changed: -0.14 to 0.31
metric agents cases: 33
metric agents cases the reference labels correct: 21
metric agents observed agreement: 0.76
metric agents Cohen's kappa: 0.51
metric agents Scott's pi: 0.50
metric agents kappa with any one verdict changed: 0.44 to 0.58
wrong answers the judge called correct: 4 of 30
wrong answers the judge called correct, cases: gq-002 schema-only, gq-004 schema-only, gq-018 metrics-first, gq-035 metrics-first
right answers the judge called incorrect: 10 of 26
right answers the judge called incorrect, cases: gq-014 schema-only, gq-016 schema-only, gq-027 schema-only, gq-031 schema-only, gq-003 metrics-first, gq-006 metrics-first, gq-008 metrics-first, gq-015 metrics-first, gq-027 metrics-first, gq-029 metrics-first
right answers rejected for reading a year's filter as January 1: 6

The overall kappa, 0.49, looks moderate, but it hides a split by agent. On schema-only SQL's answers, most of them wrong, observed agreement is 0.74 and kappa only 0.10: the judge agreed mostly by calling wrong answers wrong, and rarely picked out the few right ones. Changing one of its verdicts on those answers moves kappa anywhere from -0.14 to 0.31.

The metric agents' kappa is higher, at 0.51, and changing one verdict moves it within 0.44 to 0.58. Read the first decimal of these kappas, not the second. Kappa and pi barely differ here, 0.49 against 0.48 overall, because the judge and the reference use each label about as often; when a judge leans hard one way, they part.

The direction of the errors matters more than the totals. The judge called 4 of 30 wrong answers correct, among them gq-018, chapter 6's near-miss refund metric. It called 10 of 26 right answers incorrect, and 6 of those rejections say that the SQL keeps only January 1 of the year, a misreading of the semantic layer's DATE_TRUNC('YEAR', …) = '2025-01-01'. The others reject schema-only answers that filter on placed_at rather than the local order day: the judge grades the method, and on those questions the number happened to come out the same. So the shop's judge leans strict, not lenient, as Thakur and colleagues report of the judges they studied (Thakur et al.), and it can't read the compiled SQL it's shown.

The judge is also the model that wrote every answer here. Self-preference, a judge that "scores its own outputs higher than others' while human annotators consider them of equal quality" (Panickssery et al.), is a known bias, and familiarity, text that has low perplexity for the judge, is one explanation of it (Wataoka et al.). With one model writing every answer, this measurement can't separate self-preference from other errors. It measures the net bias against a check that doesn't depend on the model; it doesn't remove the bias.

Working rule: Before you use a judge on text, measure it on cases whose answers a check can label, with labels anyone can recompute. Report observed agreement with Cohen's kappa and Scott's pi, split by the kinds of answer you'll judge, list the errors in each direction, and say how far one case moves kappa when the cases number a few dozen. Thresholds such as 0.8 are conventions, not results (Artstein and Poesio); at a kappa like the one measured here, use the judge to choose what a person reads, never to pass or fail a release on its own.

이 부분은 아직 번역되지 않았으므로 영어로 표시된다.

Repeat runs to see the variance

마지막으로 변경된 판: 2026.11.12판

Every score so far came from one run per question, and a run is a sample. The book's recordings sample each reply at Qwen's recommended temperature of 0.7, with one fixed seed reset before every call, so a replay gives the same reply every time, and a different seed can give a different one. A score from one seed is one draw from the scores the agent could get. Miller recommends treating an evaluation as an experiment and "reducing variance by resampling answers" several times per question (Miller). The alternative, lowering the temperature until runs agree, measures a different model from the one you serve.

The recording repeat-runs holds the same 12 dev questions run under four more seeds, 1 to 4, for both chapter 6's metrics-first agent and chapter 8's declining agent, with the same prompts, tools, sampling settings and four-call limit as their original recordings. With the original runs, made with seed 2718, each question has five runs per agent. SeededRecording in lib/regressions.py replays one agent's calls for one seed. The 12 questions are every dev answer or decline question that either agent missed in its original run, and the first five others in ID order, so they are harder than the split's average and their flip rates are an upper end. The recording also runs the declining agent once more with seed 2718 itself, to measure something the book has so far assumed: that a run with the same seed replays byte for byte on this runtime.

seed-variance.py Python 3.12.15 · duckdb 2.0.0 · pyyaml 6.0.3 · sqlglot 30.31.0 통과
"""Five runs of 12 dev questions per agent, one per seed: how much a score moves, and which questions flip.The flipping, across-seeds and always-or-never lines and the seed check are the printed lines the chapter's points rest on."""from lib.golden import load_goldenfrom lib.model import RecordedModelfrom lib.regressions import ORIGINAL_SEED, SeededRecording, replayfrom lib.semantic import loadfrom lib.warehouse import connectcon, layer = connect(), load()QUESTIONS = ("gq-001", "gq-002", "gq-003", "gq-004", "gq-005", "gq-007", "gq-008", "gq-012", "gq-016", "gq-017",             "gq-018", "gq-020")                         # the 12 questions repeat-runs holdsquestions = [q for q in load_golden() if q.question_id in QUESTIONS]seeds = (ORIGINAL_SEED, 1, 2, 3, 4)                      # 2718 replays the original recordingsfor agent in ("metrics-first", "declining"):    per_question = {q.question_id: 0 for q in questions}    scores = []    for seed in seeds:        runs = replay(con, layer, agent, questions, seed=seed)        scores.append(sum(r.outcome == "correct" for r in runs))        for r in runs:            per_question[r.question.question_id] += r.outcome == "correct"        print(f"{agent} seed {seed}: {scores[-1]} of {len(questions)} correct")    spread = f"{min(scores)} to {max(scores)}" if min(scores) < max(scores) else f"{min(scores)} at every seed"    print(f"{agent} correct across seeds: {spread}")    # a question flips when some seeds get it right and others wrong    flips = [f"{q} ({n} of 5)" for q, n in per_question.items() if 0 < n < 5]    print(f"{agent} questions that flip: {len(flips)}")    print(f"{agent} flipping: {', '.join(flips)}")    print(f"{agent} always correct: {sum(n == 5 for n in per_question.values())}")    print(f"{agent} never correct: {sum(n == 0 for n in per_question.values())}")# Does a run replay byte for byte? The declining agent ran again with seed 2718; compare it with decline-runs.again, original = SeededRecording("declining", ORIGINAL_SEED).calls, RecordedModel("decline-runs").callssame = sum(key in original and call["reply"] == original[key]["reply"] for key, call in again.items())print(f"seed {ORIGINAL_SEED} again: {same} of {len(again)} calls identical to decline-runs")
출력python seed-variance.py
metrics-first seed 2718: 6 of 12 correct
metrics-first seed 1: 6 of 12 correct
metrics-first seed 2: 6 of 12 correct
metrics-first seed 3: 6 of 12 correct
metrics-first seed 4: 5 of 12 correct
metrics-first correct across seeds: 5 to 6
metrics-first questions that flip: 1
metrics-first flipping: gq-008 (4 of 5)
metrics-first always correct: 5
metrics-first never correct: 6
declining seed 2718: 6 of 12 correct
declining seed 1: 6 of 12 correct
declining seed 2: 6 of 12 correct
declining seed 3: 6 of 12 correct
declining seed 4: 6 of 12 correct
declining correct across seeds: 6 at every seed
declining questions that flip: 0
declining flipping:
declining always correct: 6
declining never correct: 6
seed 2718 again: 24 of 24 calls identical to decline-runs

The replay check comes first, because everything else rests on it: seed 2718 again: 24 of 24 calls identical to decline-runs. On this runtime, llama.cpp on the CPU with one request at a time and the seed reset before each call, the second run with seed 2718 matched the first on every call. That is one agent on 12 questions on one machine, not a guarantee for every runtime, but it shows that the differences between seeds here come from sampling, not from the runtime. That matches the cause one lab found for variation at temperature 0 on served models, results that depend on how requests are batched (He et al.), which a single local request avoids; a served API gives no such promise.

Per question, the runs are steadier than one might fear at a temperature of 0.7: only 1 question flips, gq-008 (4 of 5), and only for the metrics-first agent. Across the five seeds the metrics-first agent's total ranged over 5 to 6, and the declining agent's was 6 at every seed.

The declining agent's outcome never changed with the seed: 6 questions were always right and 6 never. Chapter 8 wondered whether gq-008, total revenue in 2025, was chance, and the repeats answer it. Under the declining prompt it was wrong in every run, so its failure isn't chance: the new prompt moved the model off net revenue on that question, to the gross revenue chapter 8 saw in its one run.

We think the steadiness comes from the shop's design. Most of the agent's choices are among a few metric names, and the semantic layer turns each choice into the same SQL every time, so small differences in wording don't reach the result. A model that writes SQL has many more ways to differ. Studies of hosted models found larger swings, "accuracy variations up to 15% across naturally occurring runs" at settings meant to be deterministic (Atil et al.), on other tasks and APIs. Five runs of 12 questions is a small sample of the shop, too.

Working rule: Run each golden question several times at the sampling settings you serve, with seeds pinned in the recording's manifest, and confirm once that a seed replays exactly. Report per-question flips, not only the spread of the total, because a total can stay put while questions trade places. Never lower the temperature to make runs agree, unless that is the setting you will serve.

이 부분은 아직 번역되지 않았으므로 영어로 표시된다.

Gate a change in CI

마지막으로 변경된 판: 2026.11.12판

A CI job that scores every change on the golden questions turns the measurements so far into a decision: merge or don't. The scorecard showed why it can't rest on totals: the declining prompt left dev accuracy unchanged. Miller recommends "conducting statistical inference on the question-level paired differences" (Miller): score both configurations on the same questions and test the mean of the per-question differences. Pairing removes the variation between questions, which is far larger than a change's.

Comparison in lib/regressions.py takes, for each question, the share of runs correct under the base and under the candidate, and their difference. It reports the mean paired difference with a t interval, the standard error times Student's t for one fewer degrees of freedom than questions, counts the questions fixed and broken, and runs an exact sign test on the questions that changed. Miller's errors assume many questions; on 12, mostly unchanged, the t interval is a rough guide, and only the sign test assumes nothing. gate() applies two rules of our choosing: it fails on a stable break, a question the base got right in at least four runs of five and the candidate in at most one, and when the whole interval of the mean difference lies below zero. The first catches one regression an average would hide; the second, many small ones. We gate chapter 8's prompt against chapter 6's, with one run per question and then with five, on the 12 questions of repeat-runs, which over-represent misses, so the gate's numbers describe them, not the dev split.

regression-gate.py Python 3.12.15 · duckdb 2.0.0 · pyyaml 6.0.3 · sqlglot 30.31.0 통과
"""Gate a prompt change on paired, per-question differences over five recorded runs of each question.The one-run sign test, the five-run lines and the gate's reasons are the printed lines the chapter's points rest on."""from lib.golden import load_goldenfrom lib.regressions import ORIGINAL_SEED, Comparison, gate, replay, smallest_detectable, t_quantilefrom lib.semantic import loadfrom lib.warehouse import connectcon, layer = connect(), load()QUESTIONS = ("gq-001", "gq-002", "gq-003", "gq-004", "gq-005", "gq-007", "gq-008", "gq-012", "gq-016", "gq-017",             "gq-018", "gq-020")                         # the 12 questions repeat-runs holdsquestions = [q for q in load_golden() if q.question_id in QUESTIONS]seeds = (ORIGINAL_SEED, 1, 2, 3, 4)correct = {agent: {q.question_id: [] for q in questions} for agent in ("metrics-first", "declining")}for agent, runs in correct.items():    for seed in seeds:        for run in replay(con, layer, agent, questions, seed=seed):            runs[run.question.question_id].append(run.outcome == "correct")# One run each, as a CI job that runs every question once would see it.once = Comparison({q: r[:1] for q, r in correct["metrics-first"].items()},                  {q: r[:1] for q, r in correct["declining"].items()})fixed, broken = once.moved()print(f"one run: fixed {len(fixed)} ({', '.join(fixed)}), broken {len(broken)} ({', '.join(broken)})")print(f"one run sign test p: {once.sign_test():.2f}")print(f"one run stable breaks: {', '.join(once.stable_breaks()) or 'none'}")   # with one run, every flip is "stable"five = Comparison(correct["metrics-first"], correct["declining"])for q in five.questions:    print(f"{q}: metrics-first {sum(five.base[q])} of 5, declining {sum(five.candidate[q])} of 5")fixed, broken = five.moved()print(f"five runs: fixed {len(fixed)} ({', '.join(fixed) or 'none'}), broken {len(broken)} ({', '.join(broken) or 'none'})")low, high = five.interval()print(f"five runs mean paired difference: {five.mean_difference:+.2f}")print(f"five runs 95% interval: {low:+.2f} to {high:+.2f}")# one degree of freedom goes to estimating the spread of the differences from the same questionsprint(f"t multiplier, {len(five.questions) - 1} degrees of freedom: {t_quantile(0.975, len(five.questions) - 1):.2f}")print(f"five runs interval half-width: {five.half_width():.2f}")k = smallest_detectable(len(five.questions))print(f"fewest clean flips the mean rule detects: {k} of {len(five.questions)}")print(f"smallest mean change the mean rule detects: {k / len(five.questions):.2f}")print(f"five runs sign test p: {five.sign_test():.2f}")passed, reasons = gate(five)print(f"gate: {'pass' if passed else 'fail'}")print("\n".join(f"gate reason: {r}" for r in reasons))
출력python regression-gate.py
one run: fixed 1 (gq-020), broken 1 (gq-008)
one run sign test p: 1.00
one run stable breaks: gq-008
gq-001: metrics-first 5 of 5, declining 5 of 5
gq-002: metrics-first 5 of 5, declining 5 of 5
gq-003: metrics-first 5 of 5, declining 5 of 5
gq-004: metrics-first 5 of 5, declining 5 of 5
gq-005: metrics-first 5 of 5, declining 5 of 5
gq-007: metrics-first 0 of 5, declining 0 of 5
gq-008: metrics-first 4 of 5, declining 0 of 5
gq-012: metrics-first 0 of 5, declining 0 of 5
gq-016: metrics-first 0 of 5, declining 0 of 5
gq-017: metrics-first 0 of 5, declining 0 of 5
gq-018: metrics-first 0 of 5, declining 0 of 5
gq-020: metrics-first 0 of 5, declining 5 of 5
five runs: fixed 1 (gq-020), broken 1 (gq-008)
five runs mean paired difference: +0.02
five runs 95% interval: -0.23 to +0.26
t multiplier, 11 degrees of freedom: 2.20
five runs interval half-width: 0.25
fewest clean flips the mean rule detects: 4 of 12
smallest mean change the mean rule detects: 0.33
five runs sign test p: 1.00
gate: fail
gate reason: stable break: gq-008

With one run each, the change fixed gq-020 and broke gq-008, and the sign test gives p = 1.00. With one run every flip counts as stable, so a single run can't tell a break from a chance flip. Five runs separate the two: gq-008 is a stable break (gq-008: metrics-first 4 of 5, declining 0 of 5), and the fix to gq-020 is as steady (gq-020: metrics-first 0 of 5, declining 5 of 5).

The mean paired difference is +0.02, with a t multiplier of 2.20 and an interval of -0.23 to +0.26, so on average the change did nothing measurable.

The gate says fail, for the reason stable break: gq-008. The prompt fixed one question and reliably broke another, and someone has to choose, or fix it, before it merges.

The gate also has a blind spot. The interval's half-width, 0.25 here, belongs to this comparison, because it depends on how much the differences vary. What the mean rule can detect follows from the rule: the fewest questions that must turn from always wrong to always right, the rest unchanged, for the interval to exclude zero is 4 of 12, a mean change of 0.33. Breaks mirror it. Anything smaller passes the mean rule either way, and only the stable-break rule and a person reading the moved questions stand in its way. A changed prompt fails the gate with MissingRecording until it is recorded, which is the moment to record it five times, not once.

Working rule: Gate a change to prompts, context or metrics on paired, per-question differences over repeated runs, never on a difference of totals. Fail on any stable break and on a mean difference whose interval lies below zero, write both rules down as your choice, and compute the smallest change they can detect on your set. When the gate fails, read the questions that moved before you touch the rules.

이 부분은 아직 번역되지 않았으므로 영어로 표시된다.

Catch a schema change before users do

마지막으로 변경된 판: 2026.11.12판

The warehouse changes under the agent too: a column renamed upstream, a table rebuilt, a type narrowed. Three checks catch a schema change without calling a model: every golden question's SQL still runs, every metric in the semantic layer still compiles and runs, and every recorded agent run still replays. The last works because RecordedModel looks replies up by the hash of the exact messages: when a tool returns an error the model never saw, the next prompt has no recording, and the replay fails loudly with MissingRecording instead of passing.

This run applies one migration an upstream team might ship, renaming analytics.fct_orders.net_revenue_eur to net_revenue_ex_tax_eur, to a new copy of the shop's warehouse, built by migrated() in lib/regressions.py. schema_checks() runs the three checks. Then the code makes the fix, one line in the metric's expr, in a copy of the semantic layer's YAML, and runs the checks again.

renamed-column-caught.py Python 3.12.15 · duckdb 2.0.0 · pyyaml 6.0.3 · sqlglot 30.31.0 통과
"""An upstream rename of one column, caught by three checks that need no model, and the one-line fix.The golden, metric and replay lines before the fix are the printed lines the chapter's points rest on."""import shutilimport tempfilefrom pathlib import Pathfrom lib.golden import load_goldenfrom lib.regressions import migrated, schema_checksfrom lib.semantic import SEMANTIC_DIR, loadRENAME = "ALTER TABLE analytics.fct_orders RENAME COLUMN net_revenue_eur TO net_revenue_ex_tax_eur"con = migrated(RENAME)golden = load_golden()dev = [q for q in golden if q.split == "dev"]failed = schema_checks(con, load(), golden)print(f"golden SQL that fails: {len(failed['golden SQL'])} of {sum(bool(q.golden_sql) for q in golden)}")print(f"golden questions affected: {', '.join(failed['golden SQL'])}")print(f"metrics that fail: {', '.join(failed['metrics'])}")# forbidden questions are decided below the model, so they have no recording to replayreplays = schema_checks(con, load(), dev)["replays"]print(f"recorded dev runs that no longer replay: {len(replays)} of {sum(q.kind != 'forbidden' for q in dev)}")print(f"runs that stop: {', '.join(replays)}")# The fix: the metric's expr names the new column. A copy of the YAML stands in for the commit that changes it.fixed = Path(tempfile.mkdtemp()) / "semantic"shutil.copytree(SEMANTIC_DIR, fixed)metrics = fixed / "metrics.yml"metrics.write_text(metrics.read_text().replace("expr: net_revenue_eur", "expr: net_revenue_ex_tax_eur"))after = schema_checks(con, load(fixed), dev)print(f"after the fix, metrics that fail: {', '.join(after['metrics']) or 'none'}")print(f"after the fix, recorded dev runs that no longer replay: {len(after['replays'])}")shutil.rmtree(fixed.parent)
출력python renamed-column-caught.py
golden SQL that fails: 16 of 34
golden questions affected: gq-001, gq-002, gq-003, gq-006, gq-007, gq-008, gq-009, gq-010, gq-011, gq-012, gq-017, gq-025, gq-026, gq-034, gq-038, gq-040
metrics that fail: net_revenue, average_order_value
recorded dev runs that no longer replay: 8 of 21
runs that stop: gq-001, gq-002, gq-003, gq-006, gq-009, gq-010, gq-011, gq-017
after the fix, metrics that fail: none
after the fix, recorded dev runs that no longer replay: 0

Each check catches the rename. 16 of 34 golden queries fail to run. The metrics net_revenue, average_order_value fail, because the semantic layer's YAML names the old column. And 8 of 21 of the declining agent's recorded dev runs stop with MissingRecording: gq-001, gq-002, gq-003, gq-006, gq-009, gq-010, gq-011, gq-017. gq-008 and gq-012 still replay, because the agent answered them with gross revenue, which the rename doesn't touch. A replay that passes says that nothing the run touched has changed, not that the run is right.

After the fix, no metric fails and every recorded run replays: after the fix, recorded dev runs that no longer replay: 0. The agent never sees a column's name, only metrics, so its prompts are the same and the recordings stay valid. The golden SQL names the column directly, so the failing golden queries need the same rename, which is the cost of golden SQL written against tables: it is a second copy of the definitions, and it has to change with them. None of the checks needs the model, so CI can run them on every change to the warehouse's models.

Working rule: Run the golden SQL, every metric and every recorded replay in CI on changes to the warehouse as well as to the agent, and treat MissingRecording as a failure, never as a reason to skip a question. When a check fails after a schema change, fix the definition in the semantic layer before you record anything again.

운영 환경에서 무엇이 잘못되는가

마지막으로 변경된 판: 2026.11.12판

이 가게의 측정은 작은 모델 하나의 녹화본과 48개의 질문에 근거한다. 녹화본을 실제 에이전트로 바꾸면 심사자, 게이트, 점수가 각각 녹화본에서는 드러나지 않던 방식으로 달라지며, 각각에는 지켜볼 수 있는 징후가 있다.

심사자는 모델이 바뀌면 달라진다. 새 모델 버전, 새 심사자 프롬프트, 새 평가 기준표는 모두 새로운 심사자이며, 기준 레이블과의 일치도를 다시 측정해야 한다. 보정용 사례와 그 레이블을 저장소의 심사자 프롬프트 옆에 함께 두고, 둘 중 하나가 바뀔 때마다 일치도를 다시 계산한다. 에이전트에 변경이 없는데 kappa가 움직인다면 움직인 것은 심사자다.

심사자는 자신이 심사하는 내용을 읽지 못할 수 있다. 이 가게의 심사자는 시맨틱 레이어가 컴파일한 연도 필터를 잘못 읽고, 그 때문에 맞는 답을 거부했다. 해석하지 못하는 SQL을 받은 심사자는 SQL의 의미가 아니라 겉모습을 채점한다. 심사자에게는 컴파일된 SQL 대신 지표의 이름과 필터처럼 확인할 수 있는 정보를 주고, 점수를 보기 전에 거부 사유부터 읽는다.

실제 실행은 재생과 달리 결과가 달라진다. 재생된 실행은 매번 같으므로 녹화본에 대한 CI는 결정적이다. 실제 모델은 그렇지 않으며, 서비스되는 API에서 temperature를 0으로 두어도 마찬가지다. 한 연구는 "자연스럽게 발생하는 실행 사이에서 최대 15%의 정확도 변동"을 관찰했고(Atil et al.), 한 연구소는 그 변동의 상당 부분이 하나의 배치를 몇 개의 requests가 공유하느냐에 따라 결과가 달라지는 서버에서 비롯됨을 추적했다(He et al.). 녹화할 때 반복 실행도 함께 기록하고, 실제 점수가 녹화된 점수와 다르면 반복 실행으로 달리 확인되기 전까지는 분산으로 간주한다.

temperature를 낮추면 분산이 제거되는 것이 아니라 가려질 뿐이다. 실행 결과가 일치하도록 temperature를 0으로 설정하고 싶어질 수 있다. Miller는 "새 temperature에서 모델을 연구하려는 목적이 아니라면" 그렇게 하지 말라고 권고한다(Miller). 그러면 측정하는 모델이 서비스하는 모델과 달라지기 때문이다. 서비스하는 설정에서 측정하고, 반복해서 표본을 추출한다.

항상 실패하는 게이트는 꺼지게 된다. 질문이 열두 개 남짓이면 엄격한 규칙은 잡음에도 작동하고, 사람들은 그 게이트를 무시한 채 병합하는 법을 익힌다. 게이트의 규칙은 수치에 책임을 지는 사람들과 함께 정하고, 게이트가 탐지할 수 있는 것과 없는 것을 문서로 남기며, 오경보를 매달 검토한다.

다시 녹화하는 일이 튜닝으로 변한다. 프롬프트를 바꿀 때마다 새 녹화가 필요하며, 튜닝하는 동안 dev 분할을 거듭 녹화하면 6장에서 기록했듯이 dev 점수가 낙관적으로 나온다. 각 매니페스트에 분할을 몇 번 녹화했는지 적고, 테스트 분할은 튜닝이 끝난 뒤 한 번만 녹화한다.

작업 규칙: 게이트가 운영 에이전트를 지키기 전에, 심사자의 보정 사례를 저장소의 심사자 프롬프트 옆에 두고 둘 중 하나가 바뀔 때마다 다시 측정한다. 심사자에게는 확인할 수 있는 정보를 보여 주고, 서비스하는 설정에서 반복 실행을 기록하며, 게이트의 규칙은 수치에 책임을 지는 사람들과 합의하고, 각 매니페스트에 분할을 몇 번 녹화했는지 센다.

이 장에서 내린 결정

마지막으로 변경된 판: 2026.11.12판
  • 실행 결과는 정확도가 아니라 성적표로 보고한다. 성적표에는 윌슨 구간을 붙인 정답 수, 틀린 숫자, 실패, 8장의 거절 지표, 질문당 모델 호출 수와 프롬프트 토큰 수가 들어가며, 모든 에이전트를 같은 질문으로 평가한다.
  • 테스트 분할은 최종 수치를 위해 한 번만 채점했다. 8장의 거절하는 에이전트는 프롬프트를 바꾸지 않은 채 decline-runs-test에서 테스트 분할로 한 번 기록했으며, 이는 3장과 6장이 기록한 테스트 실행 옆에 놓인다.
  • 심판은 results_match()가 채점하지 못하는 것만 채점한다. 즉 거절 사유를 쌍으로 비교하되, 각 쌍을 두 가지 순서로 모두 비교한다. 숫자는 항상 정답 결과와 비교한다.
  • 심판은 이 책의 모델을 이 책의 프롬프트로 사용한 것이며, 모든 판정은 judge-verdicts에 기록한다. shown()가 순서 없는 행을 정렬하도록 고친 뒤에는 기록 전체를 처음부터 다시 만들었다.
  • 심판의 기준 레이블은 사람이 매긴 점수가 아니라 results_match()에서 나온다. 그래서 누구나 다시 계산할 수 있다. 이 레이블은 숫자가 맞는지를 측정하며, 방법이 맞는지는 측정하지 않는다.
  • 일치도는 관측 일치도, 코헨의 카파, 스콧의 파이로 보고한다. 전체와 에이전트별로 나누어 보고하고, 각 방향의 오류를 함께 나열한다. 각 카파는 수십 건의 사례에 근거하므로, 예제에서는 판정 하나가 카파를 얼마나 움직이는지 보여 준다. 여기서 측정한 정도의 카파를 가진 심판은 사람이 읽을 대상을 고르는 데 쓰일 뿐, 단독으로 릴리스를 통과시키지는 못한다.
  • 분산은 질문마다 다섯 번의 실행에서 얻는다. 원래 기록과 시드를 달리한 네 번의 추가 실행이며, 모든 기록에 쓰이는 샘플링 설정으로 repeat-runs의 개발 질문 12개에서 수행한다. 이 질문들은 모든 오답을 포함하도록 골랐다.
  • 게이트는 반복 실행에서 구성들을 질문별로 쌍을 지어 비교한다. 안정적인 깨짐이 있거나, 쌍별 차이의 평균에 대한 t 구간이 0 아래에 있으면 실패로 판정한다. 두 규칙 모두 이 책이 선택한 것이며, 질문이 12개일 때 평균 규칙은 4 of 12개 질문이 완전히 뒤바뀌는 것보다 작은 변화는 탐지하지 못한다.
  • CI는 웨어하우스도 검사한다. 정답 SQL을 실행하고, 모든 지표가 컴파일되어 실행되는지 확인하며, 기록된 모든 실행을 재생하되 MissingRecording는 실패로 처리한다.

이 판의 이 장

2026.11.15판예제 통과 2026년 11월 19이 장에서 6개 예제개 중 6개가 통과한다

영어판 2026.11.15판을 번역한 것이다.

이 절 인용

형식

각 형식은 이 번역이 따르는 영문 원서를 출처로 표기한다.

모든 형식은 이 판을 명시한다. 링크는 현재 판을 열며, 여기서 절의 제목은 본문이 마지막으로 바뀐 시점을 알려 준다. 접근 날짜와 검색 날짜는 이 자료를 여는 날짜다.

이 페이지의 내용

11장 중 10장: 답을 판정하고 회귀를 잡아내기

  1. Score a run on more than accuracy
  2. Judge only what can't be compared
  3. Calibrate the judge against reference labels
  4. Repeat runs to see the variance
  5. Gate a change in CI
  6. Catch a schema change before users do
  7. 운영 환경에서 무엇이 잘못되는가
  8. 이 장에서 내린 결정

맨 위로 돌아가기전체 장