回答を評価し、リグレッションを検出する

読了まで約72 分分、8 節 節の 6 例 例。

第2026.11.15版 2026年11月19日時点で例が通過 78 例中 78 例が合格 変更点

AIによる翻訳。英語版は人による検証済みである。 当社の書籍の作り方

英語版の書籍(版2026.11.15)から翻訳した。 原文を読む

この章を終えると、次のことができるようになる。エージェントの1回の実行を複数の指標で同時に採点する。結果を直接比較できない場合に限って、モデルを評価者として使う。その評価者を、誰でも再計算できる基準ラベルに照らして較正する。スコアが実行ごとにどれだけ変動するかを把握する。プロンプト、コンテキスト、指標への変更を、記録済みの実行に基づいて CI でゲートにかける。

ほとんどの章がショップのエージェントを変更してきた。第4章ではプロンプトにコンテキストを加え、第6章ではエージェントをセマンティックレイヤーの指標に移行した。第8章では、理由を添えて回答を断るようにプロンプトを書き換えた。各変更は、golden questions の dev 分割で採点した。golden questions とは、検証済みの回答を持つショップの48個の質問である。採点は質問ごとに1回の実行で行ったため、大きな変化は示せても、小さな変化は示せない。第8章の新しいプロンプトは、dev の質問を1つ修正し、別の1つを壊した。壊れたのは2025年の総売上を問うgq-008である。このような1件の反転について、第8章は「偶然かもしれない」としか言えなかった。

この章では、そうした疑問に決着をつける測定を構築する。まず、実行結果を正解率だけでなく、数値をどこまで信頼できるかを示す区間とともに報告する。次に評価者を追加する。評価者とは、golden との比較では採点できないものを採点するモデル呼び出しであり、信頼する前にその評価者自体を測定する。さらに、質問を5回ずつ実行して分散を確認する。最後に、これらすべてを、変更のたびに CI のジョブが記録から、オフラインで実行できるゲートにまとめる。

この部分はまだ翻訳されていないため、英語で表示している。

Score a run on more than accuracy

最終変更:版2026.11.12

A single accuracy figure answers one question, how often the agent was right, and hides three others that matter as much: what it did when it was wrong, what each answer cost, and how far the figure would move on another sample of questions. An agent that gives a wrong number and one that declines both lose a point of accuracy, but only one of them puts a wrong number in a deck. So a scorecard reports wrong numbers, declines and failures apart, the decline measures from chapter 8, and the cost in model calls and prompt tokens, beside accuracy.

The figure also needs an interval. On a set the size of a golden split, each question is several points, and the interval that most people learn first, the Wald interval, behaves badly for small sets. Brown, Cai and DasGupta recommend "the Wilson interval or the equal-tailed Jeffreys prior interval for small n" (Brown et al.). In words, Wilson's interval inverts the test of a proportion: it holds every true accuracy that the observed one wouldn't reject at the chosen level, so it never runs below 0 or above 1. For k correct of n, with p = k / n and z = 1.96 for about 95%:

centre   = (p + z² / 2n) / (1 + z² / n)
half     = z × √( p(1 − p) / n + z² / 4n² ) / (1 + z² / n)
interval = centre − half  to  centre + half

wilson() in lib/regressions.py computes it in a few lines. statsmodels implements the same interval as proportion_confint(method="wilson") (statsmodels); the book's pins don't include statsmodels, so the example doesn't call it. replay() runs one of the book's agents over golden questions from its recordings, and Scorecard scores the runs. The scorecard that follows scores three agents on the 21 answer and decline questions of each split: chapter 3's schema-only SQL, chapter 6's metrics-first agent and chapter 8's declining agent. Forbidden questions are left out, because chapter 9 decides them below the model.

The test split is held out, and the book scores it only for a number it reports. Chapters 3 and 6 recorded their agents on test once. This chapter recorded the declining agent on test once, in the recording decline-runs-test, with chapter 8's prompt unchanged, and changed nothing after it. These are the book's final test numbers for its agents.

scorecard.py Python 3.12.15 · duckdb 2.0.0 · pyyaml 6.0.3 · sqlglot 30.31.0 合格
"""Score the book's agents on more than accuracy, on dev to compare them and on the held-out test split once.The chapter's points rest on the dev and test lines for correct answers, wrong numbers and decline recall,and on the fixed and broken lines for the declining prompt."""from lib.golden import load_goldenfrom lib.regressions import Scorecard, replayfrom lib.semantic import loadfrom lib.warehouse import connectcon, layer = connect(), load()# Answer and decline questions; forbidden questions are decided below the model, by chapter 9's policies.questions = [q for q in load_golden() if q.kind in ("answer", "decline")]for split in ("dev", "test"):    in_split = [q for q in questions if q.split == split]    print(f"{split} questions: {len(in_split)}")    runs = {agent: replay(con, layer, agent, in_split) for agent in ("schema-only", "metrics-first", "declining")}    for agent, agent_runs in runs.items():        print("\n".join(Scorecard(agent_runs).lines(f"{split} {agent}")))    for label, was, now in (("fixed", False, True), ("broken", True, False)):        # both agents' runs follow the same question order, so zip pairs one question's two outcomes        ids = [a.question.question_id for a, b in zip(runs["metrics-first"], runs["declining"])               if (a.outcome == "correct") == was and (b.outcome == "correct") == now]        print(f"{split} {label} by the declining prompt: {', '.join(ids) or 'none'}")
出力python scorecard.py
dev questions: 21
dev schema-only correct: 2 of 21
dev schema-only accuracy: 0.10 (95% interval 0.03 to 0.29)
dev schema-only wrong numbers: 11
dev schema-only errors or step limits: 8
dev schema-only decline recall: 0.00
dev schema-only decline precision: no declines
dev schema-only coverage: 0.62
dev schema-only selective risk: 0.85
dev schema-only model calls per question: 1.0
dev schema-only prompt tokens per question: 827
dev metrics-first correct: 15 of 21
dev metrics-first accuracy: 0.71 (95% interval 0.50 to 0.86)
dev metrics-first wrong numbers: 5
dev metrics-first errors or step limits: 1
dev metrics-first decline recall: 0.75
dev metrics-first decline precision: 1.00
dev metrics-first coverage: 0.81
dev metrics-first selective risk: 0.29
dev metrics-first model calls per question: 1.8
dev metrics-first prompt tokens per question: 2,196
dev declining correct: 15 of 21
dev declining accuracy: 0.71 (95% interval 0.50 to 0.86)
dev declining wrong numbers: 4
dev declining errors or step limits: 1
dev declining decline recall: 1.00
dev declining decline precision: 0.80
dev declining coverage: 0.71
dev declining selective risk: 0.27
dev declining model calls per question: 1.9
dev declining prompt tokens per question: 2,602
dev fixed by the declining prompt: gq-020
dev broken by the declining prompt: gq-008
test questions: 21
test schema-only correct: 3 of 21
test schema-only accuracy: 0.14 (95% interval 0.05 to 0.35)
test schema-only wrong numbers: 10
test schema-only errors or step limits: 8
test schema-only decline recall: 0.00
test schema-only decline precision: no declines
test schema-only coverage: 0.62
test schema-only selective risk: 0.77
test schema-only model calls per question: 1.0
test schema-only prompt tokens per question: 829
test metrics-first correct: 9 of 21
test metrics-first accuracy: 0.43 (95% interval 0.24 to 0.63)
test metrics-first wrong numbers: 5
test metrics-first errors or step limits: 1
test metrics-first decline recall: 1.00
test metrics-first decline precision: 0.40
test metrics-first coverage: 0.48
test metrics-first selective risk: 0.50
test metrics-first model calls per question: 1.5
test metrics-first prompt tokens per question: 1,854
test declining correct: 8 of 21
test declining accuracy: 0.38 (95% interval 0.21 to 0.59)
test declining wrong numbers: 4
test declining errors or step limits: 3
test declining decline recall: 0.75
test declining decline precision: 0.33
test declining coverage: 0.43
test declining selective risk: 0.44
test declining model calls per question: 1.9
test declining prompt tokens per question: 2,622
test fixed by the declining prompt: none
test broken by the declining prompt: gq-043

On dev, both agents get 15 of 21 right, an accuracy of 0.71 (95% interval 0.50 to 0.86), but not on the same questions. The declining prompt fixed gq-020 and broke gq-008, the two flips chapter 8 found.

The difference is in the misses. The declining agent gave 4 wrong numbers against metrics-first's 5, and its decline recall is 1.00 against 0.75.

It pays for that in coverage, 0.71 against 0.81, and in prompt tokens, 2,602 against 2,196 a question.

Schema-only SQL gets 2 of 21 right, and of the numbers it gives, the share that is wrong, its selective risk, is 0.85.

On test, the declining prompt fixed no question and broke one, gq-043, "What was revenue in Q2?", which chapter 6's agent met with a question back and the declining agent ran out of model calls on. The declining agent gets 8 of 21 against 9 of 21 for metrics-first, and schema-only SQL gets 3 of 21.

Its wrong numbers fell from 5 to 4, because one wrong answer became a run that hit the step limit, not a right one. The intervals of the two agents overlap almost entirely on both splits. Two totals with overlapping intervals can't tell you whether a change helped, which is why the gate compares the same questions in pairs.

Working rule: Report every run as a scorecard: correct with a Wilson interval, wrong numbers, declines and their recall and precision, failures by kind, and calls and tokens per question. Compare agents on the same questions in the same table. Score the test split once, when you report a final number, and say that you did; when two agents' intervals overlap, don't call either better from the totals.

この部分はまだ翻訳されていないため、英語で表示している。

Judge only what can't be compared

最終変更:版2026.11.12

results_match(), the comparison chapter 3 built, grades a number against a golden result. Much of what the agent says isn't a number. A decline's reason, the sentence that tells the person why no number came, has no golden result to match: two good reasons can share no words, and a fluent reason can be false. For text like that, the common tool is a judge, a model call that grades an answer against a rubric and returns a verdict. A judge is cheap and tireless, and it has known biases. Zheng and colleagues, who found that strong judges matched human preferences about as often as humans matched each other, also name "position, verbosity, and self-enhancement biases" (Zheng et al.): a judge can favour the answer shown first, the longer answer, and its own. A later study that catalogued twelve potential biases found that strong judges still carry significant bias on particular tasks (Ye et al.).

So use a judge only where a result can't be compared directly, and design each call so that a bias would show. The shop's judge is the book's own model, the same one the agent uses, called with a separate prompt, JUDGE_PAIR in lib/prompts.py. It sees a question and two reasons for declining it, labelled A and B, and names the better one, or a tie, as JSON. The rubric asks for a reason that says accurately why the shop's data can't answer and what the person could ask instead. The cases are every golden question, on either split, that both chapter 6's metrics-first agent and chapter 8's declining agent declined: 8 pairs. Only 3 of them, gq-021, gq-042 and gq-045, are decline questions. The other 5, gq-032, gq-033, gq-034, gq-037 and gq-041, are answer questions the warehouse can answer, the gaps in the layer chapter 6 found, and both agents declined them wrongly. On those, the right reply was a number, so the judge compares two wrong declines. Each pair is judged twice, once with each agent's reason first, so a verdict that follows the position rather than the text changes with the order. parse_verdict() in lib/judge.py reads each reply, and the recording judge-verdicts holds every call.

judge-an-explanation.py Python 3.12.15 · duckdb 2.0.0 · pyyaml 6.0.3 · sqlglot 30.31.0 合格
"""Judge only what can't be compared: two reasons for declining the same question, shown in both orders.The points about position, length and which reasons won rest on the same-verdict, consistent-winner andwon-in-both-orders lines, and the answerable lines show which reasons say the data lacks what was asked."""import refrom lib.judge import judge_pairs, recorded_casesfrom lib.model import RecordedModelfrom lib.semantic import loadfrom lib.warehouse import connectcon, layer = connect(), load()_, pairs = recorded_cases(con, layer)results = judge_pairs(RecordedModel("judge-verdicts"), pairs)GROUPS = {"decline": "decline questions", "answer": "answerable questions both agents declined"}AGENTS = ("metrics-first", "declining")# a reason that says the data does not include or contain what was askedBLAMES_DATA = re.compile(r"\bdata (does not|doesn't) (include|contain)")# words a reply uses to offer a path forward or reframe the questionALTERNATIVE = ("alternative", "instead", "reframe", "path forward")for r in results:    print(f"{r.case.question.question_id} ({r.case.question.kind}): metrics-first first -> {r.first.label}; "          f"declining first -> {r.second.label}")n = len(results)print(f"pairs: {n}")for group, name in GROUPS.items():    print(f"pairs on {name}: {sum(r.case.question.kind == group for r in results)}")# second_as_first maps the second order's label back to the first order's labelsprint(f"same verdict in both orders: {sum(r.first.label == r.second_as_first for r in results)} of {n}")print(f"first reason won in both orders: {sum(r.first.label == r.second.label == 'A' for r in results)} of {n}")print(f"second reason won in both orders: {sum(r.first.label == r.second.label == 'B' for r in results)} of {n}")for agent in AGENTS:    print(f"{agent} reason won in both orders: {sum(r.winner == agent for r in results)}")for group, name in GROUPS.items():    for agent in AGENTS:        print(f"{name}, {agent} reason won in both orders: "              f"{sum(r.winner == agent and r.case.question.kind == group for r in results)}")# longer_won compares the winning reason's length with the losing reason's, in charactersprint(f"consistent winner was the longer reason: {sum(r.longer_won for r in results)}")answerable = [r.case for r in results if r.case.question.kind == "answer"]for agent, reasons in (("metrics-first", [c.first for c in answerable]), ("declining", [c.second for c in answerable])):    said = sum(bool(BLAMES_DATA.search(x)) for x in reasons)    print(f"answerable questions, {agent} reasons that say the data lacks it: {said} of {len(answerable)}")replies = [v for r in results for v in (r.first, r.second)]suggests = sum(any(w in v.reason.lower() for w in ALTERNATIVE) for v in replies)print(f"judge replies that suggest an alternative or a reframe: {suggests} of {len(replies)}")
出力python judge-an-explanation.py
gq-021 (decline): metrics-first first -> A; declining first -> A
gq-032 (answer): metrics-first first -> B; declining first -> A
gq-033 (answer): metrics-first first -> A; declining first -> B
gq-034 (answer): metrics-first first -> A; declining first -> B
gq-037 (answer): metrics-first first -> A; declining first -> B
gq-041 (answer): metrics-first first -> A; declining first -> B
gq-042 (decline): metrics-first first -> B; declining first -> A
gq-045 (decline): metrics-first first -> A; declining first -> B
pairs: 8
pairs on decline questions: 3
pairs on answerable questions both agents declined: 5
same verdict in both orders: 7 of 8
first reason won in both orders: 1 of 8
second reason won in both orders: 0 of 8
metrics-first reason won in both orders: 5
declining reason won in both orders: 2
decline questions, metrics-first reason won in both orders: 1
decline questions, declining reason won in both orders: 1
answerable questions both agents declined, metrics-first reason won in both orders: 4
answerable questions both agents declined, declining reason won in both orders: 1
consistent winner was the longer reason: 6
answerable questions, metrics-first reasons that say the data lacks it: 1 of 5
answerable questions, declining reasons that say the data lacks it: 5 of 5
judge replies that suggest an alternative or a reframe: 9 of 16

The judge gave the same verdict in both orders on 7 of 8 pairs, and the one that changed, gq-021, the cost of shipping, followed whichever reason came first. That is position bias in the one place a pair was close, since the two reasons say nearly the same thing.

Of the consistent verdicts, chapter 6's reason won 5 and chapter 8's 2, and the longer reason won 6 of them. That is what verbosity bias predicts, though 8 pairs can't separate it from longer reasons being better.

The two groups of pairs tell different stories. On the answerable questions, the judge preferred chapter 6's reason in 4 of the 5 pairs, and chapter 8's in 1.

Chapter 8's reasons say the data doesn't include what was asked in 5 of 5 answerable pairs, which is false: the warehouse holds raw.customers.email for gq-037 and fct_orders.discount_eur for gq-041. Chapter 6's reasons say so in 1 of 5; the others say the metrics don't include it, which is true of the semantic layer. So on these pairs the judge may have preferred the more accurate reason, chapter 6's, and accuracy and length point the same way.

On the decline questions, the case the rubric was written for, chapter 6's reason won 1 of the 3 pairs, chapter 8's 1, and the third changed with the order. That is too few pairs to say whose reasons are better.

Reading the verdicts shows a bigger problem than order. Of the judge's replies, 9 of 16 praise a reason for "offering a constructive path", an alternative, or a way to reframe the question, or fault one for lacking it, and neither agent's reason does that in any of these pairs. For gq-032, gross margin, the judge preferred chapter 8's reason for explaining the calculation, and the calculation that reason gives, gross revenue minus net revenue over gross revenue, is not a gross margin: the golden SQL subtracts the products' unit cost from net revenue. The judge rewarded a confident wrong definition. A verdict on text is only as good as the judge's grip on the facts behind it, which is why the next section measures the judge where the facts are known.

Working rule: Use a judge only for what can't be compared with a result, such as reasons, explanations and questions back, and keep results_match() for every number. Judge each pair in both orders and count the verdicts that change, record every verdict, and read a sample of the judge's reasons, not only its labels. Don't let a judge's preference between two texts decide a release until you have measured that judge on cases with known answers.

この部分はまだ翻訳されていないため、英語で表示している。

Calibrate the judge against reference labels

最終変更:版2026.11.15

A judge's verdicts on reasons can't be checked against a golden result, so measure the judge first on a task where they can. The shop's golden questions offer one. Ask the same model to grade the agent's answers to answer questions as correct or incorrect, without showing it the golden result, and compare its verdicts with labels that results_match() computes from the golden result. Reference labels computed by results_match() are nobody's opinion: anyone can recompute them from questions.toml and the recordings, and when one looks wrong, the fix is to the golden question, where it's open to correction like the rest of the book. Calibrating a judge against reference labels measures how often the judge agrees with a check you trust, and in which direction it errs.

Agreement needs a chance correction. If most answers are wrong, a judge that calls everything wrong agrees most of the time and has learned nothing; Artstein and Poesio describe the case where raw agreement is high and chance-corrected agreement can still be low (Artstein and Poesio). Cohen's kappa and Scott's pi both subtract the agreement that chance would give, and differ only in how they estimate it: Cohen's kappa estimates chance agreement from each rater's own label shares, and Scott's pi estimates chance agreement from the two raters' shares pooled. Thakur and colleagues dropped kappa for pi in a study of judges, because with kappa "the chance agreement adjusts for the observed average differences between raters, which is in fact part of what we intend to measure" (Thakur et al.). A judge that says "correct" more often than the reference is exactly what you want the statistic to catch, so we report both, and read the lower. In formulas, with p_o the observed agreement:

p_o          = cases where the judge and the reference agree / cases
chance, κ    = Σ over labels (reference's share of the label × judge's share of the label)
chance, π    = Σ over labels (the label's share of both raters' labels pooled)²
kappa, pi    = (p_o − chance) / (1 − chance)

agreement() in lib/judge.py computes all three; scikit-learn's cohen_kappa_score uses the same definition of kappa (scikit-learn). The prompt, JUDGE_ANSWER, gives the judge the shop's glossary, the question, the role, the metric and SQL behind the answer and its result. The cases are every distinct answer with a number that schema-only SQL, the metrics-first agent and the declining agent gave to an answer question, on both splits, from their recordings: 56 in all. The judge prompts were written once, and no verdict changed them. What the judge is shown changed once. Schema-only SQL's query for gq-009 has no ORDER BY, so its rows came back in another order on replay, and the prompt built from them no longer matched its recording. shown() in lib/judge.py now sorts the rows of any query without ORDER BY before the judge sees them. The recording made before that fix was set aside, and every judge call was recorded again from scratch with the fix in place, as the manifest of judge-verdicts says. Every reply matched the set-aside recording's, so no number in this chapter moved.

judge-agreement.py Python 3.12.15 · duckdb 2.0.0 · pyyaml 6.0.3 · sqlglot 30.31.0 合格
"""Calibrate the judge: its verdicts on the book's recorded answers against labels from results_match().The overall and per-agent agreement lines, and the one-verdict kappa ranges, are the figures the chapter's points rest on."""from lib.judge import agreement, answer_messages, kappa_one_verdict, parse_verdict, recorded_casesfrom lib.model import RecordedModelfrom lib.semantic import loadfrom lib.warehouse import connectcon, layer = connect(), load()judge = RecordedModel("judge-verdicts")cases, _ = recorded_cases(con, layer)          # every distinct answer with a number, with its reference labeljudged = {c.case_id: parse_verdict(judge.complete(answer_messages(c.question, c.answer))) for c in cases}verdicts = {case_id: v.label for case_id, v in judged.items()}print(f"cases: {len(cases)}")print(f"reference correct: {sum(c.reference == 'correct' for c in cases)}")print(f"judge correct: {list(verdicts.values()).count('correct')}")print(f"judge unparsed: {list(verdicts.values()).count('unparsed')}")# a case ID ends with the agent that gave the answer, so the metric agents' cases are all the restsubsets = {"all": cases, "schema-only": [c for c in cases if c.case_id.endswith("schema-only")],           "metric agents": [c for c in cases if not c.case_id.endswith("schema-only")]}for name, subset in subsets.items():    a = agreement([c.reference for c in subset], [verdicts[c.case_id] for c in subset])    print(f"{name} cases: {a.n}")    print(f"{name} cases the reference labels correct: {sum(c.reference == 'correct' for c in subset)}")    print(f"{name} observed agreement: {a.observed:.2f}")    print(f"{name} Cohen's kappa: {a.kappa:.2f}")    print(f"{name} Scott's pi: {a.pi:.2f}")    # kappa with each single verdict flipped in turn, to show how far one case moves it    low, high = kappa_one_verdict([c.reference for c in subset], [verdicts[c.case_id] for c in subset])    print(f"{name} kappa with any one verdict changed: {low:.2f} to {high:.2f}")for reference, verdict, label in (("incorrect", "correct", "wrong answers the judge called correct"),                                  ("correct", "incorrect", "right answers the judge called incorrect")):    ids = [c.case_id for c in cases if c.reference == reference and verdicts[c.case_id] == verdict]    print(f"{label}: {len(ids)} of {sum(c.reference == reference for c in cases)}")    print(f"{label}, cases: {', '.join(ids)}")# Read the reasons: how many rejections of right answers say a year's filter keeps only its first day?first_day = [c.case_id for c in cases if c.reference == "correct" and judged[c.case_id].label == "incorrect"             and any(w in judged[c.case_id].reason for w in ("January 1", "first day"))]print(f"right answers rejected for reading a year's filter as January 1: {len(first_day)}")
出力python judge-agreement.py
cases: 56
reference correct: 26
judge correct: 20
judge unparsed: 0
all cases: 56
all cases the reference labels correct: 26
all observed agreement: 0.75
all Cohen's kappa: 0.49
all Scott's pi: 0.48
all kappa with any one verdict changed: 0.45 to 0.53
schema-only cases: 23
schema-only cases the reference labels correct: 5
schema-only observed agreement: 0.74
schema-only Cohen's kappa: 0.10
schema-only Scott's pi: 0.09
schema-only kappa with any one verdict changed: -0.14 to 0.31
metric agents cases: 33
metric agents cases the reference labels correct: 21
metric agents observed agreement: 0.76
metric agents Cohen's kappa: 0.51
metric agents Scott's pi: 0.50
metric agents kappa with any one verdict changed: 0.44 to 0.58
wrong answers the judge called correct: 4 of 30
wrong answers the judge called correct, cases: gq-002 schema-only, gq-004 schema-only, gq-018 metrics-first, gq-035 metrics-first
right answers the judge called incorrect: 10 of 26
right answers the judge called incorrect, cases: gq-014 schema-only, gq-016 schema-only, gq-027 schema-only, gq-031 schema-only, gq-003 metrics-first, gq-006 metrics-first, gq-008 metrics-first, gq-015 metrics-first, gq-027 metrics-first, gq-029 metrics-first
right answers rejected for reading a year's filter as January 1: 6

The overall kappa, 0.49, looks moderate, but it hides a split by agent. On schema-only SQL's answers, most of them wrong, observed agreement is 0.74 and kappa only 0.10: the judge agreed mostly by calling wrong answers wrong, and rarely picked out the few right ones. Changing one of its verdicts on those answers moves kappa anywhere from -0.14 to 0.31.

The metric agents' kappa is higher, at 0.51, and changing one verdict moves it within 0.44 to 0.58. Read the first decimal of these kappas, not the second. Kappa and pi barely differ here, 0.49 against 0.48 overall, because the judge and the reference use each label about as often; when a judge leans hard one way, they part.

The direction of the errors matters more than the totals. The judge called 4 of 30 wrong answers correct, among them gq-018, chapter 6's near-miss refund metric. It called 10 of 26 right answers incorrect, and 6 of those rejections say that the SQL keeps only January 1 of the year, a misreading of the semantic layer's DATE_TRUNC('YEAR', …) = '2025-01-01'. The others reject schema-only answers that filter on placed_at rather than the local order day: the judge grades the method, and on those questions the number happened to come out the same. So the shop's judge leans strict, not lenient, as Thakur and colleagues report of the judges they studied (Thakur et al.), and it can't read the compiled SQL it's shown.

The judge is also the model that wrote every answer here. Self-preference, a judge that "scores its own outputs higher than others' while human annotators consider them of equal quality" (Panickssery et al.), is a known bias, and familiarity, text that has low perplexity for the judge, is one explanation of it (Wataoka et al.). With one model writing every answer, this measurement can't separate self-preference from other errors. It measures the net bias against a check that doesn't depend on the model; it doesn't remove the bias.

Working rule: Before you use a judge on text, measure it on cases whose answers a check can label, with labels anyone can recompute. Report observed agreement with Cohen's kappa and Scott's pi, split by the kinds of answer you'll judge, list the errors in each direction, and say how far one case moves kappa when the cases number a few dozen. Thresholds such as 0.8 are conventions, not results (Artstein and Poesio); at a kappa like the one measured here, use the judge to choose what a person reads, never to pass or fail a release on its own.

この部分はまだ翻訳されていないため、英語で表示している。

Repeat runs to see the variance

最終変更:版2026.11.12

Every score so far came from one run per question, and a run is a sample. The book's recordings sample each reply at Qwen's recommended temperature of 0.7, with one fixed seed reset before every call, so a replay gives the same reply every time, and a different seed can give a different one. A score from one seed is one draw from the scores the agent could get. Miller recommends treating an evaluation as an experiment and "reducing variance by resampling answers" several times per question (Miller). The alternative, lowering the temperature until runs agree, measures a different model from the one you serve.

The recording repeat-runs holds the same 12 dev questions run under four more seeds, 1 to 4, for both chapter 6's metrics-first agent and chapter 8's declining agent, with the same prompts, tools, sampling settings and four-call limit as their original recordings. With the original runs, made with seed 2718, each question has five runs per agent. SeededRecording in lib/regressions.py replays one agent's calls for one seed. The 12 questions are every dev answer or decline question that either agent missed in its original run, and the first five others in ID order, so they are harder than the split's average and their flip rates are an upper end. The recording also runs the declining agent once more with seed 2718 itself, to measure something the book has so far assumed: that a run with the same seed replays byte for byte on this runtime.

seed-variance.py Python 3.12.15 · duckdb 2.0.0 · pyyaml 6.0.3 · sqlglot 30.31.0 合格
"""Five runs of 12 dev questions per agent, one per seed: how much a score moves, and which questions flip.The flipping, across-seeds and always-or-never lines and the seed check are the printed lines the chapter's points rest on."""from lib.golden import load_goldenfrom lib.model import RecordedModelfrom lib.regressions import ORIGINAL_SEED, SeededRecording, replayfrom lib.semantic import loadfrom lib.warehouse import connectcon, layer = connect(), load()QUESTIONS = ("gq-001", "gq-002", "gq-003", "gq-004", "gq-005", "gq-007", "gq-008", "gq-012", "gq-016", "gq-017",             "gq-018", "gq-020")                         # the 12 questions repeat-runs holdsquestions = [q for q in load_golden() if q.question_id in QUESTIONS]seeds = (ORIGINAL_SEED, 1, 2, 3, 4)                      # 2718 replays the original recordingsfor agent in ("metrics-first", "declining"):    per_question = {q.question_id: 0 for q in questions}    scores = []    for seed in seeds:        runs = replay(con, layer, agent, questions, seed=seed)        scores.append(sum(r.outcome == "correct" for r in runs))        for r in runs:            per_question[r.question.question_id] += r.outcome == "correct"        print(f"{agent} seed {seed}: {scores[-1]} of {len(questions)} correct")    spread = f"{min(scores)} to {max(scores)}" if min(scores) < max(scores) else f"{min(scores)} at every seed"    print(f"{agent} correct across seeds: {spread}")    # a question flips when some seeds get it right and others wrong    flips = [f"{q} ({n} of 5)" for q, n in per_question.items() if 0 < n < 5]    print(f"{agent} questions that flip: {len(flips)}")    print(f"{agent} flipping: {', '.join(flips)}")    print(f"{agent} always correct: {sum(n == 5 for n in per_question.values())}")    print(f"{agent} never correct: {sum(n == 0 for n in per_question.values())}")# Does a run replay byte for byte? The declining agent ran again with seed 2718; compare it with decline-runs.again, original = SeededRecording("declining", ORIGINAL_SEED).calls, RecordedModel("decline-runs").callssame = sum(key in original and call["reply"] == original[key]["reply"] for key, call in again.items())print(f"seed {ORIGINAL_SEED} again: {same} of {len(again)} calls identical to decline-runs")
出力python seed-variance.py
metrics-first seed 2718: 6 of 12 correct
metrics-first seed 1: 6 of 12 correct
metrics-first seed 2: 6 of 12 correct
metrics-first seed 3: 6 of 12 correct
metrics-first seed 4: 5 of 12 correct
metrics-first correct across seeds: 5 to 6
metrics-first questions that flip: 1
metrics-first flipping: gq-008 (4 of 5)
metrics-first always correct: 5
metrics-first never correct: 6
declining seed 2718: 6 of 12 correct
declining seed 1: 6 of 12 correct
declining seed 2: 6 of 12 correct
declining seed 3: 6 of 12 correct
declining seed 4: 6 of 12 correct
declining correct across seeds: 6 at every seed
declining questions that flip: 0
declining flipping:
declining always correct: 6
declining never correct: 6
seed 2718 again: 24 of 24 calls identical to decline-runs

The replay check comes first, because everything else rests on it: seed 2718 again: 24 of 24 calls identical to decline-runs. On this runtime, llama.cpp on the CPU with one request at a time and the seed reset before each call, the second run with seed 2718 matched the first on every call. That is one agent on 12 questions on one machine, not a guarantee for every runtime, but it shows that the differences between seeds here come from sampling, not from the runtime. That matches the cause one lab found for variation at temperature 0 on served models, results that depend on how requests are batched (He et al.), which a single local request avoids; a served API gives no such promise.

Per question, the runs are steadier than one might fear at a temperature of 0.7: only 1 question flips, gq-008 (4 of 5), and only for the metrics-first agent. Across the five seeds the metrics-first agent's total ranged over 5 to 6, and the declining agent's was 6 at every seed.

The declining agent's outcome never changed with the seed: 6 questions were always right and 6 never. Chapter 8 wondered whether gq-008, total revenue in 2025, was chance, and the repeats answer it. Under the declining prompt it was wrong in every run, so its failure isn't chance: the new prompt moved the model off net revenue on that question, to the gross revenue chapter 8 saw in its one run.

We think the steadiness comes from the shop's design. Most of the agent's choices are among a few metric names, and the semantic layer turns each choice into the same SQL every time, so small differences in wording don't reach the result. A model that writes SQL has many more ways to differ. Studies of hosted models found larger swings, "accuracy variations up to 15% across naturally occurring runs" at settings meant to be deterministic (Atil et al.), on other tasks and APIs. Five runs of 12 questions is a small sample of the shop, too.

Working rule: Run each golden question several times at the sampling settings you serve, with seeds pinned in the recording's manifest, and confirm once that a seed replays exactly. Report per-question flips, not only the spread of the total, because a total can stay put while questions trade places. Never lower the temperature to make runs agree, unless that is the setting you will serve.

この部分はまだ翻訳されていないため、英語で表示している。

Gate a change in CI

最終変更:版2026.11.12

A CI job that scores every change on the golden questions turns the measurements so far into a decision: merge or don't. The scorecard showed why it can't rest on totals: the declining prompt left dev accuracy unchanged. Miller recommends "conducting statistical inference on the question-level paired differences" (Miller): score both configurations on the same questions and test the mean of the per-question differences. Pairing removes the variation between questions, which is far larger than a change's.

Comparison in lib/regressions.py takes, for each question, the share of runs correct under the base and under the candidate, and their difference. It reports the mean paired difference with a t interval, the standard error times Student's t for one fewer degrees of freedom than questions, counts the questions fixed and broken, and runs an exact sign test on the questions that changed. Miller's errors assume many questions; on 12, mostly unchanged, the t interval is a rough guide, and only the sign test assumes nothing. gate() applies two rules of our choosing: it fails on a stable break, a question the base got right in at least four runs of five and the candidate in at most one, and when the whole interval of the mean difference lies below zero. The first catches one regression an average would hide; the second, many small ones. We gate chapter 8's prompt against chapter 6's, with one run per question and then with five, on the 12 questions of repeat-runs, which over-represent misses, so the gate's numbers describe them, not the dev split.

regression-gate.py Python 3.12.15 · duckdb 2.0.0 · pyyaml 6.0.3 · sqlglot 30.31.0 合格
"""Gate a prompt change on paired, per-question differences over five recorded runs of each question.The one-run sign test, the five-run lines and the gate's reasons are the printed lines the chapter's points rest on."""from lib.golden import load_goldenfrom lib.regressions import ORIGINAL_SEED, Comparison, gate, replay, smallest_detectable, t_quantilefrom lib.semantic import loadfrom lib.warehouse import connectcon, layer = connect(), load()QUESTIONS = ("gq-001", "gq-002", "gq-003", "gq-004", "gq-005", "gq-007", "gq-008", "gq-012", "gq-016", "gq-017",             "gq-018", "gq-020")                         # the 12 questions repeat-runs holdsquestions = [q for q in load_golden() if q.question_id in QUESTIONS]seeds = (ORIGINAL_SEED, 1, 2, 3, 4)correct = {agent: {q.question_id: [] for q in questions} for agent in ("metrics-first", "declining")}for agent, runs in correct.items():    for seed in seeds:        for run in replay(con, layer, agent, questions, seed=seed):            runs[run.question.question_id].append(run.outcome == "correct")# One run each, as a CI job that runs every question once would see it.once = Comparison({q: r[:1] for q, r in correct["metrics-first"].items()},                  {q: r[:1] for q, r in correct["declining"].items()})fixed, broken = once.moved()print(f"one run: fixed {len(fixed)} ({', '.join(fixed)}), broken {len(broken)} ({', '.join(broken)})")print(f"one run sign test p: {once.sign_test():.2f}")print(f"one run stable breaks: {', '.join(once.stable_breaks()) or 'none'}")   # with one run, every flip is "stable"five = Comparison(correct["metrics-first"], correct["declining"])for q in five.questions:    print(f"{q}: metrics-first {sum(five.base[q])} of 5, declining {sum(five.candidate[q])} of 5")fixed, broken = five.moved()print(f"five runs: fixed {len(fixed)} ({', '.join(fixed) or 'none'}), broken {len(broken)} ({', '.join(broken) or 'none'})")low, high = five.interval()print(f"five runs mean paired difference: {five.mean_difference:+.2f}")print(f"five runs 95% interval: {low:+.2f} to {high:+.2f}")# one degree of freedom goes to estimating the spread of the differences from the same questionsprint(f"t multiplier, {len(five.questions) - 1} degrees of freedom: {t_quantile(0.975, len(five.questions) - 1):.2f}")print(f"five runs interval half-width: {five.half_width():.2f}")k = smallest_detectable(len(five.questions))print(f"fewest clean flips the mean rule detects: {k} of {len(five.questions)}")print(f"smallest mean change the mean rule detects: {k / len(five.questions):.2f}")print(f"five runs sign test p: {five.sign_test():.2f}")passed, reasons = gate(five)print(f"gate: {'pass' if passed else 'fail'}")print("\n".join(f"gate reason: {r}" for r in reasons))
出力python regression-gate.py
one run: fixed 1 (gq-020), broken 1 (gq-008)
one run sign test p: 1.00
one run stable breaks: gq-008
gq-001: metrics-first 5 of 5, declining 5 of 5
gq-002: metrics-first 5 of 5, declining 5 of 5
gq-003: metrics-first 5 of 5, declining 5 of 5
gq-004: metrics-first 5 of 5, declining 5 of 5
gq-005: metrics-first 5 of 5, declining 5 of 5
gq-007: metrics-first 0 of 5, declining 0 of 5
gq-008: metrics-first 4 of 5, declining 0 of 5
gq-012: metrics-first 0 of 5, declining 0 of 5
gq-016: metrics-first 0 of 5, declining 0 of 5
gq-017: metrics-first 0 of 5, declining 0 of 5
gq-018: metrics-first 0 of 5, declining 0 of 5
gq-020: metrics-first 0 of 5, declining 5 of 5
five runs: fixed 1 (gq-020), broken 1 (gq-008)
five runs mean paired difference: +0.02
five runs 95% interval: -0.23 to +0.26
t multiplier, 11 degrees of freedom: 2.20
five runs interval half-width: 0.25
fewest clean flips the mean rule detects: 4 of 12
smallest mean change the mean rule detects: 0.33
five runs sign test p: 1.00
gate: fail
gate reason: stable break: gq-008

With one run each, the change fixed gq-020 and broke gq-008, and the sign test gives p = 1.00. With one run every flip counts as stable, so a single run can't tell a break from a chance flip. Five runs separate the two: gq-008 is a stable break (gq-008: metrics-first 4 of 5, declining 0 of 5), and the fix to gq-020 is as steady (gq-020: metrics-first 0 of 5, declining 5 of 5).

The mean paired difference is +0.02, with a t multiplier of 2.20 and an interval of -0.23 to +0.26, so on average the change did nothing measurable.

The gate says fail, for the reason stable break: gq-008. The prompt fixed one question and reliably broke another, and someone has to choose, or fix it, before it merges.

The gate also has a blind spot. The interval's half-width, 0.25 here, belongs to this comparison, because it depends on how much the differences vary. What the mean rule can detect follows from the rule: the fewest questions that must turn from always wrong to always right, the rest unchanged, for the interval to exclude zero is 4 of 12, a mean change of 0.33. Breaks mirror it. Anything smaller passes the mean rule either way, and only the stable-break rule and a person reading the moved questions stand in its way. A changed prompt fails the gate with MissingRecording until it is recorded, which is the moment to record it five times, not once.

Working rule: Gate a change to prompts, context or metrics on paired, per-question differences over repeated runs, never on a difference of totals. Fail on any stable break and on a mean difference whose interval lies below zero, write both rules down as your choice, and compute the smallest change they can detect on your set. When the gate fails, read the questions that moved before you touch the rules.

この部分はまだ翻訳されていないため、英語で表示している。

Catch a schema change before users do

最終変更:版2026.11.12

The warehouse changes under the agent too: a column renamed upstream, a table rebuilt, a type narrowed. Three checks catch a schema change without calling a model: every golden question's SQL still runs, every metric in the semantic layer still compiles and runs, and every recorded agent run still replays. The last works because RecordedModel looks replies up by the hash of the exact messages: when a tool returns an error the model never saw, the next prompt has no recording, and the replay fails loudly with MissingRecording instead of passing.

This run applies one migration an upstream team might ship, renaming analytics.fct_orders.net_revenue_eur to net_revenue_ex_tax_eur, to a new copy of the shop's warehouse, built by migrated() in lib/regressions.py. schema_checks() runs the three checks. Then the code makes the fix, one line in the metric's expr, in a copy of the semantic layer's YAML, and runs the checks again.

renamed-column-caught.py Python 3.12.15 · duckdb 2.0.0 · pyyaml 6.0.3 · sqlglot 30.31.0 合格
"""An upstream rename of one column, caught by three checks that need no model, and the one-line fix.The golden, metric and replay lines before the fix are the printed lines the chapter's points rest on."""import shutilimport tempfilefrom pathlib import Pathfrom lib.golden import load_goldenfrom lib.regressions import migrated, schema_checksfrom lib.semantic import SEMANTIC_DIR, loadRENAME = "ALTER TABLE analytics.fct_orders RENAME COLUMN net_revenue_eur TO net_revenue_ex_tax_eur"con = migrated(RENAME)golden = load_golden()dev = [q for q in golden if q.split == "dev"]failed = schema_checks(con, load(), golden)print(f"golden SQL that fails: {len(failed['golden SQL'])} of {sum(bool(q.golden_sql) for q in golden)}")print(f"golden questions affected: {', '.join(failed['golden SQL'])}")print(f"metrics that fail: {', '.join(failed['metrics'])}")# forbidden questions are decided below the model, so they have no recording to replayreplays = schema_checks(con, load(), dev)["replays"]print(f"recorded dev runs that no longer replay: {len(replays)} of {sum(q.kind != 'forbidden' for q in dev)}")print(f"runs that stop: {', '.join(replays)}")# The fix: the metric's expr names the new column. A copy of the YAML stands in for the commit that changes it.fixed = Path(tempfile.mkdtemp()) / "semantic"shutil.copytree(SEMANTIC_DIR, fixed)metrics = fixed / "metrics.yml"metrics.write_text(metrics.read_text().replace("expr: net_revenue_eur", "expr: net_revenue_ex_tax_eur"))after = schema_checks(con, load(fixed), dev)print(f"after the fix, metrics that fail: {', '.join(after['metrics']) or 'none'}")print(f"after the fix, recorded dev runs that no longer replay: {len(after['replays'])}")shutil.rmtree(fixed.parent)
出力python renamed-column-caught.py
golden SQL that fails: 16 of 34
golden questions affected: gq-001, gq-002, gq-003, gq-006, gq-007, gq-008, gq-009, gq-010, gq-011, gq-012, gq-017, gq-025, gq-026, gq-034, gq-038, gq-040
metrics that fail: net_revenue, average_order_value
recorded dev runs that no longer replay: 8 of 21
runs that stop: gq-001, gq-002, gq-003, gq-006, gq-009, gq-010, gq-011, gq-017
after the fix, metrics that fail: none
after the fix, recorded dev runs that no longer replay: 0

Each check catches the rename. 16 of 34 golden queries fail to run. The metrics net_revenue, average_order_value fail, because the semantic layer's YAML names the old column. And 8 of 21 of the declining agent's recorded dev runs stop with MissingRecording: gq-001, gq-002, gq-003, gq-006, gq-009, gq-010, gq-011, gq-017. gq-008 and gq-012 still replay, because the agent answered them with gross revenue, which the rename doesn't touch. A replay that passes says that nothing the run touched has changed, not that the run is right.

After the fix, no metric fails and every recorded run replays: after the fix, recorded dev runs that no longer replay: 0. The agent never sees a column's name, only metrics, so its prompts are the same and the recordings stay valid. The golden SQL names the column directly, so the failing golden queries need the same rename, which is the cost of golden SQL written against tables: it is a second copy of the definitions, and it has to change with them. None of the checks needs the model, so CI can run them on every change to the warehouse's models.

Working rule: Run the golden SQL, every metric and every recorded replay in CI on changes to the warehouse as well as to the agent, and treat MissingRecording as a failure, never as a reason to skip a question. When a check fails after a schema change, fix the definition in the semantic layer before you record anything again.

本番環境で何がうまくいかなくなるか

最終変更:版2026.11.12

このショップの測定は、小さな1つのモデルの録画と48件の質問に基づいている。録画を実際のエージェントに置き換えると、判定器、ゲート、スコアのそれぞれが、録画では見えなかった形で変化する。それぞれに、注意して見ておくべき兆候がある。

判定器は、そのモデルが変わると変わる。 新しいモデルバージョン、新しい判定器プロンプト、新しい評価基準は、いずれも新しい判定器であり、参照ラベルとの一致度を測り直す必要がある。較正用のケースとそのラベルは、判定器プロンプトと並べてリポジトリに置き、どちらかが変わるたびに一致度を再計測する。エージェントに変更がないのにカッパ係数が動いたなら、動いたのは判定器である。

判定器は、自分が判定する対象を読めないことがある。 このショップの判定器は、セマンティックレイヤーがコンパイルした年フィルタを読み誤り、そのせいで正しい回答を不合格にした。解析できない SQL を見せられた判定器は、SQL の意味ではなく見た目を採点する。判定器には、コンパイル済みの SQL ではなく指標の名前とフィルタなど、判定器が確認できるものを与える。また、スコアを見る前に、不合格にした理由を読む。

実際の実行には、再生にはないばらつきがある。 再生した実行は毎回同じなので、録画に対する CI は決定的である。実際のモデルはそうではなく、提供される API で temperature を 0 にしても同様である。ある研究では「自然に発生する実行間で最大15%の精度変動」が観察された(Atil et al.)。また、ある研究所は、その大部分を、結果が1つのバッチを共有する requests の数に依存するサーバーに起因するものとして突き止めた(He et al.)。録画するときは、繰り返し実行も記録する。録画したスコアと異なる実行時のスコアは、繰り返し実行で別の結果が示されるまではばらつきとして扱う。

temperature を下げても、ばらつきは取り除かれず隠れるだけである。 実行結果をそろえるために temperature を 0 にしたくなる。Miller は、これを「新しい temperature でそのモデルを調べることが目的でない限り」勧めない(Miller)。そうすると、測定しているモデルは提供するモデルと同じではなくなるからである。提供するときと同じ設定で測定し、再サンプリングする。

常に失敗するゲートは、無効にされる。 質問が十数件しかないと、厳格なルールはノイズで作動し、人々はそれを無視してマージすることを覚える。ゲートのルールは、その数値に責任を負う人々とともに定め、ゲートが検出できることとできないことを書き残し、誤警報を毎月見直す。

再録画はチューニングに変わってしまう。 プロンプトを変更するたびに新しい録画が必要になり、チューニング中に dev 分割を繰り返し録画すると、第6章で記録したように dev は楽観的になる。各マニフェストに分割を何回録画したかを記し、test 分割はチューニングが終わった後に1回だけ録画する。

実践ルール: ゲートが本番のエージェントを守る前に、判定器の較正用ケースを判定器プロンプトと並べて保管し、どちらかを変更するたびに再計測する。判定器には確認できるものを見せる。繰り返し実行は、提供するときと同じ設定で記録する。ゲートのルールは、数値に責任を負う人々と合意して定める。各マニフェストには、分割を何回録画したかを数えて記す。

本章での決定事項

最終変更:版2026.11.12
  • 実行結果は正解率ではなくスコアカードとして報告する。 スコアカードには、Wilson 区間つきの正解数、誤った数値、失敗、第8章の回答拒否に関する指標、1問あたりのモデル呼び出し回数とプロンプトトークン数を載せる。すべてのエージェントについて、同じ質問で算出する。
  • テスト分割は、最終的な数値のために1回だけ採点した。 第8章の回答を拒否するエージェントは、プロンプトを変えないまま、decline-runs-test でテストデータに対して1回だけ記録した。第3章と第6章で記録したテスト実行も、これと並べて示してある。
  • 審判が採点するのは、results_match() では判定できないものだけである。 対象は回答を拒否した理由であり、ペアで比較し、各ペアを両方の順序で評価する。数値は常に正解の結果と比較する。
  • 審判には、本書自身のモデルと本書自身のプロンプトを使う。 すべての判定は judge-verdicts に記録される。shown() を修正して順序の定まらない行を並べ替えるようにしたあと、記録全体を最初からやり直した。
  • 審判の基準ラベルは、人間による採点ではなく results_match() から得る。 そのため、誰でもラベルを再計算できる。このラベルが測るのは数値が正しいかどうかであり、方法が正しいかどうかではない。
  • 一致度は、観測一致率、Cohen の kappa、Scott の pi で報告する。 全体とエージェント別に示し、各方向の誤りも一覧にする。各 kappa は数十件の事例に基づくため、1件の判定で kappa がどれだけ動くかを例で示す。ここで測定した程度の kappa である審判は、人間が何を読むかを選ぶために使うものであり、単独でリリースを通すことはない。
  • ばらつきは、1問あたり5回の実行から求める。 元の記録に4つのシードを加えた実行であり、すべての記録で使うサンプリング設定のまま、repeat-runs の12問の開発用質問で行う。この12問は、誤答をすべて含むように選んだ。
  • ゲートは、設定どうしを質問ごとにペアで比較する。 繰り返し実行したうえで、安定した退行があるとき、または対応のある差の平均について t 区間が0を下回るときに、ゲートは失敗とする。どちらの規則も本書の選択であり、12問では、平均の規則は 4 of 12 問が完全に入れ替わる場合より小さい変化を検出できない。
  • CI はウェアハウスも検査する。 正解の SQL を実行し、すべての指標がコンパイルされて実行されることを確かめ、記録されたすべての実行を再生する。このとき MissingRecording は失敗として扱う。

この版における本章

第2026.11.15版2026年11月19日時点で例が通過この章の6 例件のうち6件が合格

英語版の書籍(版2026.11.15)から翻訳した。

この節を引用する

形式

いずれの形式も、本訳書が従う英語版の書籍を出典として示す。

いずれの形式もこの版を明示している。そのリンクは現行版を開き、節の見出しには本文が最後に変更された日付が示される。アクセス日と取得日は、これを開いた日である。

このページの内容

全11章中の第10章: 回答を評価し、リグレッションを検出する

  1. Score a run on more than accuracy
  2. Judge only what can't be compared
  3. Calibrate the judge against reference labels
  4. Repeat runs to see the variance
  5. Gate a change in CI
  6. Catch a schema change before users do
  7. 本番環境で何がうまくいかなくなるか
  8. 本章での決定事項

ページの先頭に戻るすべての章