Teaching kit
From Edition 2026.11.3, the book's current edition. It changes when the book does. Use and adapt it freely, with credit to the book.
On this page
Syllabus
Reading times are for the chapter's text. Students also run its examples, in the Lab or on their own machine.
A chapter a week: 11 weeks
In 5 weeks
Objectives and outlines
Each chapter's objectives are its first sentence. Each outline follows the chapter's sections, with the terms it defines, the predictions it asks for before an example runs, and the rule each section ends on.
1. Measure an assistant before you change it
About 21 minutes to read, 11 examples.
By the end, students can measure a documentation assistant with a fixed set of questions, read the result with its interval, and say when twenty questions are too few to tell two prompts apart.
Lecture outline: 11 sections
-
The tool and its docs
- Run
list-the-docs.py - Working rule Write the version into every question, and let that version's own docs be the only source for its answer. When two versions use the same wording and give different answers, make them two items, one for each version. Never let the newest version's page stand in for an older one.
- Run
-
The twenty items
- Run
twenty-questions.py - Working rule Write each item with its version, its kind and its expected answer before you run the assistant, and label any item written to catch a specific mistake. Keep the set small enough to read in one sitting, and never change an expected answer to fit what the assistant said.
- Run
-
Retrieve, then answer
- Run
retrieve-top-sections.py - Working rule Check retrieval before you check the answer. For each item, record the sections the assistant was given next to its reply. A wrong reply that lacks the right section is a retrieval problem, and a wrong reply that had the right section is a problem with the prompt or the model. Keep the wording of questions plain, because a tokeniser that drops one-character tokens will lose a measurement or a version number.
- Run
-
Replay the answers
- Run
replay-one-answer.py - Working rule Replay for every comparison. Record again only when the model, the prompt or the sections the assistant sees have changed, and note in the recording what changed. Each recording names its prompt by hash, and the example checks that hash against the recording's manifest, so one recording can't stand in for a different prompt.
- Run
- Two code checks
-
Citations
- Run
check-citations.py - Working rule Run the citation check on every reply, and report its rates apart from the answer rate, since the two count different mistakes. A rate of zero is only a result if the check could have found a problem, so keep a fixture reply that cites outside its prompt, and run the check on it, to show the check still fires.
- Run
-
Declines
- Run
check-declines.py - Working rule Keep the two decline rates apart, and leave version-dependent and ambiguous items out of both. A reply counts as a correct decline only when it is the decline phrase and nothing else, so a reply that gives the sentence and then an answer counts as the error it is. Read each false refusal by hand before you change the prompt, since one of them may point at a real gap in the docs. Read each decline on a version-dependent item too, since the rate doesn't count those.
- Run
-
One judge
- Run
judge-one-answer.py - Working rule Give the judge a written reference for every item, and read its FAIL verdicts as closely as its PASS verdicts before you trust a rate built on them. Don't count agreement with your own labels as calibration. Until a judge has been checked against human ratings, report its rate as the judge's rate, not as a measure of the assistant's quality.
- Run
-
A pass rate and its interval
- Run
pass-rate-interval.py - Working rule Report every pass rate with its denominator, its interval and its rate for each kind of item. Treat a kind with only a few items as a note, not a result, and say how many items it rests on.
- Run
-
Two prompts
- Run
compare-two-prompts.py - Run
judge-gets-one-wrong.py - Working rule Compare prompts on the same items, paired, and report the difference with its interval. Call one prompt better only when the whole interval lies above zero, and worse only when it lies below zero. When the interval includes zero, say that no difference was detected at this size, which is not the same as saying the prompts are equal.
- Run
-
What twenty questions can't show
- Run
twenty-cannot-tell.py - Working rule Report the interval beside every comparison. Twenty items can find broken answers; they can't rank two prompts.
- Run
2. Read the replies to find what to measure
About 17 minutes to read, 2 examples.
By the end, students can read a set of replies from an assistant, sort what is wrong with each one into a failure mode, and say which of those failures a check can see.
Lecture outline: 6 sections
-
What the replies look like
- Run
read-replies-dev.py - Working rule Before you count anything, read replies from every variant, grouped by kind, and write down what is wrong with each one. A count can say that a variant fails. Only the replies say how.
- Run
-
Coding the failures
- Run
tally-failure-modes.py - Working rule Give each reply one note and one mode, and print one labelled line per count, so that every number the prose gives can be checked against the output.
- Run
-
Over-declining
- Working rule Count a decline by what the reply does, not by whether the phrase stands alone, and count over-declining apart from wrong answers, because their checks differ.
-
Wrong-version answers
- Working rule Check each reply's version in two places. Its citations must belong to the asked version, and its claim must match that version's reference. A citation check alone passes an answer that is right about the wrong version.
-
Missed refusals
- Working rule Require a decline on every unanswerable item, and read every answer to one against the section it cites before you count it as right.
-
From a failure to a check
- Working rule Code the replies by their main defect before you choose a check, and write each check for one mode. A check built from the failures you expected tests for those failures, and it passes the ones you didn't expect.
3. Build a test set you can defend, and read public benchmarks
About 18 minutes to read, 6 examples.
By the end, students can build a test set whose score tells you something, check its schema and coverage with code, and say what a public benchmark score does and does not tell you about your own assistant.
Lecture outline: 10 sections
-
What a test set is for
- Working rule Write the set before you change anything you will measure with it, and keep its file under version control, so that every comparison names the items it used. A set you edit after seeing the scores measures your edits, not the assistant.
-
The item schema
- Run
check-item-schema.py - Working rule Run the schema check in the build and treat any printed problem as a failure, and read a sample of items by hand before you trust the set.
- Run
-
Kinds and coverage
- Run
kinds-by-split.py - Run
behaviour-coverage.py - Working rule Count coverage for each behaviour before you quote a score for it, and treat a behaviour with only a few items as untested. Passing those items shows that one behaviour, and nothing about the rest of the tool.
- Run
-
Planted items and their traps
- Working rule Plant an item only for a trap you can name from the list. Write down the wrong reply it should catch, and check against the cited sections that the wrong reply is wrong, before the item enters the set.
-
Synthetic items and what they hide
- Working rule Ask each question the way a user would, from the docs but not in their sentences, and read each question against its sections before the set is frozen, since the copy check does not read questions. For a behaviour that changed between versions, name the version in the question and write its twin.
-
Dev and held-out
- Working rule Tune only on dev, freeze every setting before you open held-out, and run held-out once for each comparison. A held-out score that you read before a change, and then lifted by changing something, is a dev score, and you should say so.
-
Public benchmarks
- Working rule Quote a public score only beside its sample, model build, prompt, parser, reply cap and interval, and never beside a published number.
-
GSM8K
- Run
score-gsm8k.py - Working rule Count the replies with no final line before you quote a score, and compare two configurations only with a paired comparison that clears its interval.
- Run
-
TruthfulQA
- Run
score-truthfulqa.py - Run
score-truthfulqa-shuffled.py - Working rule Before you trust a multiple-choice score, print where the correct option sits, and set the share of replies that chose each position beside the chance share. When the correct option is always first, shuffle the options with a fixed seed, score the same questions again, and compare the two runs with a paired interval.
- Run
-
Contamination
- Working rule Name contamination beside every public score, and never present a benchmark score as the model's ability.
4. Write checks in code for the failures you found
About 18 minutes to read, 5 examples.
By the end, students can write the format and citation checks that run on every reply at no cost, say which of the failures from chapter 2 each one can and can't see, and show where two standard overlap scores rank a wrong answer above a right one.
Lecture outline: 9 sections
-
What a code check can see
- Working rule Write a code check for a failure only when the reply and the item already hold the fact the check needs. For a failure whose evidence is the claim itself, send it to a judge with a written rubric, as chapter 5 does, and keep the code check for the parts of the failure that can be read.
-
Format checks
- Run
format-problems-baseline.py - Run
format-problems-variants.py - Working rule Run the format check and the order check on every reply before a judge reads it, and report each name's count on each split. A zero counts as a result only when the check could have fired, so keep the fixture run with the report. When a count rises, read the replies behind it before you change the check, because a name says which shape broke and not why it broke.
- Run
- Citation checks
-
Existence and version
- Working rule Report the wrong-version and missing outcomes as their own rates, and never let a zero there stand for a right answer. For each version-dependent item, read the claim against the reference for the version asked. Grounded answers builds that reading into a rate of its own.
-
Support
- Working rule Count a citation as supported only when its ID is on the item's list, and never as correct. Read each unlisted citation against its section before you call it a problem. Report unlisted apart from wrong-version and missing, because a fuller list would move some of the unlisted citations to supported.
-
Exact and fuzzy match
- Run
fuzzy-match.py - Working rule Use an exact match wherever the expected text is fixed. Use a fuzzy ratio only to order replies for reading, never as a pass or a fail, until a recorded run shows where a threshold should sit for this book's answers. When you call
ratio(), pass the reference first every time, and say in the check's docstring which string is which.
- Run
-
Where BLEU and ROUGE mislead
- Run
overlap-misleads.py - Working rule Don't use BLEU or ROUGE as the pass mark for an answer with one right form. If you report them, name the formula and say whether it is smoothed. Read every answer whose overlap is high and whose claim is in doubt, since the pairs above show that overlap rewards a wrong answer that copies the reference. Check the claim with a judge, as chapter 5 describes.
- Run
-
Declines in code
- Working rule Count both decline errors on every split, and read each version-dependent decline against the docs for its version by hand, because the decline rate leaves those items out.
- Running the checks, and what they settle
5. Write a rubric a judge can apply
About 17 minutes to read, 3 examples.
By the end, students can write a rubric that a judge model applies the same way to each reply, read its verdict with code that never counts anything it cannot read as a verdict, and say when a pairwise comparison or a scale is worth its cost.
Lecture outline: 6 sections
-
What a judge is asked
- Run
rubric-text.py - Working rule Write each rubric as text that a team can read, and hash it with the run it produces. Name it by its first line, so a reader can tell which rubric a number came from. Change a rubric's text only by making another rubric with its own recording, and never by editing the rubric that produced the numbers already on the page.
- Run
-
Reference-guided or reference-free
- Working rule Use a reference whenever the set has one, and write the rule for an empty reference into the rubric, as the answer rubric does. Use a reference-free rubric only when no reference exists. In its text, name what the judge must not count against a reply, such as length, tone or incompleteness, so the rubric makes that choice and the judge does not.
-
Pass, fail, or a scale
- Working rule Use pass and fail when the team needs a yes or no on correctness, and report only rates of such decisions. Use a scale only when each step means something the team will act on, and when human ratings exist at each step to check the judge against. Say in the text which form the team chose and why, because that choice sets every rate the team reports. Chapter 6 shows how the judge is compared with human ratings.
-
One answer or two
- Run
pairwise-both-orders.py - Working rule Judge every pair in both orders, map each verdict back to its response before you count, and report the order-changed count beside the winner count. Count an order-changed pair as no preference, and do not let it decide a comparison. If the order-changed count is high on a set, the comparison is measuring position, and a larger sample in one order does not fix that.
- Run
-
Verdict first
- Run
parse-verdicts.py - Working rule Read the first non-empty line only. Never map an unreadable verdict to PASS or FAIL. Say in each rate what the run does with an unreadable verdict, whether it stops or leaves the reply out, and print the count beside any rate that leaves replies out. Check the parser on the replies the judge actually wrote, since a rubric that works on a sample can still lose verdicts to a format it did not ask for.
- Run
-
Tuning the wording on dev
- Working rule Write the wording down before you run anything on held-out, and tune it on dev only. When the wording has to change, make it another rubric with its own recording and hash. Report each number with the hash of the wording that made it, so two wordings cannot be read as one.
6. Measure a judge's errors against human ratings, then correct a pass rate
About 20 minutes to read, 6 examples.
By the end, students can measure how a judge's verdicts differ from human ratings, split those differences into misses and false passes, and correct an observed pass rate for them, with the assumption the correction rests on written beside it.
Lecture outline: 7 sections
-
Human ratings, and what they are
- Working rule Calibrate against labels made for the question the judge answers, or say in the text how far they are from it. Write down the rule that turns a score into the judge's pass or fail before you look at any result. Keep the split you tune on apart from the one you report once.
-
Agreement and kappa
- Run
calibration-agreement.py - Run
rubric-b-agreement.py - Working rule Report agreement beside kappa, and read kappa against the chance figure printed next to it. Name the rubric and the split with every number, so two wordings are never read as one. Call one wording better only when the lower bound of its paired interval is above zero, and only on held-out, or on dev with that caveat beside the claim. When the interval spans zero, say that no difference was detected at the n you have, and do not turn a higher point estimate into a claim.
- Run
-
True positive and true negative rates
- Run
tpr-tnr.py - Working rule Report TPR and TNR together, with their counts, and say which way the judge errs. Take the rates a correction uses from dev alone, and quote both beside any corrected rate. Report held-out once, after the choices are made, and keep it out of every choice.
- Run
-
Correcting a pass rate
- Run
corrected-pass-rate.py - Working rule Correct a pass rate only with TPR and TNR measured on a calibration set. Quote both beside the corrected rate, refuse when the denominator is small, and say in the same paragraph what the correction assumes. The next section gives the interval and the one-row moves that go beside any corrected rate.
- Run
-
How much one row moves the corrected rate
- Run
correction-sensitivity.py - Working rule Give the corrected rate with the bootstrap interval the book uses for its rates, the share of resamples at the cap, and the one-row moves beside it, so a reader can see how far single rows matter.
- Run
-
The transfer assumption
- Working rule Write the transfer assumption beside every corrected rate, in the same paragraph, and name the difference between the calibration judge and the judge the rate describes. Check the arithmetic on the rows the rates came from, and say that such a check does not test transfer. Until human labels on the answers the rate describes exist and have been checked against the judge, call the corrected rate an estimate that holds if the assumption does.
-
Pairwise calibration
- Ask first Before you run the example, predict whether a pair that counts only when both orders pick the human's response will score above or below either order taken alone.
- Run
pairwise-agreement.py - Working rule When a judge compares two replies, run it in both orders, and report agreement counted both ways with the order-changed pairs beside it. Never read a single-order rate as the judge's agreement with people. Report the interval over pairs, and say that an order-changed pair is a miss, because it is the cost of position.
7. Find where a judge's verdict stops being reliable
About 16 minutes to read, 3 examples.
By the end, students can test a judge for two biases that this book's data can show, position and length.
Lecture outline: 4 sections
-
Position
- Run
position-flips.py - Working rule Judge every pairwise comparison in both orders before you read its verdict, and report the order-change rate beside the verdict counts. Count an order-changed pair as no verdict for either reply, not as a win for whichever reply came first or second. When the order changes lean one way, say which way, how many, and with what interval, and don't assume that another judge leans the same way.
- Run
-
Length
- Run
length-and-verdicts.py - Working rule Don't name a length effect, or its absence, from a few bands. Compare the longest band with the shortest, with the gap's interval and each band's count. When the interval includes zero, say "no difference detected" with those counts, and stop there. If length could matter to your product, build a test in which length changes and the content stays the same, and measure that.
- Run
-
Self-preference, which this book cannot test
- Working rule When the judge and the assistant are the same model, report that beside the pass rate, and don't report self-preference as tested. Don't use the judge to compare its own model with another, unless the check above is in place: answers of matched quality, produced or attributed to both models, read by the same judge. Chapter 6 sets up the human ratings that such a check needs.
-
Where to stop trusting a verdict
- Run
verdict-limits.py - Working rule Stop trusting a verdict, and say so in the report, when it changes with the order or when it can't be read. When the judge is the model whose answers it grades, report that beside the rate, and don't use the judge to compare its own model with another, as the self-preference section sets out. Before a pass rate is published, check the judge against human ratings, as Chapter 6 does, and print the order-change rate, the unreadable count and the same-model fact beside the rate. A pass rate describes the answers only as far as the judge agrees with people, so the rate should always travel with its calibration.
- Run
8. Decide whether a change is better, and how many questions you need
About 18 minutes to read, 5 examples.
By the end, students can say whether a difference between two versions of the assistant is more than noise, using an interval on a paired comparison.
Lecture outline: 6 sections
-
Variance within one item
- Run
flip-rate.py - Working rule Before you trust one run's rate, run the variance set at the temperature you ship, and read the flip rate beside the mean pass rate. The flip rate says how far sampling moves one prompt's verdicts. It bounds a comparison only when that comparison samples its replies too.
- Run
-
The bootstrap over items
- Working rule Resample items, never runs within an item, and keep an item's samples together. Fix the resample count and the seed in the code, so the same data gives the same interval. When the set holds few items, call the interval rough and say so beside it.
-
Paired comparisons
- Run
paired-prompt-b.py - Working rule Compare variants on the same items, paired, and report the difference with its interval and n. Call one variant better only when the whole interval sits above zero, and worse only when the whole interval sits below it. When the interval includes zero, report no difference detected at that n, and say how many items the difference would need, as How many questions a difference needs works out.
- Run
-
Held-out, once
- Run
held-out-variants.py - Working rule Fix the variants, the level and the reading rule before you open the held-out set, and run each comparison once. An interval that reaches zero is no difference detected at that n. Don't rerun the comparison, lower the level or change the reading to get a different answer. Add held-out items only before they are scored.
- Run
-
Many comparisons
- Working rule Count the comparisons of one metric before you run them, and set each interval's level at 1 minus 0.05 divided by that count. Show the level beside every interval. A comparison added after the results is reported apart as exploratory, and the family's intervals are not set again.
-
How many questions a difference needs
- Run
sample-size.py - Run
sample-size-limits.py - Working rule Before a comparison, run a paired comparison on dev to estimate how often the two variants disagree, and state that share as the assumption. Compute the items needed for the difference you care about, at the level the comparison will be read at, and give its range from the interval on that share. If the set is smaller, report that it can't detect a difference that size, and don't read its no difference detected as a ruling against the change. Use the floor only as a lower bound, and only for a comparison that samples its replies.
- Run
9. Check that answers rest on the cited docs, in the asked version
About 18 minutes to read, 6 examples.
By the end, students can count whether an answer's citations support what it says, whether every claim in it rests on the sections the assistant was given, whether it is right for the version the reader asked about, and whether retrieval reached the right section on the way.
Lecture outline: 8 sections
-
What grounded means here
- Working rule Give each check its own count over its own denominator, and treat a pass as the start of the reading, not the end of it.
-
Citation accuracy
- Run
citation-accuracy-counts.py - Working rule Read the item rate and the citation rate together, and read each unlisted citation before you change a prompt. An unlisted citation is a question for the docs, not an error to count.
- Run
-
Faithfulness
- Run
faithfulness-rate.py - Working rule Put "uncalibrated" beside every faithfulness rate, and read the first unfaithful items before you trust the rate. A faithfulness check shows where the judge says a claim is unsupported. It does not show that the judge is right.
- Run
-
Version-correctness
- Run
version-correct-answers.py - Run
version-check-detail.py - Working rule Read each citation's ID with the version of the answer beside it. A zero from this check means no flagged ID, not no wrong-version citation. Keep version-dependent items in their own count beside the pass rate, so a version miss shows up on its own line.
- Run
- Outcome and path
-
The retrieval step
- Run
retrieval-path.py - Run
fail-with-section.py - Working rule Treat retrieval as a gate on passes and as a lead on fails. A pass without the supporting section needs an explanation, and a fail with it points at the reply.
- Run
-
Where agent harnesses picks up
- Working rule In this book the path is one retrieval and one reply, so read the retrieval count beside the verdict for each item.
-
Retrieval quality belongs to another book
- Working rule Before you call a reply grounded, read its citation, its claims and its version, and read the retrieval that came before it.
10. Decline when the docs don't say, and measure both kinds of error
About 20 minutes to read, 3 examples.
By the end, students can count both ways an assistant fails to answer from its docs, read the two rates over their own denominators, and say when a rule that declines more often is worth what it costs.
Lecture outline: 6 sections
-
Two errors, both counted
- Working rule Report the two decline rates apart, each over its own denominator, and leave version-dependent and ambiguous items out of both. Read the items behind each rate, not only the rate, because a rate can hide the few items that matter.
-
What the baseline does
- Run
decline-rates.py - Working rule Read each false refusal on the dev split by hand before you change a prompt, since one of them may point at a real gap in the docs. Compare variants on the same items, and report a paired interval only where an example prints one. Where none is printed, report the counts on these items and draw no conclusion about which variant is better. Run the held-out split once, to confirm a choice made on dev. The unfiltered variant is the warning here: on dev it answers unanswerable items that the baseline declines. This chapter reports that without an interval, so it is a reason to check, not a measured rise.
- Run
-
A guard on recorded replies
- Run
tune-guard-dev.py - Working rule Build a guard's scores from the same index that made the replies, and check that match before you read any rate. Then read the whole sweep with both rates on every line. A pick that ties with doing nothing says the rule found no reason to change a reply, and that is an answer worth keeping.
- Run
-
Too many refusals and too few
- Working rule Before you tune a threshold, write the cost of each error in the team's own terms, and give the rule those weights before anyone reads the sweep.
-
What a reader should accept
- Run
guard-held-out.py - Working rule A guard earns its place only when its held-out run changes replies in the direction its dev sweep promised. When it doesn't, keep the result, since it shows which errors the assistant has and that a threshold on the retrieval score does not fix them.
- Run
-
Outside this book
- Working rule Before adopting a decline rule because a paper reports that models over-refuse, measure the rule on your own assistant and your own items, with both kinds of error counted. Keep the paper's numbers out of your results, and cite the paper for the idea that prompted the measurement.
11. Put the evaluation in CI, and compare variants on traffic
About 16 minutes to read, 4 examples.
By the end, students can put the evaluation into a gate that blocks a change only when the change is measurably worse.
Lecture outline: 5 sections
-
A gate that respects the interval
- Run
gate-real-drop.py - Working rule Report every paired difference with its interval and its n, and let the decision follow from the upper bound. Don't call a change better or worse from a point estimate, and don't read a pass with a wide interval as proof that the candidate matches the baseline.
- Run
-
Flag flaky items, never edit them
- Run
flaky-items.py - Working rule Flag an item whose verdicts differ across its samples, print its count beside the flag, and leave the item as it is. Report the flip rate with its denominator, and say that a low rate holds for this assistant, this judge and this set only.
- Run
-
The evaluation's own tests
- Run
own-tests-run.py - Working rule Run the evaluation's own tests before any run that reads a number, and treat exit code 5 as a failure, not a pass. A failing test in the scoring code means the numbers are not yet trustworthy, whatever they say.
- Run
-
Running it as a framework task
- Working rule When you map an evaluation onto a framework, name the framework's version and the date of its documentation. Keep the test set, the checks and the replay model as the files the book's examples run, and treat the map as a reading of the documentation until someone runs it.
-
Variants on traffic
- Run
feedback-table.py - Working rule Don't compare variants from traffic without random assignment and the variant recorded beside each vote or reply. Read a vote table as a share of the votes cast, with its count, and never as a measure of correctness.
- Run
Quiz bank
Questions on every chapter, with answer keys, for verified educators with the Educator plan. Each answer is tied to its section. The Educator plan
Loading the quiz bank…