Teaching kit
From Edition 2026.11.2, the book's current edition. It changes when the book does. Use and adapt it freely, with credit to the book.
On this page
Syllabus
Reading times are for the chapter's text. Students also run its examples, in the Lab or on their own machine.
A chapter a week: 12 weeks
In 5 weeks
Objectives and outlines
Each chapter's objectives are its first sentence. Each outline follows the chapter's sections, with the terms it defines, the predictions it asks for before an example runs, and the rule each section ends on.
1. Give a model tools and a loop
About 17 minutes to read, 4 examples.
By the end, students can run a model in a loop that gives it three tools:
- one reads a file
- one runs the tests
- one finishes with a changed file
Lecture outline: 8 sections
-
What the agent does on one task
- Run
read-the-issue.py - Working rule Score a run only against its own copy and its task's check. Never score it against the case itself, because a change that reached the case would count for a run that never earned it.
- Run
-
A loop in about seventy lines
- Working rule Keep every request and its result in the messages, and give each result the ID of the request it answers. A loop that drops a result leaves the model without an answer to its request, and the messages are where that gap shows.
-
Two tools and one stop tool
- Run
tool-refusal.py - Working rule Give each tool one job, and let only the stop tool end a run. A refused read or write should come back to the model as an error it can act on, and it should never stop the loop.
- Run
-
Stop when it's finished, or when the budget runs out
- Working rule Set the step budget before the run starts, and treat it as a hard limit. A run that reaches the budget did not finish, whatever its last step looked like, and it should be reported as unfinished.
-
Run it on a scripted model
- Run
scripted-loop.py - Working rule Run the loop on a scripted model before you run it on any model. If the scripted run fails, the fault is in the harness, and a model run would only hide it. If the scripted run passes and the model run fails, the fault is in the model or the prompt.
- Run
-
Run it on the recorded model
- Run
recorded-run.py - Working rule Replay a recording the same way on every run, and never edit it to make a run pass. A failing recording shows what the model really did. The fix belongs in the harness, the prompt or the model choice, and the result of that fix is a new recording.
- Run
-
What goes wrong, and what is counted
- Working rule Count steps and tool calls the same way on every run, so that two runs can be compared. When a run ends at its budget, read its error lines before you change the prompt or the model. The error lines show whether the model is repeating one mistake, and that tells you which part of the harness to look at.
-
Decisions in this chapter
- Working rule Before you change any of these decisions, work out what the change does to the numbers this chapter prints. A new step budget changes the step counts, a new stop tool means a new recording, and a sandbox changes what the tools can reach. Make the change in its own chapter's terms, and rerun the scripted and recorded examples before you trust the result.
2. Stop the loop when it's done or stuck
About 17 minutes to read, 4 examples.
By the end, students can say why any run stopped, from the one word the harness records for it, and add the two rules that end a run which has stopped making progress.
Lecture outline: 6 sections
-
Ways a run ends
- Working rule Record one
endedvalue with every run, and never report a run's result without it. Count a run as done by the rule in the score, never by itsendedvalue alone.
- Working rule Record one
-
The budget counts steps
- Run
budget-stops-a-loop.py - Working rule Set the step budget from runs on your own tasks, not from a guess. Take the most steps a passing run needed, leave room above it, and write in the code what that room is for.
- Run
-
The same call, again
- Run
repeated-call-stops.py - Working rule Compare calls by tool name and sorted arguments, and end the run at three identical calls in a row. Raise the threshold only after your own traces show three identical calls in a row that were making progress. Then measure the higher number on your task set before you keep it.
- Run
-
No progress across edits
- Run
no-progress-stops.py - Working rule Watch the counts the harness gets from the tests after each edit, not what the model says it changed. Stop a run after two edits in a row that leave the counts unchanged. Record it as
no_progress, and keep the counts it stopped on beside the run.
- Run
-
Asking, stopping and counting the endings
- Run
stop-rules-over-task-set.py - Working rule Count each run by the
endedvalue it recorded, and report the counts with the rules on and off, next to the done count. A stop can only turn a run into one that is not done, so when you change a rule, check that the done count holds before you keep the change.
- Run
- Decisions in this chapter
3. Give the model tools it can't misuse
About 18 minutes to read, 5 examples.
By the end, students can give a model tools whose arguments the harness checks before any tool runs, and write refusals the model can act on.
Lecture outline: 6 sections
-
A schema says what is allowed
- Run
schema-from-model.py - Run
tool-descriptions.py - Working rule Write each bound, type and default in the argument model, where the harness checks it, and give each field a description the model can read. Keep the tool's sentence to what the tool is for. A rule that lives only in the sentence is a hint the harness will never enforce.
- Run
-
Lax by default, strict on purpose
- Run
lax-and-strict.py - Working rule Use lax mode where a conversion is what the model meant, and strict mode where a quiet conversion would change what the tool does. Test each model with a value of the wrong type before you rely on it, and read the error type the refusal names, since that is what the model will see.
- Run
-
Errors the model can read
- Run
bad-arguments-handed-back.py - Working rule Return each refusal as a tool result that names the field, the error type and the message. Never end the run on a bad call, except through the stop rules in chapter 2, when they are on and the same bad call keeps coming back. A call the parser cannot read gets the parser's reason as its refusal, and no tool runs on it.
- Run
-
One name per tool
- Working rule Take the allowed names from the tools the loop actually runs, never from a list kept beside them, and put them in every refusal for an unknown name. A name the set lacks gets a refusal, and the prompt does not promise it. Accept an alias only through an explicit alias map that the harness keeps, never by a guess.
-
Edit errors that name the nearest lines
- Run
edit-error-nearest-lines.py - Working rule When an edit's text is not in the file, return the file's nearest lines with their numbers, and say what to do next. Do not return only that the text is missing, since that leaves the model to guess which quote is wrong. A file short enough to return whole can be returned whole, numbered, in its place.
- Run
- Decisions in this chapter
4. One interface for scripted, replayed, local and hosted models
About 17 minutes to read, 4 examples.
By the end, students can run the same loop on a scripted model, a replayed run, a local model and an OpenAI-compatible endpoint, without changing the loop.
Lecture outline: 5 sections
-
Ask every model the same question
- Working rule Test each adapter by calling
reply()with the messages and tool schemas the loop sends, and check the fields of theReplythat come back. Useisinstanceonly to see that a method exists, never to show that its signature is right.
- Working rule Test each adapter by calling
-
Script the replies, or replay a stored run
- Run
same-loop-two-models.py - Run
replay-byte-for-byte.py - Working rule Replay a recorded run only with the prompt and the check it was recorded with. When either changes, its hash in the manifest no longer matches, the example fails, and the run has to be recorded again. Changing a stored reply to make the example pass hides the change that caused the failure.
- Run
-
A local model, and why its settings are fixed
- Working rule Record with the sampling the manifest names, and keep the model file's hash in the manifest beside it. If you change a setting, the replies change with it, so record the run again rather than compare it with the old one.
-
A hosted-style endpoint, tested on loopback
- Run
openai-compatible-loopback.py - Run
adapter-refuses-unsafe-endpoints.py - Working rule Parse
function.argumentsas JSON, check the result against the tool's argument model, and send a failure back to the model as a tool result. An endpoint that enforces a strict schema makes such a failure rarer, but the check still runs.
- Run
-
One loop, several models
- Working rule An adapter keeps the raw text and the counts, reads the tool calls in one place, and returns a
Reply. It never hands the loop an SDK object. Give each adapter a test that callsreply()with the loop's arguments, and treat a changed prompt or a changed request as a reason to record again.
- Working rule An adapter keeps the raw text and the counts, reads the tool calls in one place, and returns a
5. Keep the context useful as the task grows
About 17 minutes to read, 5 examples.
By the end, students can work out how much room a run's prompt has, estimate a prompt's size when the harness cannot count it exactly, and compact a run's older steps without cutting a tool call away from its result.
Lecture outline: 10 sections
-
The window the model has
- Run
budget-counts-tokens.py - Working rule Take the window from the load setting the run used, as the manifest records it, and subtract the reply's limit before you set any budget. Never take it from a model's advertised length or a vendor's default, since a budget copied from a larger model lets a run fill the window before the budget acts.
- Run
-
Where the recorded runs came closest to the budget
- Working rule Compare every step's count with the prompt budget before you read anything into a run.
-
Estimate the prompt when the harness can't count it
- Run
estimate-vs-recorded.py - Working rule Estimate with three characters per token, and check the estimate against the counts the model reports on the first runs of any new model. If the estimate ever falls under the reported count, use a smaller number of characters per token before you trust the budget.
- Run
-
Put the last result at the end
- Working rule Put the issue and the notes first and the last tool result last. Treat that as a design choice, and claim a gain only once a run on your own tasks shows one.
-
Compact the older steps, and keep the last pair
- Run
compact-keeps-last-result.py - Working rule Keep the last call and its result together, refuse any cut that falls on a tool result, and check that the fixed part fits the budget before you rely on compaction to make the rest fit.
- Run
-
Keep notes outside the context
- Run
notes-survive.py - Working rule Write a fact that a later step needs into a note, not only into the summary. A summary keeps a selection, and the notes are the one place the book keeps a fact whole.
- Run
-
What the recorded model did with the budget
- Run
recorded-context-use.py - Working rule Before you judge a budget or a tool by a recording, check how each run ended. A run that stops early never tests the budget, and a model that never calls a tool never tests the tool.
- Run
-
Measure the effect instead of quoting it
- Working rule Measure how a budget changes your own task set, with the counts the runs report, before you claim that it helps. Quote a published effect only with its setup and its source, and only as that report's result.
-
How a budget goes wrong, and how to notice
- Working rule Log the estimate and the reported count on every step of a real run, and treat an estimate under the count as a fault. When a run asks for a file that its own summary already covers, check what the summary kept before the run reads the file again.
-
Decisions this chapter made
- Working rule Name the runs a budget or a tool was tested on. A claim about a budget that no run reached is a claim about the design, not about the model.
6. Gate the tools that change things
About 28 minutes to read, 12 examples.
By the end, students can keep an agent's changes in its own copy until a person says yes, check that the yes covers the exact change they read, and hand a no back to the agent as the reason its run stopped.
Lecture outline: 12 sections
-
The copy comes first
- Run
apply-all-or-nothing.py - Working rule Give each run its own copy, refuse every path that resolves outside it, and let exactly one function write to the original, called only after a yes. Then name what the run's code can still do to the original, check that with a guard, and do not count the gate as the protection.
- Run
-
Which mode this harness is in
- Working rule Name the mode before you add a gate. When the caller has a callback that can answer, make the gate call it and wait for the answer. A gate built with no approver refuses to start: this chapter's code raises an error. A design that hands off when nothing can answer needs its own path, and no example in this chapter shows one.
-
The gate sits in finish
- Run
gate-applies-only-on-yes.py - Run
finish-body-waits-for-yes.py - Working rule Propose the copy's whole diff from the one call that can change the library, and hash that diff again immediately before the apply. A copy that moved after the person looked is refused, whatever the answer says, because the apply writes the bytes that were hashed. Run a tool's own body only after a yes.
- Run
-
An approval answers one request
- Run
hidden-diff-lines-shown.py - Run
answer-binds-bytes.py - Run
approval-in-transcript.py - Working rule Give every proposal an ID and the hash of its bytes, require the answer to name both, and check both again before you apply. Show the person the text with every control character written out. Log each answer as the pair, next to its proposal, so a reader can match a yes to the change it applied.
- Run
-
Some changes are refused without a question
- Run
never-class-refused.py - Run
gutted-test-refused.py - Working rule Refuse a change that the project forbids by reading its diff against a list the harness owns, and name the rule it broke in the handoff. Keep the list short, and say what it does not check, because a check on names does not protect what a function does. A yes cannot make a forbidden change allowed, so do not ask for one.
- Run
-
Refusals go back to the model
- Run
refusal-handed-back.py - Working rule Return a no to the model as a tool result that carries the person's reason, and end the run as a handoff whose question is the same text. Do not swallow the no, and do not raise it as an exception the loop has to catch.
- Run
-
The wrapper checks every call
- Run
library-guard-fails-closed.py - Run
unknown-tool-refused.py - Working rule Class every tool in the table, and refuse any name the table does not list before its body runs.
- Run
-
Allow lists do not skip the check
- Working rule Do not let a name, a description or a flag from a tool or its server decide whether the check runs. Put the check on the path that every change the tools make has to take. Code that the tests run is outside that path, as the copy section says, and the flag describes the tool and nothing more.
-
Across the task set
- Run
gate-over-task-set.py - Working rule Before you enforce a gate, replay the recorded runs through it without enforcement, and compare each ending and each count with what you expected. A count that surprises you is a finding about the gate or about the runs, and you settle it before anyone relies on the gate.
- Run
-
Vendor defaults that would let a change through
- Working rule When a vendor's default lets an answer, a flag or a hook stand in for a check, do not copy the default. Write the check so that it names what it approves: the proposal, its bytes and the rule it applies. Then test that a wrong answer, a moved copy and a claimed approval each fail.
-
What this gate does not cover
- Working rule Write each limit beside the code it comes from, and name it once, in one place, so a reader can see what the gate does not do before they rely on it.
-
Decisions this chapter made
- Working rule Write each decision a gate makes down with the example that shows it, and list what the gate leaves to its caller. A decision with no example behind it is a claim, and the chapter labels it as one.
7. Run the agent's code in a sandbox, and keep secrets out of its view
About 17 minutes to read, 7 examples.
By the end, students can run a command for the agent with a time limit, per-process memory, CPU and file-size caps, an environment built from an allowlist, and a scratch copy as its working folder.
Lecture outline: 5 sections
-
A teaching sandbox, not a container
- Working rule Run code an agent wrote inside a container or a virtual machine, with this sandbox as a second layer inside it. Where neither is available, do not run that code on a machine you need to keep intact. The sandbox is teaching code, and the table above lists what it leaves open.
-
Only listed programs
- Run
argv-allowlist.py - Run
injected-text-refused.py - Run
same-effect-through-a-test-file.py - Working rule List the exact argv of every program the agent may start, parse each command once, and compare the parsed list with that list. Never pass the model's string to a shell, and never decide by searching the string for a word. Then fix what each listed program loads as well: its configuration and plugins come from outside the copy, and conftest files are off. Treat any test module as code that runs with the harness's rights.
- Run
-
Limits, and the grandchild they miss
- Run
limits-stop-a-hog.py - Run
escaped-child.py - Working rule Cap memory, CPU time and file size on every command, and kill its whole process group on the time limit and on every other exit. Then count what the command can start. Each process has its own caps, so a cap on the total needs a cgroup or a container.
- Run
-
A network namespace, where the kernel allows one
- Run
network-label.py - Working rule Where a command must not reach the network, require the namespace and refuse to run without it. Check the claim with a listener the command must fail to reach, not with the label alone.
- Run
-
Secrets stay out of the environment
- Run
secret-not-in-env.py - Working rule Build a command's environment from the names it needs, never by copying the harness's. Keep secrets out of the harness's arguments and out of the copy's files. Treat the harness's environment as readable by any command until a PID namespace is in place.
- Run
8. Hand a subtask to a child, and hand back a stop
About 19 minutes to read, 4 examples.
By the end, students can give a subtask to a child loop with a step budget of its own, and get back a summary and a stop reason in place of the child's whole transcript.
Lecture outline: 7 sections
-
A child is a loop with its own budget
- Run
child-with-budget.py - Run
parent-starts-child.py - Working rule Give each child a budget of its own, counted in its own model calls and never shared with the parent. Set it from runs that finished on the task set, not from a guess. A budget below what finished runs need turns finished work into stops.
- Run
-
A brief with four parts
- Working rule Write all four parts into a brief before a child starts, and write the boundaries as files and tools, not as intentions. A brief that can't say when the child is done, or which files it may edit, isn't ready to send.
-
What comes back
- Run
summary-to-parent.py - Working rule Return to the parent a child's ID, how it ended, its counts, its files and its stop or its summary, and nothing more. Keep the transcript in the child's own record, where a trace can read it.
- Run
-
A stop handed back
- Run
handoff-on-stuck.py - Working rule Hand every stuck stop (
step_cap,repeated_call,no_progress) to a person as a question, with its stop reason, its counts and its files. A parent works from the handoff. The child's record stays with the child, where a trace can read it.
- Run
-
Two designs, and whose numbers
- Working rule Use a child when its result can be used without its reasons: a fact, a list of files, or a stop with its question. Keep one loop when the parent must act on the files the child would edit. This book doesn't measure a split against one loop, so the rule rests on the argument above, not on a measurement.
-
What goes wrong in production
- Working rule Count a child's budget in model calls, give its parent only the means to ask, and check each split's edits against the files the other children changed before you merge them. Trace each child's tokens on its own record, so a split's cost is counted where it is spent.
- Decisions this chapter made
9. Connect the harness to MCP servers
About 19 minutes to read, 5 examples.
By the end, students can connect the harness to MCP servers that other people run, see each server's tools under names that cannot clash with the harness's own, and send every call to one of those tools through the approval gate. Chapter 6 built that gate and said that MCP tools would go through its approvals.
Lecture outline: 7 sections
-
The revision this book runs
- Working rule Take the revision from the first example's output, not from a constant or from memory. Before you point the harness at a server, check the revision on its line in that output. A server on an older revision is not covered by this chapter's examples, so test it on its own before you rely on the harness.
-
Listing a server's tools
- Run
list-server-tools.py - Working rule List every server when the run starts, and keep the list in the run's record. Chapter 11 traces the names offered in each record, but no MCP server is traced there. Offer the model only the names your table gives a class. When a server adds a tool in an update, treat the added tool as unknown until someone adds it to the table.
- Run
-
Names that cannot clash
- Run
namespace-clash.py - Working rule Prefix every server tool with its server's name, in a single module that the rest of the harness imports. Assert that the names the model sees are distinct, and check each prefixed name against the server it names, not only against the list.
- Run
-
Another party's tools need approval
- Run
annotation-is-a-hint.py - Working rule Give each server tool a class in the harness's table, by its harness name, and never let an annotation set that class. A tool the table does not name is not offered, and a call to it is refused. Read a server's hints when you review it, and let the table decide whether a call waits for a person.
- Run
-
Output is untrusted
- Run
fence-cannot-be-closed.py - Run
untrusted-output.py - Working rule Label every result from another party, fence it, and pass it to the model as data. Keep the stop rules, the allowed commands and the approvals out of anything the model reads. A call that a server's text asks for is an ordinary call, and a change waits for the gate like any other change.
- Run
-
What goes wrong in production
- Working rule Review each server before you connect it, keep its tools in your own table, and read the run's refusals after every server update. Review a server's code before the harness starts it, and run one you did not write in a container or a virtual machine.
- Decisions this chapter made
10. Test the agent, then measure it on a task set
About 18 minutes to read, 8 examples.
By the end, students can test the harness on scripted replies, score the agent on the task set with one rule for done, and state the count with its range.
Lecture outline: 7 sections
-
Scenario tests over a scripted model
- Run
scenario-refusal.py - Run
scenario-gate.py - Run
scenario-stop-rule.py - Run
scenario-injected-output.py - Working rule Name the replies a scenario test fixes, and say what it measures: the harness's handling of that sequence, not the model.
- Run
-
The hidden check runs in a copy
- Run
hidden-check-in-copy.py - Run
scoring-ignores-planted-config.py - Working rule A check lives in a fresh copy, and only that copy's result counts. A planted file in the agent's copy is something to review, never a setting the check obeys.
- Run
-
Score: tasks done out of 20
- Run
score-task-set.py - Working rule Write the done rule before the run, count a run as done only when the rule's end state holds, and never count a run that hit a limit.
- Run
-
T-18 counts through a path
- Working rule Check a stop task's count against its question, not only against its reason words.
-
How sure the count is
- Working rule A count and its range go together, in one sentence.
-
Two harness versions, compared as counts
- Run
compare-v1-v2.py - Working rule Compare two versions as counts, state the result whichever way it falls, and credit no single change in a bundle.
- Run
-
What the score does not say
- Working rule Report a score with its tasks, runs, model and setting, and say it is not a benchmark result.
11. Trace each run and count what it cost
About 17 minutes to read, 4 examples.
By the end, students can make every run write one record per step to a file that a later reader can parse, give each record IDs that tie it to its run and its call, and say what the run cost in tokens.
Lecture outline: 8 sections
-
One line per event
- Run
trace-one-run.py - Working rule A reader of a trace skips an empty line only on purpose, and it counts the lines it skipped. Otherwise it stops at the first empty line. The writer never adds a blank line between records.
- Run
-
IDs without clocks
- Working rule Number runs and calls from a counter passed in, never from a clock or a random generator. An ID that changes between runs changes the request, and a replay of that run stops at the first request it cannot match against the recording.
-
Tokens as counts
- Run
token-counts-from-recording.py - Working rule Store each count with the model file it came from, and with any setting that changes it, such as the beginning-of-sequence token. Do not compare counts from different tokenizers. A count that no check has re-derived is reported as a stored count, never as a live one.
- Run
-
A child's records sit under its parent's call
- Run
child-trace-under-parent.py - Working rule Give each child its own run ID and its own records, each with
parentset to the call that started it. When you report what a parent run cost, add its children's counts and say that you did so. This trace prints them apart and sums none of them into the parent.
- Run
-
Cost as counts, not money
- Run
cost-as-counts.py - Working rule Report a cost as the steps, tool calls and tokens of each run, grouped by how it ended, with the groups printed beside each other. Quote no money. Read a step-capped group as tokens spent without a finish, and not as evidence about the task until a second measure says otherwise.
- Run
-
Keep your own field names, and map them at the edge
- Working rule Name the trace's fields for what they count, in your own project. Map them to a standard's names at the edge, in one table, with the revision of the standard written beside the table. Check the table again when the standard's revision changes.
-
What goes wrong, and how you would notice
- Working rule Before you trust a trace for a decision, read its
resultandstopfields, then check its printed sums against the stored totals. A trace that does not match its totals has a bug in its writer, and the run it records is not yet a finding about the agent.
- Working rule Before you trust a trace for a decision, read its
-
What this chapter decided
- Working rule Keep one line per step, with its IDs and its stored counts, for every run you report. A figure that cannot be traced back to those lines is not reported.
12. When an agent is the wrong design
About 17 minutes to read, 2 examples.
By the end, students can take any task in the case, sort it into a fixed pipeline, a single model call or an agent with gates, and say what the recorded runs show for it.
Lecture outline: 6 sections
-
Start with the fixed pipeline
- Run
fixed-pipeline-same-task.py - Working rule Before you give a model the loop, write the steps out, with the check that follows each one. If every step is known and a check decides the result without a model, build the pipeline and measure it. Decide where its stop goes before the first run: to a person, or to an agent given the failed step.
- Run
-
When splitting the work helps
- Working rule Split a task only when you can name the condition that makes splitting help, such as a context the single agent cannot hold, and when each part has a check that does not depend on the other parts. Otherwise keep one agent, or none.
-
What goes wrong when you do use an agent
- Working rule Before you trust an agent's design, sort its stopped runs by how they ended and read the record of each kind. A count of failures without their kinds does not say which part to fix, and one passing run does not predict the next.
-
The check stays outside the agent
- Working rule Put the check that decides a task's result where the agent cannot edit it or see it before the run ends. If a finish or a stop depends on the agent's own account of its work, add a check the agent did not write.
-
The decision table
- Run
decision-table.py - Working rule Sort each task in this order before you choose a design: a refusal is a question to a person, not a design; a person's yes is an agent behind a gate; known steps with a check are a pipeline; known steps without a check are a single call whose output a person reads. Open steps are an agent, and then you count its stops by kind, as chapter 10 does, before you believe the design works.
- Run
-
Decisions this chapter made
- Working rule Treat a rule's pipeline as a prediction until a run has measured it, and write down where its stop goes before you build it.
Quiz bank
Questions on every chapter, with answer keys, for verified educators with the Educator plan. Each answer is tied to its section. The Educator plan
Loading the quiz bank…