Glossary
The 86 terms Agent Harnesses defines, each in the book's own words, with the section that defines it.
From Edition 2026.11.3, the book's current edition. Where a definition depends on a package, it says the version this edition pins and runs its examples on.
A
- action also actions
-
A pipeline has actions, which are its tool calls
- Contrasts with: step
Defined in Start with the fixed pipeline, chapter 12. In the book since Edition 2026.11.3. Permanent link to action
- agent also agents
-
an agent when an LLM dynamically directs its own processes and tool use
Defined in Start with the fixed pipeline, chapter 12. In the book since Edition 2026.11.3. Permanent link to agent
- agent harness also harness
-
The harness supplies the loop, the tools and the limits on what those tools may do
Defined in Give a model tools and a loop, chapter 1. In the book since Edition 2026.11.3. Permanent link to agent harness
- allow list also allow lists
-
A tool on an allow list runs without a question, and a gate placed at the question is then bypassed for that tool.
Defined in Allow lists do not skip the check, chapter 6. In the book since Edition 2026.11.3. Permanent link to allow list
- argument model also argument models
-
an argument model, a small pydantic class that lists the fields the tool takes
True for pydantic 2.14.3, as this edition pins it.
- Comes before: tool schema
- Implemented by: pydantic 2.14.3
Defined in A schema says what is allowed, chapter 3. In the book since Edition 2026.11.3. Permanent link to argument model
- argv allowlist also allowlist
-
The allowlist is the check a command tool would use: it parses a command string once with shlex.split, and compares the argv with a short list of exact argv tuples.
Defined in Only listed programs, chapter 7. In the book since Edition 2026.11.3. Permanent link to argv allowlist
B
- brief
-
The brief is the job as the child receives it.
Defined in A brief with four parts, chapter 8. In the book since Edition 2026.11.3. Permanent link to brief
C
- child also subagent, child run
-
A child is the loop from chapter 1, started by a parent in the middle of its own run.
Defined in A child is a loop with its own budget, chapter 8. In the book since Edition 2026.11.3. Permanent link to child
- child budget
-
The child's budget counts the child's own model calls, from its first call, so the parent's steps don't pay for the child's work, and a child that loops runs out of its own budget.
- Part of: child
Defined in A child is a loop with its own budget, chapter 8. In the book since Edition 2026.11.3. Permanent link to child budget
- class table also POLICY
-
The harness's class table, POLICY in lib/mcp_tools.py, gives each server tool a class, read or change, and only names that table gives a class appear in that line.
Defined in Listing a server's tools, chapter 9. In the book since Edition 2026.11.3. Permanent link to class table
- command also argv list
-
A command is an argv list, never a shell string, and the runner starts it with shell=False, so no shell reads its quotes or its semicolons (subprocess).
Defined in A teaching sandbox, not a container, chapter 7. In the book since Edition 2026.11.3. Permanent link to command
- completion count also completion tokens, completion_tokens
-
the completion count is the tokens it wrote
Defined in Tokens as counts, chapter 11. In the book since Edition 2026.11.3. Permanent link to completion count
- copy also scratch copy
-
Each run gets its own copy, made by scratch_copy in run_task, and the tools work through a Workspace rooted at that copy.
Defined in The copy comes first, chapter 6. In the book since Edition 2026.11.3. Permanent link to copy
- cost
-
a run's cost is three numbers: its steps, its tool calls and its tokens
Defined in Cost as counts, not money, chapter 11. In the book since Edition 2026.11.3. Permanent link to cost
D
- decision table
-
The decision table sorts each task by three readings: whether its steps are known before the run, whether a check can decide the result without a person, and whether a person has to say yes.
Defined in The decision table, chapter 12. In the book since Edition 2026.11.3. Permanent link to decision table
- done (fix or implement task) also done
-
A fix or an implement task is done when the run ended with finish, its hidden check passes on the final copy, and the task's protected paths match the case byte for byte.
- Measured from: hidden check
Defined in Score: tasks done out of 20, chapter 10. In the book since Edition 2026.11.3. Permanent link to done (fix or implement task)
- done (stop task)
-
A stop task is done when the run ended with ask_user, its question contains one of the task's reason words, and no file in the copy differs from the case.
Defined in Score: tasks done out of 20, chapter 10. In the book since Edition 2026.11.3. Permanent link to done (stop task)
E
- edit also edits
-
An edit names the text it replaces.
Defined in Edit errors that name the nearest lines, chapter 3. In the book since Edition 2026.11.3. Permanent link to edit
- ended also stop reason, ended value
-
Every run ends for one stated reason, and the harness writes that reason into the run's ended field.
Defined in Ways a run ends, chapter 2. In the book since Edition 2026.11.3. Permanent link to ended
- estimate also prompt estimate
-
The estimate counts the characters of every message and of the tool schemas, and divides by three, rounding up: one token for every 3 characters.
- Contrasts with: prompt count
Defined in Estimate the prompt when the harness can't count it, chapter 5. In the book since Edition 2026.11.3. Permanent link to estimate
F
- fence
-
The fence is an opening tag that names the tool, and a closing tag.
Defined in Output is untrusted, chapter 9. In the book since Edition 2026.11.3. Permanent link to fence
- finish reason also finish_reason
-
the finish reason, which says whether the model stopped on its own or ran out of tokens
Defined in Ask every model the same question, chapter 4. In the book since Edition 2026.11.3. Permanent link to finish reason
- fixed part
-
The fixed part, the system prompt, the tool schemas and the issue
Defined in Compact the older steps, and keep the last pair, chapter 5. In the book since Edition 2026.11.3. Permanent link to fixed part
- fixed pipeline also pipeline
-
A fixed pipeline is a task's list of steps, written as code before the run starts.
- Contrasts with: agent
Defined in Start with the fixed pipeline, chapter 12. In the book since Edition 2026.11.3. Permanent link to fixed pipeline
G
- GGUF also GGUF file
-
a GGUF file, the single-file format llama.cpp reads
Defined in A local model, and why its settings are fixed, chapter 4. In the book since Edition 2026.11.3. Permanent link to GGUF
H
- handoff also handoffs
-
a handoff is a run's end with a question back to a person, instead of a finish
Defined in Hand a subtask to a child, and hand back a stop, chapter 8. In the book since Edition 2026.11.3. Permanent link to handoff
- harness v1
-
Harness v1 is the loop of chapter 1 with tool set v1 from chapter 3, a step cap of 16 steps, and no stop rule beyond it.
Defined in Two harness versions, compared as counts, chapter 10. In the book since Edition 2026.11.3. Permanent link to harness v1
- harness v2
-
Harness v2 is the configuration of chapter 5: it runs tool set v2, with strict argument models, typed error results and edit errors that name the nearest lines of the file.
- Contrasts with: harness v1
Defined in Two harness versions, compared as counts, chapter 10. In the book since Edition 2026.11.3. Permanent link to harness v2
- headless run also headless run with a callback
-
This harness has no terminal and no session: it is a headless run, and the only party who can answer is a callback that the caller passes in.
Defined in Which mode this harness is in, chapter 6. In the book since Edition 2026.11.3. Permanent link to headless run
- hybrid
-
The hybrid, a pipeline first and an agent only on a pipeline_stop
Defined in Start with the fixed pipeline, chapter 12. In the book since Edition 2026.11.3. Permanent link to hybrid
I
- identical call also identical calls
-
A call is identical when it has the same tool name and the same arguments, compared with the keys sorted.
- Comes before: repeated call
Defined in The same call, again, chapter 2. In the book since Edition 2026.11.3. Permanent link to identical call
- indirect prompt injection also prompt injection
-
The threat is indirect prompt injection: text that an application retrieves from outside can carry instructions, and a model that reads that text may follow them.
Defined in Output is untrusted, chapter 9. In the book since Edition 2026.11.3. Permanent link to indirect prompt injection
L
- lax mode
-
In pydantic's default, lax mode, a value is converted when the conversion is clear and refused when it is not.
True for pydantic 2.14.3, as this edition pins it.
- Implemented by: pydantic 2.14.3
Defined in Lax by default, strict on purpose, chapter 3. In the book since Edition 2026.11.3. Permanent link to lax mode
- library guard also guard
-
a guard that compares the library's bytes around every tool call
Defined in Gate the tools that change things, chapter 6. In the book since Edition 2026.11.3. Permanent link to library guard
- library_changed
-
when the library has changed, the run ends as library_changed, the harness applies nothing, and it restores nothing
- A kind of: ended
Defined in Gate the tools that change things, chapter 6. In the book since Edition 2026.11.3. Permanent link to library_changed
- local compaction also compaction
-
Compaction here is local: the harness writes the summary itself, and the model never sees it being written.
Defined in Compact the older steps, and keep the last pair, chapter 5. In the book since Edition 2026.11.3. Permanent link to local compaction
M
- malformed call also malformed calls
-
The harness never runs a block that doesn't parse: it records it as a malformed call and hands the reason back to the model as an error result, so the model can try again.
Defined in A loop in about seventy lines, chapter 1. In the book since Edition 2026.11.3. Permanent link to malformed call
- max_tokens
-
max_tokens, the most the reply may take
Defined in The window the model has, chapter 5. In the book since Edition 2026.11.3. Permanent link to max_tokens
- model adapter also adapter, adapters
-
The adapter is the code that makes each source, from a scripted list of strings to an HTTP endpoint, look the same to the loop
Defined in One interface for scripted, replayed, local and hosted models, chapter 4. In the book since Edition 2026.11.3. Permanent link to model adapter
N
- naive range
-
The naive range treats each task as an independent trial, and it is built with the usual multiplier for a 95% interval, which is slightly under two.
- Computed from: standard error of the proportion
Defined in How sure the count is, chapter 10. In the book since Edition 2026.11.3. Permanent link to naive range
- nearest lines also nearest-line errors
-
The harness slides a window of the same length as old down the file, keeps the window that matches best, and breaks a tie toward the earlier window, so the answer is the same on every run.
Defined in Edit errors that name the nearest lines, chapter 3. In the book since Edition 2026.11.3. Permanent link to nearest lines
- never class
-
The table in lib/permissions.py is the only source of a tool's class, and a tool outside it is classed never.
- A kind of: tool class
Defined in The wrapper checks every call, chapter 6. In the book since Edition 2026.11.3. Permanent link to never class
- no progress also no_progress, no-progress rule
-
No progress is two edits in a row that leave the harness's test counts unchanged.
- A kind of: ended
Defined in Decisions in this chapter, chapter 2. In the book since Edition 2026.11.3. Permanent link to no progress
- no_tool_call
-
A second reply with no call in a row ends the run as no_tool_call.
- A kind of: ended
Defined in Ways a run ends, chapter 2. In the book since Edition 2026.11.3. Permanent link to no_tool_call
- notes also note tool
-
The harness adds a note tool: the model writes one line per fact, and the harness sends the notes after the issue at every step.
- Contrasts with: local compaction
Defined in Keep notes outside the context, chapter 5. In the book since Edition 2026.11.3. Permanent link to notes
P
- pass@1 also score
-
The score is pass@1: the number of done tasks, divided by the number of tasks.
- Computed from: done (fix or implement task), done (stop task)
- Contrasts with: pass^k
Defined in Score: tasks done out of 20, chapter 10. In the book since Edition 2026.11.3. Permanent link to pass@1
- pass^k
-
tau-bench defines pass^k as the chance that all k trials of a task succeed, "averaged across tasks", and it needs k trials per task.
Defined in Score: tasks done out of 20, chapter 10. In the book since Edition 2026.11.3. Permanent link to pass^k
- permission gate also gate
-
a gate that reads the diff at finish
Defined in Gate the tools that change things, chapter 6. In the book since Edition 2026.11.3. Permanent link to permission gate
- prompt budget
-
This chapter calls the tokens left for everything the model reads on one step the prompt budget.
- Computed from: max_tokens
Defined in The window the model has, chapter 5. In the book since Edition 2026.11.3. Permanent link to prompt budget
- prompt count also prompt tokens, prompt_tokens
-
The prompt count is the tokens the model read
Defined in Tokens as counts, chapter 11. In the book since Edition 2026.11.3. Permanent link to prompt count
- proposal also proposals
-
A proposal is the copy's whole diff against the library, not the summary the model wrote.
Defined in The gate sits in finish, chapter 6. In the book since Edition 2026.11.3. Permanent link to proposal
- proposal hash
-
The proposal's identity is a hash of the changed bytes
- Part of: proposal
Defined in The gate sits in finish, chapter 6. In the book since Edition 2026.11.3. Permanent link to proposal hash
- proposal ID
-
Each proposal takes the next ID from a counter on its gate, p-1, then p-2, and so on
- Part of: proposal
Defined in An approval answers one request, chapter 6. In the book since Edition 2026.11.3. Permanent link to proposal ID
R
- ReAct
-
ReAct interleaves written reasoning with actions
Defined in A loop in about seventy lines, chapter 1. In the book since Edition 2026.11.3. Permanent link to ReAct
- refusal also refusals
-
each refusal goes back as the text of a tool result, the same kind of text as any other result, which the model reads on its next step
Defined in Errors the model can read, chapter 3. In the book since Edition 2026.11.3. Permanent link to refusal
- repeated call also repeated_call, repeated-call rule
-
Three identical calls in a row end the run as repeated_call.
- A kind of: ended
Defined in The same call, again, chapter 2. In the book since Edition 2026.11.3. Permanent link to repeated call
- replay also replayed run, ReplayModel
-
Replay checks that each request matches the one the recording holds, where a request is the messages and tool schemas sent at one step, and then returns the recorded reply.
Defined in Run it on the recorded model, chapter 1. In the book since Edition 2026.11.3. Permanent link to replay
- request
-
a request is the messages and tool schemas sent at one step
- Comes before: replay
Defined in Run it on the recorded model, chapter 1. In the book since Edition 2026.11.3. Permanent link to request
- request hash
-
ReplayModel hashes the request it was given, the sha256 of its messages and tool schemas in canonical JSON
- Computed from: request
Defined in Script the replies, or replay a stored run, chapter 4. In the book since Edition 2026.11.3. Permanent link to request hash
S
- scenario test also scenario tests
-
A scenario test asks whether the harness does the right thing with one fixed sequence of replies.
- Contrasts with: pass@1
Defined in Test the agent, then measure it on a task set, chapter 10. In the book since Edition 2026.11.3. Permanent link to scenario test
- scripted model also ScriptedModel
-
A scripted model gives back the replies you give it, in order, and nothing else.
Defined in Run it on a scripted model, chapter 1. In the book since Edition 2026.11.3. Permanent link to scripted model
- server tool prefix also server__tool
-
the harness gives each server's tools a prefix: server__tool, with a double underscore as the separator
Defined in Names that cannot clash, chapter 9. In the book since Edition 2026.11.3. Permanent link to server tool prefix
- single model call
-
A single model call fits a step that needs a model's judgement, with no tool and no loop, and whose output a person reads afterwards.
Defined in The decision table, chapter 12. In the book since Edition 2026.11.3. Permanent link to single model call
- standard error of the proportion
-
The standard error of the proportion is the square root of a fraction: the proportion times one minus the proportion, divided by the number of tasks.
- Computed from: pass@1
Defined in How sure the count is, chapter 10. In the book since Edition 2026.11.3. Permanent link to standard error of the proportion
- step also steps
-
A step is one model call.
- Comes before: step budget
Defined in A loop in about seventy lines, chapter 1. In the book since Edition 2026.11.3. Permanent link to step
- step budget also step cap, step_cap
-
The step budget is the most steps a run may take, and the loop stops when it reaches that number.
Defined in The budget counts steps, chapter 2. In the book since Edition 2026.11.3. Permanent link to step budget
- stored count also stored counts
-
Every count in this chapter is read from a recording or printed by a script, and none is taken from a live model, so the counts are stored counts, and the one check that would re-derive them from the model has not been run.
Defined in Trace each run and count what it cost, chapter 11. In the book since Edition 2026.11.3. Permanent link to stored count
- strict mode
-
In strict mode, a value must already have the declared type.
True for pydantic 2.14.3, as this edition pins it.
- Contrasts with: lax mode
- Implemented by: pydantic 2.14.3
Defined in Lax by default, strict on purpose, chapter 3. In the book since Edition 2026.11.3. Permanent link to strict mode
- structured handoff
-
the harness's structured handoff after a stop, which carries the stop reason, what the run tried and a question
- A kind of: handoff
Defined in Hand a subtask to a child, and hand back a stop, chapter 8. In the book since Edition 2026.11.3. Permanent link to structured handoff
- stuck stop also stuck stops
-
For the three stops that end a child stuck, step_cap, repeated_call and no_progress, the harness builds the handoff from the run's own record: its trace, its streak of repeated calls and its test counts.
- Comes before: structured handoff
Defined in A stop handed back, chapter 8. In the book since Edition 2026.11.3. Permanent link to stuck stop
T
- task set
-
the task set: the 20 tasks that testing the agent scores, run with tool set v1, whose tools include edits
Defined in Stop the loop when it's done or stuck, chapter 2. In the book since Edition 2026.11.3. Permanent link to task set
- teaching sandbox also sandbox
-
What the book builds is the sandbox in sandbox.py, which is teaching code and a second layer, not the protection itself.
Defined in Run the agent's code in a sandbox, and keep secrets out of its view, chapter 7. In the book since Edition 2026.11.3. Permanent link to teaching sandbox
- token also tokens
-
a token is a unit of one vocabulary, and the same text can split into different numbers of units under another
- Comes before: prompt count, completion count
Defined in Tokens as counts, chapter 11. In the book since Edition 2026.11.3. Permanent link to token
- tool annotation also annotation, annotations
-
An annotation is the server's own description of its tool, and a server that is wrong, or compromised, sets the same field as one that is right.
True for mcp 2.3.0, as this edition pins it.
- Contrasts with: class table
- Implemented by: mcp 2.3.0
Defined in Another party's tools need approval, chapter 9. In the book since Edition 2026.11.3. Permanent link to tool annotation
- tool call also tool calls
-
A tool call is one request inside a step
- Part of: step
Defined in A loop in about seventy lines, chapter 1. In the book since Edition 2026.11.3. Permanent link to tool call
- tool class
-
Chapter 6 divided the harness's tools into classes: a read runs, and a change waits.
Defined in Another party's tools need approval, chapter 9. In the book since Edition 2026.11.3. Permanent link to tool class
- tool schema also schema, tool schemas
-
Think of a tool's schema as the form the model fills in before it makes a call.
Defined in A schema says what is allowed, chapter 3. In the book since Edition 2026.11.3. Permanent link to tool schema
- tool sentence also tool description
-
A tool's sentence says what the tool is for.
Defined in A schema says what is allowed, chapter 3. In the book since Edition 2026.11.3. Permanent link to tool sentence
- tool set v1 also v1 tools
-
Tool set v1 is the seven tools the task-set recording was made with: list_files, read_file, run_tests, write_file, edit_file, ask_user and finish.
Defined in Give the model tools it can't misuse, chapter 3. In the book since Edition 2026.11.3. Permanent link to tool set v1
- tool set v2 also v2 tools
-
Tool set v2 has the same seven tools as tool set v1, with the same sentences and the same system prompt, and three changes: strict argument models, the error type named in each refusal, and nearest-line errors for a failed edit.
- Contrasts with: tool set v1
Defined in Give the model tools it can't misuse, chapter 3. In the book since Edition 2026.11.3. Permanent link to tool set v2
- trace also traces
-
The record of each step is the trace.
Defined in Trace each run and count what it cost, chapter 11. In the book since Edition 2026.11.3. Permanent link to trace
- turn
-
This book keeps "turn" for an exchange with the person who gave the task.
- Contrasts with: step
Defined in Ways a run ends, chapter 2. In the book since Edition 2026.11.3. Permanent link to turn
U
- untrusted-output label
-
The label says that the text comes from another party, that it is to be read as data, and that it cannot change the stop rules, the allowed commands or an approval, because the harness enforces those whatever the text says.
Defined in Output is untrusted, chapter 9. In the book since Edition 2026.11.3. Permanent link to untrusted-output label
V
- visible tests also visible suite
-
the visible tests, the case's own test suite
- Contrasts with: hidden check
Defined in No progress across edits, chapter 2. In the book since Edition 2026.11.3. Permanent link to visible tests
W
- workflow
-
a workflow when LLMs and tools are orchestrated through predefined code paths
- Contrasts with: agent
Defined in Start with the fixed pipeline, chapter 12. In the book since Edition 2026.11.3. Permanent link to workflow