Glossary

The 86 terms Agent Harnesses defines, each in the book's own words, with the section that defines it.

From Edition 2026.11.3, the book's current edition. Where a definition depends on a package, it says the version this edition pins and runs its examples on.

A

action also actions

A pipeline has actions, which are its tool calls

  • Contrasts with: step

Defined in Start with the fixed pipeline, chapter 12. In the book since Edition 2026.11.3. Permanent link to action

agent also agents

an agent when an LLM dynamically directs its own processes and tool use

Defined in Start with the fixed pipeline, chapter 12. In the book since Edition 2026.11.3. Permanent link to agent

agent harness also harness

The harness supplies the loop, the tools and the limits on what those tools may do

Defined in Give a model tools and a loop, chapter 1. In the book since Edition 2026.11.3. Permanent link to agent harness

allow list also allow lists

A tool on an allow list runs without a question, and a gate placed at the question is then bypassed for that tool.

Defined in Allow lists do not skip the check, chapter 6. In the book since Edition 2026.11.3. Permanent link to allow list

argument model also argument models

an argument model, a small pydantic class that lists the fields the tool takes

True for pydantic 2.14.3, as this edition pins it.

  • Comes before: tool schema
  • Implemented by: pydantic 2.14.3

Defined in A schema says what is allowed, chapter 3. In the book since Edition 2026.11.3. Permanent link to argument model

argv allowlist also allowlist

The allowlist is the check a command tool would use: it parses a command string once with shlex.split, and compares the argv with a short list of exact argv tuples.

Defined in Only listed programs, chapter 7. In the book since Edition 2026.11.3. Permanent link to argv allowlist

B

brief

The brief is the job as the child receives it.

Defined in A brief with four parts, chapter 8. In the book since Edition 2026.11.3. Permanent link to brief

C

child also subagent, child run

A child is the loop from chapter 1, started by a parent in the middle of its own run.

Defined in A child is a loop with its own budget, chapter 8. In the book since Edition 2026.11.3. Permanent link to child

child budget

The child's budget counts the child's own model calls, from its first call, so the parent's steps don't pay for the child's work, and a child that loops runs out of its own budget.

Defined in A child is a loop with its own budget, chapter 8. In the book since Edition 2026.11.3. Permanent link to child budget

class table also POLICY

The harness's class table, POLICY in lib/mcp_tools.py, gives each server tool a class, read or change, and only names that table gives a class appear in that line.

Defined in Listing a server's tools, chapter 9. In the book since Edition 2026.11.3. Permanent link to class table

command also argv list

A command is an argv list, never a shell string, and the runner starts it with shell=False, so no shell reads its quotes or its semicolons (subprocess).

Defined in A teaching sandbox, not a container, chapter 7. In the book since Edition 2026.11.3. Permanent link to command

completion count also completion tokens, completion_tokens

the completion count is the tokens it wrote

Defined in Tokens as counts, chapter 11. In the book since Edition 2026.11.3. Permanent link to completion count

copy also scratch copy

Each run gets its own copy, made by scratch_copy in run_task, and the tools work through a Workspace rooted at that copy.

Defined in The copy comes first, chapter 6. In the book since Edition 2026.11.3. Permanent link to copy

cost

a run's cost is three numbers: its steps, its tool calls and its tokens

Defined in Cost as counts, not money, chapter 11. In the book since Edition 2026.11.3. Permanent link to cost

D

decision table

The decision table sorts each task by three readings: whether its steps are known before the run, whether a check can decide the result without a person, and whether a person has to say yes.

Defined in The decision table, chapter 12. In the book since Edition 2026.11.3. Permanent link to decision table

done (fix or implement task) also done

A fix or an implement task is done when the run ended with finish, its hidden check passes on the final copy, and the task's protected paths match the case byte for byte.

Defined in Score: tasks done out of 20, chapter 10. In the book since Edition 2026.11.3. Permanent link to done (fix or implement task)

done (stop task)

A stop task is done when the run ended with ask_user, its question contains one of the task's reason words, and no file in the copy differs from the case.

Defined in Score: tasks done out of 20, chapter 10. In the book since Edition 2026.11.3. Permanent link to done (stop task)

E

edit also edits

An edit names the text it replaces.

Defined in Edit errors that name the nearest lines, chapter 3. In the book since Edition 2026.11.3. Permanent link to edit

ended also stop reason, ended value

Every run ends for one stated reason, and the harness writes that reason into the run's ended field.

Defined in Ways a run ends, chapter 2. In the book since Edition 2026.11.3. Permanent link to ended

estimate also prompt estimate

The estimate counts the characters of every message and of the tool schemas, and divides by three, rounding up: one token for every 3 characters.

Defined in Estimate the prompt when the harness can't count it, chapter 5. In the book since Edition 2026.11.3. Permanent link to estimate

F

fence

The fence is an opening tag that names the tool, and a closing tag.

Defined in Output is untrusted, chapter 9. In the book since Edition 2026.11.3. Permanent link to fence

finish reason also finish_reason

the finish reason, which says whether the model stopped on its own or ran out of tokens

Defined in Ask every model the same question, chapter 4. In the book since Edition 2026.11.3. Permanent link to finish reason

fixed part

The fixed part, the system prompt, the tool schemas and the issue

Defined in Compact the older steps, and keep the last pair, chapter 5. In the book since Edition 2026.11.3. Permanent link to fixed part

fixed pipeline also pipeline

A fixed pipeline is a task's list of steps, written as code before the run starts.

Defined in Start with the fixed pipeline, chapter 12. In the book since Edition 2026.11.3. Permanent link to fixed pipeline

G

GGUF also GGUF file

a GGUF file, the single-file format llama.cpp reads

Defined in A local model, and why its settings are fixed, chapter 4. In the book since Edition 2026.11.3. Permanent link to GGUF

H

handoff also handoffs

a handoff is a run's end with a question back to a person, instead of a finish

Defined in Hand a subtask to a child, and hand back a stop, chapter 8. In the book since Edition 2026.11.3. Permanent link to handoff

harness v1

Harness v1 is the loop of chapter 1 with tool set v1 from chapter 3, a step cap of 16 steps, and no stop rule beyond it.

Defined in Two harness versions, compared as counts, chapter 10. In the book since Edition 2026.11.3. Permanent link to harness v1

harness v2

Harness v2 is the configuration of chapter 5: it runs tool set v2, with strict argument models, typed error results and edit errors that name the nearest lines of the file.

Defined in Two harness versions, compared as counts, chapter 10. In the book since Edition 2026.11.3. Permanent link to harness v2

headless run also headless run with a callback

This harness has no terminal and no session: it is a headless run, and the only party who can answer is a callback that the caller passes in.

Defined in Which mode this harness is in, chapter 6. In the book since Edition 2026.11.3. Permanent link to headless run

hidden check

The hidden check asks whether a finished run fixed the task.

Defined in Test the agent, then measure it on a task set, chapter 10. In the book since Edition 2026.11.3. Permanent link to hidden check

hybrid

The hybrid, a pipeline first and an agent only on a pipeline_stop

Defined in Start with the fixed pipeline, chapter 12. In the book since Edition 2026.11.3. Permanent link to hybrid

I

identical call also identical calls

A call is identical when it has the same tool name and the same arguments, compared with the keys sorted.

Defined in The same call, again, chapter 2. In the book since Edition 2026.11.3. Permanent link to identical call

indirect prompt injection also prompt injection

The threat is indirect prompt injection: text that an application retrieves from outside can carry instructions, and a model that reads that text may follow them.

Defined in Output is untrusted, chapter 9. In the book since Edition 2026.11.3. Permanent link to indirect prompt injection

L

lax mode

In pydantic's default, lax mode, a value is converted when the conversion is clear and refused when it is not.

True for pydantic 2.14.3, as this edition pins it.

  • Implemented by: pydantic 2.14.3

Defined in Lax by default, strict on purpose, chapter 3. In the book since Edition 2026.11.3. Permanent link to lax mode

library guard also guard

a guard that compares the library's bytes around every tool call

Defined in Gate the tools that change things, chapter 6. In the book since Edition 2026.11.3. Permanent link to library guard

library_changed

when the library has changed, the run ends as library_changed, the harness applies nothing, and it restores nothing

Defined in Gate the tools that change things, chapter 6. In the book since Edition 2026.11.3. Permanent link to library_changed

local compaction also compaction

Compaction here is local: the harness writes the summary itself, and the model never sees it being written.

Defined in Compact the older steps, and keep the last pair, chapter 5. In the book since Edition 2026.11.3. Permanent link to local compaction

M

malformed call also malformed calls

The harness never runs a block that doesn't parse: it records it as a malformed call and hands the reason back to the model as an error result, so the model can try again.

Defined in A loop in about seventy lines, chapter 1. In the book since Edition 2026.11.3. Permanent link to malformed call

max_tokens

max_tokens, the most the reply may take

Defined in The window the model has, chapter 5. In the book since Edition 2026.11.3. Permanent link to max_tokens

model adapter also adapter, adapters

The adapter is the code that makes each source, from a scripted list of strings to an HTTP endpoint, look the same to the loop

Defined in One interface for scripted, replayed, local and hosted models, chapter 4. In the book since Edition 2026.11.3. Permanent link to model adapter

N

naive range

The naive range treats each task as an independent trial, and it is built with the usual multiplier for a 95% interval, which is slightly under two.

Defined in How sure the count is, chapter 10. In the book since Edition 2026.11.3. Permanent link to naive range

nearest lines also nearest-line errors

The harness slides a window of the same length as old down the file, keeps the window that matches best, and breaks a tie toward the earlier window, so the answer is the same on every run.

Defined in Edit errors that name the nearest lines, chapter 3. In the book since Edition 2026.11.3. Permanent link to nearest lines

never class

The table in lib/permissions.py is the only source of a tool's class, and a tool outside it is classed never.

Defined in The wrapper checks every call, chapter 6. In the book since Edition 2026.11.3. Permanent link to never class

no progress also no_progress, no-progress rule

No progress is two edits in a row that leave the harness's test counts unchanged.

Defined in Decisions in this chapter, chapter 2. In the book since Edition 2026.11.3. Permanent link to no progress

no_tool_call

A second reply with no call in a row ends the run as no_tool_call.

Defined in Ways a run ends, chapter 2. In the book since Edition 2026.11.3. Permanent link to no_tool_call

notes also note tool

The harness adds a note tool: the model writes one line per fact, and the harness sends the notes after the issue at every step.

Defined in Keep notes outside the context, chapter 5. In the book since Edition 2026.11.3. Permanent link to notes

P

pass@1 also score

The score is pass@1: the number of done tasks, divided by the number of tasks.

Defined in Score: tasks done out of 20, chapter 10. In the book since Edition 2026.11.3. Permanent link to pass@1

pass^k

tau-bench defines pass^k as the chance that all k trials of a task succeed, "averaged across tasks", and it needs k trials per task.

Defined in Score: tasks done out of 20, chapter 10. In the book since Edition 2026.11.3. Permanent link to pass^k

permission gate also gate

a gate that reads the diff at finish

Defined in Gate the tools that change things, chapter 6. In the book since Edition 2026.11.3. Permanent link to permission gate

prompt budget

This chapter calls the tokens left for everything the model reads on one step the prompt budget.

Defined in The window the model has, chapter 5. In the book since Edition 2026.11.3. Permanent link to prompt budget

prompt count also prompt tokens, prompt_tokens

The prompt count is the tokens the model read

Defined in Tokens as counts, chapter 11. In the book since Edition 2026.11.3. Permanent link to prompt count

proposal also proposals

A proposal is the copy's whole diff against the library, not the summary the model wrote.

Defined in The gate sits in finish, chapter 6. In the book since Edition 2026.11.3. Permanent link to proposal

proposal hash

The proposal's identity is a hash of the changed bytes

Defined in The gate sits in finish, chapter 6. In the book since Edition 2026.11.3. Permanent link to proposal hash

proposal ID

Each proposal takes the next ID from a counter on its gate, p-1, then p-2, and so on

Defined in An approval answers one request, chapter 6. In the book since Edition 2026.11.3. Permanent link to proposal ID

R

ReAct

ReAct interleaves written reasoning with actions

Defined in A loop in about seventy lines, chapter 1. In the book since Edition 2026.11.3. Permanent link to ReAct

refusal also refusals

each refusal goes back as the text of a tool result, the same kind of text as any other result, which the model reads on its next step

Defined in Errors the model can read, chapter 3. In the book since Edition 2026.11.3. Permanent link to refusal

repeated call also repeated_call, repeated-call rule

Three identical calls in a row end the run as repeated_call.

Defined in The same call, again, chapter 2. In the book since Edition 2026.11.3. Permanent link to repeated call

replay also replayed run, ReplayModel

Replay checks that each request matches the one the recording holds, where a request is the messages and tool schemas sent at one step, and then returns the recorded reply.

Defined in Run it on the recorded model, chapter 1. In the book since Edition 2026.11.3. Permanent link to replay

request

a request is the messages and tool schemas sent at one step

Defined in Run it on the recorded model, chapter 1. In the book since Edition 2026.11.3. Permanent link to request

request hash

ReplayModel hashes the request it was given, the sha256 of its messages and tool schemas in canonical JSON

Defined in Script the replies, or replay a stored run, chapter 4. In the book since Edition 2026.11.3. Permanent link to request hash

S

scenario test also scenario tests

A scenario test asks whether the harness does the right thing with one fixed sequence of replies.

Defined in Test the agent, then measure it on a task set, chapter 10. In the book since Edition 2026.11.3. Permanent link to scenario test

scripted model also ScriptedModel

A scripted model gives back the replies you give it, in order, and nothing else.

Defined in Run it on a scripted model, chapter 1. In the book since Edition 2026.11.3. Permanent link to scripted model

server tool prefix also server__tool

the harness gives each server's tools a prefix: server__tool, with a double underscore as the separator

Defined in Names that cannot clash, chapter 9. In the book since Edition 2026.11.3. Permanent link to server tool prefix

single model call

A single model call fits a step that needs a model's judgement, with no tool and no loop, and whose output a person reads afterwards.

Defined in The decision table, chapter 12. In the book since Edition 2026.11.3. Permanent link to single model call

standard error of the proportion

The standard error of the proportion is the square root of a fraction: the proportion times one minus the proportion, divided by the number of tasks.

Defined in How sure the count is, chapter 10. In the book since Edition 2026.11.3. Permanent link to standard error of the proportion

step also steps

A step is one model call.

Defined in A loop in about seventy lines, chapter 1. In the book since Edition 2026.11.3. Permanent link to step

step budget also step cap, step_cap

The step budget is the most steps a run may take, and the loop stops when it reaches that number.

Defined in The budget counts steps, chapter 2. In the book since Edition 2026.11.3. Permanent link to step budget

stored count also stored counts

Every count in this chapter is read from a recording or printed by a script, and none is taken from a live model, so the counts are stored counts, and the one check that would re-derive them from the model has not been run.

Defined in Trace each run and count what it cost, chapter 11. In the book since Edition 2026.11.3. Permanent link to stored count

strict mode

In strict mode, a value must already have the declared type.

True for pydantic 2.14.3, as this edition pins it.

  • Contrasts with: lax mode
  • Implemented by: pydantic 2.14.3

Defined in Lax by default, strict on purpose, chapter 3. In the book since Edition 2026.11.3. Permanent link to strict mode

structured handoff

the harness's structured handoff after a stop, which carries the stop reason, what the run tried and a question

Defined in Hand a subtask to a child, and hand back a stop, chapter 8. In the book since Edition 2026.11.3. Permanent link to structured handoff

stuck stop also stuck stops

For the three stops that end a child stuck, step_cap, repeated_call and no_progress, the harness builds the handoff from the run's own record: its trace, its streak of repeated calls and its test counts.

Defined in A stop handed back, chapter 8. In the book since Edition 2026.11.3. Permanent link to stuck stop

T

task set

the task set: the 20 tasks that testing the agent scores, run with tool set v1, whose tools include edits

Defined in Stop the loop when it's done or stuck, chapter 2. In the book since Edition 2026.11.3. Permanent link to task set

teaching sandbox also sandbox

What the book builds is the sandbox in sandbox.py, which is teaching code and a second layer, not the protection itself.

Defined in Run the agent's code in a sandbox, and keep secrets out of its view, chapter 7. In the book since Edition 2026.11.3. Permanent link to teaching sandbox

token also tokens

a token is a unit of one vocabulary, and the same text can split into different numbers of units under another

Defined in Tokens as counts, chapter 11. In the book since Edition 2026.11.3. Permanent link to token

tool annotation also annotation, annotations

An annotation is the server's own description of its tool, and a server that is wrong, or compromised, sets the same field as one that is right.

True for mcp 2.3.0, as this edition pins it.

Defined in Another party's tools need approval, chapter 9. In the book since Edition 2026.11.3. Permanent link to tool annotation

tool call also tool calls

A tool call is one request inside a step

Defined in A loop in about seventy lines, chapter 1. In the book since Edition 2026.11.3. Permanent link to tool call

tool class

Chapter 6 divided the harness's tools into classes: a read runs, and a change waits.

Defined in Another party's tools need approval, chapter 9. In the book since Edition 2026.11.3. Permanent link to tool class

tool schema also schema, tool schemas

Think of a tool's schema as the form the model fills in before it makes a call.

Defined in A schema says what is allowed, chapter 3. In the book since Edition 2026.11.3. Permanent link to tool schema

tool sentence also tool description

A tool's sentence says what the tool is for.

Defined in A schema says what is allowed, chapter 3. In the book since Edition 2026.11.3. Permanent link to tool sentence

tool set v1 also v1 tools

Tool set v1 is the seven tools the task-set recording was made with: list_files, read_file, run_tests, write_file, edit_file, ask_user and finish.

Defined in Give the model tools it can't misuse, chapter 3. In the book since Edition 2026.11.3. Permanent link to tool set v1

tool set v2 also v2 tools

Tool set v2 has the same seven tools as tool set v1, with the same sentences and the same system prompt, and three changes: strict argument models, the error type named in each refusal, and nearest-line errors for a failed edit.

Defined in Give the model tools it can't misuse, chapter 3. In the book since Edition 2026.11.3. Permanent link to tool set v2

trace also traces

The record of each step is the trace.

Defined in Trace each run and count what it cost, chapter 11. In the book since Edition 2026.11.3. Permanent link to trace

turn

This book keeps "turn" for an exchange with the person who gave the task.

  • Contrasts with: step

Defined in Ways a run ends, chapter 2. In the book since Edition 2026.11.3. Permanent link to turn

U

untrusted-output label

The label says that the text comes from another party, that it is to be read as data, and that it cannot change the stop rules, the allowed commands or an approval, because the harness enforces those whatever the text says.

Defined in Output is untrusted, chapter 9. In the book since Edition 2026.11.3. Permanent link to untrusted-output label

V

visible tests also visible suite

the visible tests, the case's own test suite

Defined in No progress across edits, chapter 2. In the book since Edition 2026.11.3. Permanent link to visible tests

W

workflow

a workflow when LLMs and tools are orchestrated through predefined code paths

Defined in Start with the fixed pipeline, chapter 12. In the book since Edition 2026.11.3. Permanent link to workflow