Glossary

The 125 terms Evaluating LLM Applications defines, each in the book's own words, with the section that defines it.

From Edition 2026.11.3, the book's current edition. Where a definition depends on a package, it says the version this edition pins and runs its examples on.

A

agreement

Agreement is the share of rows on which the judge and the human give the same verdict.

Defined in Agreement and kappa, chapter 6. In the book since Edition 2026.11.3. Permanent link to agreement

ambiguous item

Ambiguous items could mean two things, and their reference answer covers both readings.

Defined in The twenty items, chapter 1. In the book since Edition 2026.11.3. Permanent link to ambiguous item

ambiguous target also ambiguous-target

ambiguous-target: a pronoun or a phrase such as "the file" has no clear referent without the sections.

Defined in Planted items and their traps, chapter 3. In the book since Edition 2026.11.3. Permanent link to ambiguous target

answer judge

The answer judge checks a reply about sablefile against the written reference answer for its question.

Defined in What a judge is asked, chapter 5. In the book since Edition 2026.11.3. Permanent link to answer judge

answerable item

Answerable items have their answer on the page.

Defined in The twenty items, chapter 1. In the book since Edition 2026.11.3. Permanent link to answerable item

answered-declined-expected

answered-declined-expected is an answer to an item that has to be declined.

Defined in Format checks, chapter 4. In the book since Edition 2026.11.3. Permanent link to answered-declined-expected

autojunk

For sequences of 200 items or more, the documentation treats an item as junk when its repeats make up more than one per cent of the sequence, unless autojunk is turned off.

Defined in Exact and fuzzy match, chapter 4. In the book since Edition 2026.11.3. Permanent link to autojunk

B

baseline

The baseline is the setup chapter 1 measured.

Defined in What the replies look like, chapter 2. In the book since Edition 2026.11.3. Permanent link to baseline

BLEU

Papineni and colleagues proposed BLEU in 2002 as a quick and inexpensive way to score machine translation.

Defined in Where BLEU and ROUGE mislead, chapter 4. In the book since Edition 2026.11.3. Permanent link to BLEU

BM25

The index is built with BM25, which scores each section by the words it shares with the question.

True for bm25s 0.3.16, as this edition pins it.

  • Implemented by: bm25s 0.3.16

Defined in Retrieve, then answer, chapter 1. In the book since Edition 2026.11.3. Permanent link to BM25

Bonferroni correction

the Bonferroni correction, which shares the error budget between the comparisons.

1 minus 0.05 divided by m

Defined in Many comparisons, chapter 8. In the book since Edition 2026.11.3. Permanent link to Bonferroni correction

C

calibration rubric also rubric A

The calibration rubric checks correctness with no reference, on rows from HelpSteer2

Defined in What a judge is asked, chapter 5. In the book since Edition 2026.11.3. Permanent link to calibration rubric

chance share

The chance share averages one over each item's option count.

Defined in TruthfulQA, chapter 3. In the book since Edition 2026.11.3. Permanent link to chance share

check candidate

a candidate has four parts: the mode, the input the check reads, the claim it tests, and what it can't see.

Defined in From a failure to a check, chapter 2. In the book since Edition 2026.11.3. Permanent link to check candidate

citation

A citation is a claim about the docs: this section, in this version, supports the sentence it is attached to.

Defined in Citation accuracy, chapter 9. In the book since Edition 2026.11.3. Permanent link to citation

citation accuracy

Citation accuracy asks whether each citation names a section the item lists as supporting.

Defined in Check that answers rest on the cited docs, in the asked version, chapter 9. In the book since Edition 2026.11.3. Permanent link to citation accuracy

citation form

the citation form, a section ID in square brackets

Defined in Citations, chapter 1. In the book since Edition 2026.11.3. Permanent link to citation form

citation rate

The citation rate counts every citation in answered items that is supported.

Defined in Citation accuracy, chapter 9. In the book since Edition 2026.11.3. Permanent link to citation rate

citation-first

citation-first, for a reply whose first line opens with a citation

Defined in Format checks, chapter 4. In the book since Edition 2026.11.3. Permanent link to citation-first

code check

A code check reads two things: the reply's text, and the item's own data, which holds its reference answer and the list of sections that support it.

Defined in What a code check can see, chapter 4. In the book since Edition 2026.11.3. Permanent link to code check

consistent pair

A pair is consistent when the same response wins in both orders.

Defined in One answer or two, chapter 5. In the book since Edition 2026.11.3. Permanent link to consistent pair

contamination

Contamination is when a benchmark's items or answers appear in training data, so a score can reflect memory rather than ability.

Defined in Contamination, chapter 3. In the book since Edition 2026.11.3. Permanent link to contamination

corrected pass rate also corrected rate

The corrected rate estimates the share of sablefile's replies that a person would pass, under the book's correctness cut.

(observed + TNR - 1) / (TPR + TNR - 1)

Defined in Correcting a pass rate, chapter 6. In the book since Edition 2026.11.3. Permanent link to corrected pass rate

coverage

Coverage asks which parts of the tool the set exercises.

Defined in Kinds and coverage, chapter 3. In the book since Edition 2026.11.3. Permanent link to coverage

crossover point also p*

The pass-rate correction in the next section leaves a rate unchanged at the crossover point, p* = (1 − TNR) / (2 − TPR − TNR).

p* = (1 − TNR) / (2 − TPR − TNR)

Defined in True positive and true negative rates, chapter 6. In the book since Edition 2026.11.3. Permanent link to crossover point

D

decline phrase also decline sentence

When the docs don't answer a question, the assistant must reply with the decline phrase, "The docs don't say.", and nothing else.

Defined in Declines, chapter 1. In the book since Edition 2026.11.3. Permanent link to decline phrase

declined-with-text

declined-with-text is a reply that contains the decline sentence along with other text.

Defined in Format checks, chapter 4. In the book since Edition 2026.11.3. Permanent link to declined-with-text

dev also dev items

dev items, the split of the test set this book works with while it develops its measures

Defined in Read the replies to find what to measure, chapter 2. In the book since Edition 2026.11.3. Permanent link to dev

E

empty

empty is a reply with no text.

Defined in Format checks, chapter 4. In the book since Edition 2026.11.3. Permanent link to empty

exact match

An exact match asks whether two strings are equal, and it suits any text that is fixed.

Defined in Exact and fuzzy match, chapter 4. In the book since Edition 2026.11.3. Permanent link to exact match

example environment

the book keeps one example environment: one set of files that every chapter's examples import, pinned to the same versions, so the replay model, the checks and the gate are the same code in every chapter.

Defined in Running it as a framework task, chapter 11. In the book since Edition 2026.11.3. Permanent link to example environment

extra="forbid"

The loader sets extra="forbid", so an unknown field is an error and not a field that is quietly ignored.

True for pydantic 2.14.3, as this edition pins it.

  • Implemented by: pydantic 2.14.3

Defined in The item schema, chapter 3. In the book since Edition 2026.11.3. Permanent link to extra="forbid"

F

F1

F1, which combines recall with precision

Defined in Where BLEU and ROUGE mislead, chapter 4. In the book since Edition 2026.11.3. Permanent link to F1

FAIL

A FAIL means the judge judged the answer wrong

Defined in Verdict first, chapter 5. In the book since Edition 2026.11.3. Permanent link to FAIL

failure mode also mode

Each reply got one mode, its main defect

Defined in Coding the failures, chapter 2. In the book since Edition 2026.11.3. Permanent link to failure mode

faithfulness

Faithfulness asks whether each claim is in the sections the assistant was shown.

Defined in Check that answers rest on the cited docs, in the asked version, chapter 9. In the book since Edition 2026.11.3. Permanent link to faithfulness

false answer

a false answer is an unanswerable item answered.

Defined in Declines, chapter 1. In the book since Edition 2026.11.3. Permanent link to false answer

false answer rate

The false answer rate is the number of unanswerable items answered, over the number of unanswerable items.

Defined in Two errors, both counted, chapter 10. In the book since Edition 2026.11.3. Permanent link to false answer rate

false premise also false-premise

false-premise: the question asks about a flag or key that the asked version does not have.

Defined in Planted items and their traps, chapter 3. In the book since Edition 2026.11.3. Permanent link to false premise

false refusal

A false refusal is an answerable item declined

Defined in Declines, chapter 1. In the book since Edition 2026.11.3. Permanent link to false refusal

false refusal rate

The false refusal rate is the number of answerable items declined, over the number of answerable items.

Defined in Two errors, both counted, chapter 10. In the book since Edition 2026.11.3. Permanent link to false refusal rate

flip also flips, flaky

An item flips when its five verdicts are not all the same.

Defined in Variance within one item, chapter 8. In the book since Edition 2026.11.3. Permanent link to flip

flip rate

the flip rate, the share of items that flip

Defined in Flag flaky items, never edit them, chapter 11. In the book since Edition 2026.11.3. Permanent link to flip rate

format check

The format check asks whether a reply has the shape the prompt asks for, before anyone reads what it says.

Defined in Format checks, chapter 4. In the book since Edition 2026.11.3. Permanent link to format check

fuzzy match

A fuzzy match scores how alike two strings are.

Defined in Exact and fuzzy match, chapter 4. In the book since Edition 2026.11.3. Permanent link to fuzzy match

G

gate

A gate is a rule that stops a change from merging.

Defined in A gate that respects the interval, chapter 11. In the book since Edition 2026.11.3. Permanent link to gate

grounded

An answer is grounded when a reader who follows its citations finds the claim where the citation says it is, in the version they asked about.

Defined in What grounded means here, chapter 9. In the book since Edition 2026.11.3. Permanent link to grounded

GSM8K

GSM8K is grade-school math word problems, each with a worked solution that ends in one number.

Defined in GSM8K, chapter 3. In the book since Edition 2026.11.3. Permanent link to GSM8K

guard also retrieval-score guard, decline guard

A guard reads the best-matching section's score and declines when it falls below a threshold.

Defined in A guard on recorded replies, chapter 10. In the book since Edition 2026.11.3. Permanent link to guard

H

held-out also held-out items

held-out is used once, to report a result.

Defined in Dev and held-out, chapter 3. In the book since Edition 2026.11.3. Permanent link to held-out

I

interval also 95% interval

An interval answers a different question from the pass rate: how far would the rate move if the set had been another twenty items drawn the same way?

Defined in A pass rate and its interval, chapter 1. In the book since Edition 2026.11.3. Permanent link to interval

item

an item: one test question with the fields a scorer needs.

Defined in What a test set is for, chapter 3. In the book since Edition 2026.11.3. Permanent link to item

item rate

The item rate counts an answered item once if at least one of its citations is supported, so one good citation carries the item.

Defined in Citation accuracy, chapter 9. In the book since Edition 2026.11.3. Permanent link to item rate

J

judge

a judge: a model that reads the question, a written reference answer and the assistant's reply, and returns PASS or FAIL

Defined in One judge, chapter 1. In the book since Edition 2026.11.3. Permanent link to judge

judged-correct rate

The judged-correct rate is the count for version-dependent items alone, so a version miss shows on its own line, and a team can see whether a change to retrieval moved it.

Defined in Version-correctness, chapter 9. In the book since Edition 2026.11.3. Permanent link to judged-correct rate

K

k5

The variant k5 retrieves five sections instead of three.

Defined in Held-out, once, chapter 8. In the book since Edition 2026.11.3. Permanent link to k5

kappa

For two raters on pass or fail, kappa is the observed agreement minus the chance agreement, divided by one minus the chance agreement.

Defined in Agreement and kappa, chapter 6. In the book since Edition 2026.11.3. Permanent link to kappa

kind

Each item also has a kind, which says what a correct reply looks like.

Defined in What a test set is for, chapter 3. In the book since Edition 2026.11.3. Permanent link to kind

L

length also verbosity bias

length: whether the pass rate rises with the length of the reply.

Defined in Find where a judge's verdict stops being reliable, chapter 7. In the book since Edition 2026.11.3. Permanent link to length

M

MC1

The book uses TruthfulQA's multiple-choice version, MC1: exactly one option is correct, and the model answers with a letter.

Defined in TruthfulQA, chapter 3. In the book since Edition 2026.11.3. Permanent link to MC1

missed refusal

A missed refusal answers an unanswerable item with content the docs don't hold.

Defined in Missed refusals, chapter 2. In the book since Edition 2026.11.3. Permanent link to missed refusal

missing

missing means no version has a section with the cited ID.

Defined in Citation checks, chapter 4. In the book since Edition 2026.11.3. Permanent link to missing

N

naive gate

A naive gate would fail whenever the point estimate of the difference is below minus the threshold

Defined in A gate that respects the interval, chapter 11. In the book since Edition 2026.11.3. Permanent link to naive gate

near-miss term also near-miss-term

a planted near-miss term, a name close to one the docs use, which catches a careless reply

Defined in What the replies look like, chapter 2. In the book since Edition 2026.11.3. Permanent link to near-miss term

no difference detected

No difference detected is a finding about a set at its size.

Defined in Paired comparisons, chapter 8. In the book since Edition 2026.11.3. Permanent link to no difference detected

no-citation

no-citation is an answer that cites nothing.

Defined in Format checks, chapter 4. In the book since Edition 2026.11.3. Permanent link to no-citation

O

ok

a reply with no defect got ok.

Defined in Coding the failures, chapter 2. In the book since Edition 2026.11.3. Permanent link to ok

order-changed pair also order-changed

An order-changed pair is one where the verdict followed the position, so it says nothing about which reply is better.

Defined in One answer or two, chapter 5. In the book since Edition 2026.11.3. Permanent link to order-changed pair

outcome

Outcome is whether a reply passed.

  • Contrasts with: path

Defined in Outcome and path, chapter 9. In the book since Edition 2026.11.3. Permanent link to outcome

outcome against path

Outcome against path asks whether a pass came with its section in hand, and whether a fail had it.

Defined in Check that answers rest on the cited docs, in the asked version, chapter 9. In the book since Edition 2026.11.3. Permanent link to outcome against path

over-declining

An over-declining reply declines an item the docs answer.

Defined in Over-declining, chapter 2. In the book since Edition 2026.11.3. Permanent link to over-declining

P

paired comparison

A paired comparison subtracts, for each item, the verdict of the baseline from the verdict of prompt B, and averages the differences over the set.

Defined in Paired comparisons, chapter 8. In the book since Edition 2026.11.3. Permanent link to paired comparison

paired difference

The paired difference is rubric B's score minus rubric A's, averaged over the dev rows

Defined in Agreement and kappa, chapter 6. In the book since Edition 2026.11.3. Permanent link to paired difference

pairwise judge

A pairwise judge reads two replies and picks the better one, rather than passing or failing each.

Defined in Pairwise calibration, chapter 6. In the book since Edition 2026.11.3. Permanent link to pairwise judge

pairwise judgment

A pairwise judgment shows the judge two replies to the same request and asks which is better.

Defined in One answer or two, chapter 5. In the book since Edition 2026.11.3. Permanent link to pairwise judgment

pass or fail rubric

A pass or fail rubric gives the judge two answers.

Defined in Pass, fail, or a scale, chapter 5. In the book since Edition 2026.11.3. Permanent link to pass or fail rubric

pass rate

A pass rate is a count divided by a denominator

pass rate = passed items / items

Defined in A pass rate and its interval, chapter 1. In the book since Edition 2026.11.3. Permanent link to pass rate

path

Path is what the steps before the reply reached.

Defined in Outcome and path, chapter 9. In the book since Edition 2026.11.3. Permanent link to path

percentile bootstrap

The percentile bootstrap is the simplest interval built on resampling, the middle 95% of the resampled rates.

Defined in A pass rate and its interval, chapter 1. In the book since Edition 2026.11.3. Permanent link to percentile bootstrap

planted item

A planted item was written to catch one particular mistake

Defined in The twenty items, chapter 1. In the book since Edition 2026.11.3. Permanent link to planted item

position

position: whether a pairwise verdict depends on which answer was shown first.

Defined in Find where a judge's verdict stops being reliable, chapter 7. In the book since Edition 2026.11.3. Permanent link to position

public benchmark

A public benchmark is a set of questions and answers someone else published.

Defined in Public benchmarks, chapter 3. In the book since Edition 2026.11.3. Permanent link to public benchmark

R

ratio()

ratio(), which the documentation defines as twice the number of matching elements divided by the total number of elements in both sequences.

Defined in Exact and fuzzy match, chapter 4. In the book since Edition 2026.11.3. Permanent link to ratio()

recording

The recording is a folder of replies, one per item.

Defined in Replay the answers, chapter 1. In the book since Edition 2026.11.3. Permanent link to recording

reference answer

the reference answer, which is the author's statement of what a right reply says

Defined in What the replies look like, chapter 2. In the book since Edition 2026.11.3. Permanent link to reference answer

reference-free judge

A reference-free judge is given only the request and the reply, and it decides from what it knows whether the reply is right.

Defined in Reference-guided or reference-free, chapter 5. In the book since Edition 2026.11.3. Permanent link to reference-free judge

reference-guided judge

A reference-guided judge is given a written answer for the question, and it checks the reply against that answer.

Defined in Reference-guided or reference-free, chapter 5. In the book since Edition 2026.11.3. Permanent link to reference-guided judge

retrieval-path count

The retrieval-path count asks one question of each item: is a supporting section of the asked version in the top k?

Defined in Retrieval quality belongs to another book, chapter 9. In the book since Edition 2026.11.3. Permanent link to retrieval-path count

ROUGE-1 F1

ROUGE-1 F1 counts single words.

Defined in Where BLEU and ROUGE mislead, chapter 4. In the book since Edition 2026.11.3. Permanent link to ROUGE-1 F1

ROUGE-N

Lin defined ROUGE-N as an n-gram recall: the share of the reference's word runs that the candidate covers

Defined in Where BLEU and ROUGE mislead, chapter 4. In the book since Edition 2026.11.3. Permanent link to ROUGE-N

rubric

A judge works from its instructions, and the instructions are the rubric.

Defined in Write a rubric a judge can apply, chapter 5. In the book since Edition 2026.11.3. Permanent link to rubric

rubric B

Rubric B is a second wording of the calibration task.

Defined in What a judge is asked, chapter 5. In the book since Edition 2026.11.3. Permanent link to rubric B

S

scale

A scale gives the judge several answers.

Defined in Pass, fail, or a scale, chapter 5. In the book since Edition 2026.11.3. Permanent link to scale

second-model variant also second model

The second-model variant keeps the baseline's prompt and hands the answer to a smaller model, Qwen3 with 0.6 billion parameters in an 8-bit build.

Defined in What the replies look like, chapter 2. In the book since Edition 2026.11.3. Permanent link to second-model variant

self-preference

self-preference: whether the judge favours answers written by its own model.

Defined in Find where a judge's verdict stops being reliable, chapter 7. In the book since Edition 2026.11.3. Permanent link to self-preference

stale answer also stale-answer

stale-answer: the answer sits under a section ID that a version filter removes, so a filter that is too strict misses it.

Defined in Planted items and their traps, chapter 3. In the book since Edition 2026.11.3. Permanent link to stale answer

stratified

The split is stratified, so that each kind appears in both parts in about the same proportion.

Defined in Dev and held-out, chapter 3. In the book since Edition 2026.11.3. Permanent link to stratified

stray-line

stray-line, for a line that ends no sentence once its citations are taken out.

Defined in Format checks, chapter 4. In the book since Edition 2026.11.3. Permanent link to stray-line

supported

supported means the ID is on the item's list for the asked version.

Defined in Citation checks, chapter 4. In the book since Edition 2026.11.3. Permanent link to supported

supporting sections

supporting sections, the parts of the docs that a correct answer rests on.

Defined in What a test set is for, chapter 3. In the book since Edition 2026.11.3. Permanent link to supporting sections

system text

The system text states the task, the rules and the form the reply must take.

Defined in Write a rubric a judge can apply, chapter 5. In the book since Edition 2026.11.3. Permanent link to system text

T

task

a task is the recipe for an evaluation, made minimally of a dataset, a solver and a scorer

Defined in Running it as a framework task, chapter 11. In the book since Edition 2026.11.3. Permanent link to task

template

The template wraps the request, the reference and the reply in markers, so the judge can tell them apart.

Defined in Write a rubric a judge can apply, chapter 5. In the book since Edition 2026.11.3. Permanent link to template

test set

A test set is a list of questions, each with the answer it should get.

Defined in The twenty items, chapter 1. In the book since Edition 2026.11.3. Permanent link to test set

transfer assumption

The transfer assumption is that the judge's miss rate on human passes, and its pass-through rate on human fails, are the same on sablefile's replies as on the HelpSteer2 rows.

Defined in The transfer assumption, chapter 6. In the book since Edition 2026.11.3. Permanent link to transfer assumption

true negative rate also TNR

The true negative rate, TNR, is the share of human FAIL rows that the judge fails.

Defined in True positive and true negative rates, chapter 6. In the book since Edition 2026.11.3. Permanent link to true negative rate

true positive rate also TPR

The true positive rate, TPR, is the share of human PASS rows that the judge passes.

Defined in True positive and true negative rates, chapter 6. In the book since Edition 2026.11.3. Permanent link to true positive rate

TruthfulQA

TruthfulQA asks questions on which people often answer falsely, such as a common myth.

Defined in TruthfulQA, chapter 3. In the book since Edition 2026.11.3. Permanent link to TruthfulQA

twin

a twin: the same question asked about another version

Defined in The twenty items, chapter 1. In the book since Edition 2026.11.3. Permanent link to twin

U

unanswerable item

Unanswerable items ask about something the docs don't cover, and the right reply is the decline.

Defined in The twenty items, chapter 1. In the book since Edition 2026.11.3. Permanent link to unanswerable item

unfiltered variant

The unfiltered variant drops the baseline's version filter.

Defined in What the replies look like, chapter 2. In the book since Edition 2026.11.3. Permanent link to unfiltered variant

unknown-section

The check calls a citation to any section that was not in the prompt unknown-section, whichever way it went wrong.

Defined in Citations, chapter 1. In the book since Edition 2026.11.3. Permanent link to unknown-section

unlisted

unlisted means a section of the asked version that the item doesn't list.

Defined in Citation checks, chapter 4. In the book since Edition 2026.11.3. Permanent link to unlisted

unpaired comparison

An unpaired comparison takes two separate pass rates and sets them side by side.

Defined in Paired comparisons, chapter 8. In the book since Edition 2026.11.3. Permanent link to unpaired comparison

unreadable pair

An unreadable pair has a missing verdict in either order.

Defined in Position, chapter 7. In the book since Edition 2026.11.3. Permanent link to unreadable pair

unreadable verdict

an unreadable verdict means the judge gave no judgment at all.

  • Contrasts with: FAIL

Defined in Verdict first, chapter 5. In the book since Edition 2026.11.3. Permanent link to unreadable verdict

upper bound

The upper bound is the least negative reading the data still allows

Defined in A gate that respects the interval, chapter 11. In the book since Edition 2026.11.3. Permanent link to upper bound

V

variance set

The variance set holds five samples of the assistant's reply to each dev item, drawn at temperature 0.7.

Defined in Variance within one item, chapter 8. In the book since Edition 2026.11.3. Permanent link to variance set

version swap also version-swap

version-swap: the question uses another version's wording, so an assistant that matches wording answers for that version.

Defined in Planted items and their traps, chapter 3. In the book since Edition 2026.11.3. Permanent link to version swap

version-correctness

Version-correctness asks whether the answer fits the asked version.

Defined in Check that answers rest on the cited docs, in the asked version, chapter 9. In the book since Edition 2026.11.3. Permanent link to version-correctness

version-dependent item

Version-dependent items have an answer that changes from one version to the next

Defined in The twenty items, chapter 1. In the book since Edition 2026.11.3. Permanent link to version-dependent item

W

weak pass

A pass with a wide interval is a weak pass

Defined in A gate that respects the interval, chapter 11. In the book since Edition 2026.11.3. Permanent link to weak pass

wrong-version answer

A wrong-version answer gives another version's behaviour for the version the question asks about

Defined in Wrong-version answers, chapter 2. In the book since Edition 2026.11.3. Permanent link to wrong-version answer

wrong-version citation also wrong-version

wrong-version means a section that exists in another version only.

Defined in Citation checks, chapter 4. In the book since Edition 2026.11.3. Permanent link to wrong-version citation

X

XSTest

XSTest is a test suite of safe prompts that look unsafe.

Defined in Outside this book, chapter 10. In the book since Edition 2026.11.3. Permanent link to XSTest