Glossary
The 125 terms Evaluating LLM Applications defines, each in the book's own words, with the section that defines it.
From Edition 2026.11.3, the book's current edition. Where a definition depends on a package, it says the version this edition pins and runs its examples on.
A
- agreement
-
Agreement is the share of rows on which the judge and the human give the same verdict.
Defined in Agreement and kappa, chapter 6. In the book since Edition 2026.11.3. Permanent link to agreement
- ambiguous item
-
Ambiguous items could mean two things, and their reference answer covers both readings.
Defined in The twenty items, chapter 1. In the book since Edition 2026.11.3. Permanent link to ambiguous item
- ambiguous target also ambiguous-target
-
ambiguous-target: a pronoun or a phrase such as "the file" has no clear referent without the sections.
Defined in Planted items and their traps, chapter 3. In the book since Edition 2026.11.3. Permanent link to ambiguous target
- answer judge
-
The answer judge checks a reply about sablefile against the written reference answer for its question.
Defined in What a judge is asked, chapter 5. In the book since Edition 2026.11.3. Permanent link to answer judge
- answerable item
-
Answerable items have their answer on the page.
Defined in The twenty items, chapter 1. In the book since Edition 2026.11.3. Permanent link to answerable item
- answered-declined-expected
-
answered-declined-expected is an answer to an item that has to be declined.
- Part of: format check
Defined in Format checks, chapter 4. In the book since Edition 2026.11.3. Permanent link to answered-declined-expected
- autojunk
-
For sequences of 200 items or more, the documentation treats an item as junk when its repeats make up more than one per cent of the sequence, unless autojunk is turned off.
Defined in Exact and fuzzy match, chapter 4. In the book since Edition 2026.11.3. Permanent link to autojunk
B
- baseline
-
The baseline is the setup chapter 1 measured.
Defined in What the replies look like, chapter 2. In the book since Edition 2026.11.3. Permanent link to baseline
- BLEU
-
Papineni and colleagues proposed BLEU in 2002 as a quick and inexpensive way to score machine translation.
Defined in Where BLEU and ROUGE mislead, chapter 4. In the book since Edition 2026.11.3. Permanent link to BLEU
- BM25
-
The index is built with BM25, which scores each section by the words it shares with the question.
True for bm25s 0.3.16, as this edition pins it.
- Implemented by: bm25s 0.3.16
Defined in Retrieve, then answer, chapter 1. In the book since Edition 2026.11.3. Permanent link to BM25
- Bonferroni correction
-
the Bonferroni correction, which shares the error budget between the comparisons.
1 minus 0.05 divided by mDefined in Many comparisons, chapter 8. In the book since Edition 2026.11.3. Permanent link to Bonferroni correction
C
- calibration rubric also rubric A
-
The calibration rubric checks correctness with no reference, on rows from HelpSteer2
- A kind of: rubric
Defined in What a judge is asked, chapter 5. In the book since Edition 2026.11.3. Permanent link to calibration rubric
- check candidate
-
a candidate has four parts: the mode, the input the check reads, the claim it tests, and what it can't see.
Defined in From a failure to a check, chapter 2. In the book since Edition 2026.11.3. Permanent link to check candidate
- citation
-
A citation is a claim about the docs: this section, in this version, supports the sentence it is attached to.
Defined in Citation accuracy, chapter 9. In the book since Edition 2026.11.3. Permanent link to citation
- citation accuracy
-
Citation accuracy asks whether each citation names a section the item lists as supporting.
- Contrasts with: faithfulness
Defined in Check that answers rest on the cited docs, in the asked version, chapter 9. In the book since Edition 2026.11.3. Permanent link to citation accuracy
- citation form
-
the citation form, a section ID in square brackets
Defined in Citations, chapter 1. In the book since Edition 2026.11.3. Permanent link to citation form
- citation rate
-
The citation rate counts every citation in answered items that is supported.
- Contrasts with: item rate
Defined in Citation accuracy, chapter 9. In the book since Edition 2026.11.3. Permanent link to citation rate
- citation-first
-
citation-first, for a reply whose first line opens with a citation
Defined in Format checks, chapter 4. In the book since Edition 2026.11.3. Permanent link to citation-first
- code check
-
A code check reads two things: the reply's text, and the item's own data, which holds its reference answer and the list of sections that support it.
- Contrasts with: judge
Defined in What a code check can see, chapter 4. In the book since Edition 2026.11.3. Permanent link to code check
- consistent pair
-
A pair is consistent when the same response wins in both orders.
- Contrasts with: order-changed pair
Defined in One answer or two, chapter 5. In the book since Edition 2026.11.3. Permanent link to consistent pair
- contamination
-
Contamination is when a benchmark's items or answers appear in training data, so a score can reflect memory rather than ability.
Defined in Contamination, chapter 3. In the book since Edition 2026.11.3. Permanent link to contamination
- corrected pass rate also corrected rate
-
The corrected rate estimates the share of sablefile's replies that a person would pass, under the book's correctness cut.
(observed + TNR - 1) / (TPR + TNR - 1)- Computed from: true positive rate, true negative rate
Defined in Correcting a pass rate, chapter 6. In the book since Edition 2026.11.3. Permanent link to corrected pass rate
- coverage
-
Coverage asks which parts of the tool the set exercises.
Defined in Kinds and coverage, chapter 3. In the book since Edition 2026.11.3. Permanent link to coverage
- crossover point also p*
-
The pass-rate correction in the next section leaves a rate unchanged at the crossover point, p* = (1 − TNR) / (2 − TPR − TNR).
p* = (1 − TNR) / (2 − TPR − TNR)- Computed from: true positive rate, true negative rate
Defined in True positive and true negative rates, chapter 6. In the book since Edition 2026.11.3. Permanent link to crossover point
D
- decline phrase also decline sentence
-
When the docs don't answer a question, the assistant must reply with the decline phrase, "The docs don't say.", and nothing else.
Defined in Declines, chapter 1. In the book since Edition 2026.11.3. Permanent link to decline phrase
- declined-with-text
-
declined-with-text is a reply that contains the decline sentence along with other text.
- Part of: format check
Defined in Format checks, chapter 4. In the book since Edition 2026.11.3. Permanent link to declined-with-text
- dev also dev items
-
dev items, the split of the test set this book works with while it develops its measures
- Part of: test set
Defined in Read the replies to find what to measure, chapter 2. In the book since Edition 2026.11.3. Permanent link to dev
E
- empty
-
empty is a reply with no text.
- Part of: format check
Defined in Format checks, chapter 4. In the book since Edition 2026.11.3. Permanent link to empty
- exact match
-
An exact match asks whether two strings are equal, and it suits any text that is fixed.
- Contrasts with: fuzzy match
Defined in Exact and fuzzy match, chapter 4. In the book since Edition 2026.11.3. Permanent link to exact match
- example environment
-
the book keeps one example environment: one set of files that every chapter's examples import, pinned to the same versions, so the replay model, the checks and the gate are the same code in every chapter.
Defined in Running it as a framework task, chapter 11. In the book since Edition 2026.11.3. Permanent link to example environment
- extra="forbid"
-
The loader sets extra="forbid", so an unknown field is an error and not a field that is quietly ignored.
True for pydantic 2.14.3, as this edition pins it.
- Implemented by: pydantic 2.14.3
Defined in The item schema, chapter 3. In the book since Edition 2026.11.3. Permanent link to extra="forbid"
F
- F1
-
F1, which combines recall with precision
Defined in Where BLEU and ROUGE mislead, chapter 4. In the book since Edition 2026.11.3. Permanent link to F1
- FAIL
-
A FAIL means the judge judged the answer wrong
Defined in Verdict first, chapter 5. In the book since Edition 2026.11.3. Permanent link to FAIL
- failure mode also mode
-
Each reply got one mode, its main defect
- Part of: check candidate
Defined in Coding the failures, chapter 2. In the book since Edition 2026.11.3. Permanent link to failure mode
- faithfulness
-
Faithfulness asks whether each claim is in the sections the assistant was shown.
- Contrasts with: version-correctness
Defined in Check that answers rest on the cited docs, in the asked version, chapter 9. In the book since Edition 2026.11.3. Permanent link to faithfulness
- false answer
-
a false answer is an unanswerable item answered.
Defined in Declines, chapter 1. In the book since Edition 2026.11.3. Permanent link to false answer
- false answer rate
-
The false answer rate is the number of unanswerable items answered, over the number of unanswerable items.
- Computed from: false answer
Defined in Two errors, both counted, chapter 10. In the book since Edition 2026.11.3. Permanent link to false answer rate
- false premise also false-premise
-
false-premise: the question asks about a flag or key that the asked version does not have.
Defined in Planted items and their traps, chapter 3. In the book since Edition 2026.11.3. Permanent link to false premise
- false refusal
-
A false refusal is an answerable item declined
- Contrasts with: false answer
Defined in Declines, chapter 1. In the book since Edition 2026.11.3. Permanent link to false refusal
- false refusal rate
-
The false refusal rate is the number of answerable items declined, over the number of answerable items.
- Computed from: false refusal
- Contrasts with: false answer rate
Defined in Two errors, both counted, chapter 10. In the book since Edition 2026.11.3. Permanent link to false refusal rate
- flip also flips, flaky
-
An item flips when its five verdicts are not all the same.
Defined in Variance within one item, chapter 8. In the book since Edition 2026.11.3. Permanent link to flip
- flip rate
-
the flip rate, the share of items that flip
- Computed from: flip
Defined in Flag flaky items, never edit them, chapter 11. In the book since Edition 2026.11.3. Permanent link to flip rate
- format check
-
The format check asks whether a reply has the shape the prompt asks for, before anyone reads what it says.
Defined in Format checks, chapter 4. In the book since Edition 2026.11.3. Permanent link to format check
- fuzzy match
-
A fuzzy match scores how alike two strings are.
Defined in Exact and fuzzy match, chapter 4. In the book since Edition 2026.11.3. Permanent link to fuzzy match
G
- gate
-
A gate is a rule that stops a change from merging.
Defined in A gate that respects the interval, chapter 11. In the book since Edition 2026.11.3. Permanent link to gate
- grounded
-
An answer is grounded when a reader who follows its citations finds the claim where the citation says it is, in the version they asked about.
Defined in What grounded means here, chapter 9. In the book since Edition 2026.11.3. Permanent link to grounded
- GSM8K
-
GSM8K is grade-school math word problems, each with a worked solution that ends in one number.
- A kind of: public benchmark
Defined in GSM8K, chapter 3. In the book since Edition 2026.11.3. Permanent link to GSM8K
- guard also retrieval-score guard, decline guard
-
A guard reads the best-matching section's score and declines when it falls below a threshold.
Defined in A guard on recorded replies, chapter 10. In the book since Edition 2026.11.3. Permanent link to guard
H
- held-out also held-out items
-
held-out is used once, to report a result.
Defined in Dev and held-out, chapter 3. In the book since Edition 2026.11.3. Permanent link to held-out
I
- interval also 95% interval
-
An interval answers a different question from the pass rate: how far would the rate move if the set had been another twenty items drawn the same way?
- Computed from: pass rate
Defined in A pass rate and its interval, chapter 1. In the book since Edition 2026.11.3. Permanent link to interval
- item
-
an item: one test question with the fields a scorer needs.
- Part of: test set
Defined in What a test set is for, chapter 3. In the book since Edition 2026.11.3. Permanent link to item
- item rate
-
The item rate counts an answered item once if at least one of its citations is supported, so one good citation carries the item.
Defined in Citation accuracy, chapter 9. In the book since Edition 2026.11.3. Permanent link to item rate
J
- judge
-
a judge: a model that reads the question, a written reference answer and the assistant's reply, and returns PASS or FAIL
Defined in One judge, chapter 1. In the book since Edition 2026.11.3. Permanent link to judge
- judged-correct rate
-
The judged-correct rate is the count for version-dependent items alone, so a version miss shows on its own line, and a team can see whether a change to retrieval moved it.
Defined in Version-correctness, chapter 9. In the book since Edition 2026.11.3. Permanent link to judged-correct rate
K
- k5
-
The variant k5 retrieves five sections instead of three.
Defined in Held-out, once, chapter 8. In the book since Edition 2026.11.3. Permanent link to k5
- kappa
-
For two raters on pass or fail, kappa is the observed agreement minus the chance agreement, divided by one minus the chance agreement.
- Computed from: agreement
Defined in Agreement and kappa, chapter 6. In the book since Edition 2026.11.3. Permanent link to kappa
- kind
-
Each item also has a kind, which says what a correct reply looks like.
- Part of: item
Defined in What a test set is for, chapter 3. In the book since Edition 2026.11.3. Permanent link to kind
L
- length also verbosity bias
-
length: whether the pass rate rises with the length of the reply.
- Contrasts with: position
Defined in Find where a judge's verdict stops being reliable, chapter 7. In the book since Edition 2026.11.3. Permanent link to length
M
- MC1
-
The book uses TruthfulQA's multiple-choice version, MC1: exactly one option is correct, and the model answers with a letter.
- Part of: TruthfulQA
Defined in TruthfulQA, chapter 3. In the book since Edition 2026.11.3. Permanent link to MC1
- missed refusal
-
A missed refusal answers an unanswerable item with content the docs don't hold.
- A kind of: failure mode
Defined in Missed refusals, chapter 2. In the book since Edition 2026.11.3. Permanent link to missed refusal
- missing
-
missing means no version has a section with the cited ID.
Defined in Citation checks, chapter 4. In the book since Edition 2026.11.3. Permanent link to missing
N
- naive gate
-
A naive gate would fail whenever the point estimate of the difference is below minus the threshold
Defined in A gate that respects the interval, chapter 11. In the book since Edition 2026.11.3. Permanent link to naive gate
- near-miss term also near-miss-term
-
a planted near-miss term, a name close to one the docs use, which catches a careless reply
Defined in What the replies look like, chapter 2. In the book since Edition 2026.11.3. Permanent link to near-miss term
- no difference detected
-
No difference detected is a finding about a set at its size.
Defined in Paired comparisons, chapter 8. In the book since Edition 2026.11.3. Permanent link to no difference detected
- no-citation
-
no-citation is an answer that cites nothing.
- Part of: format check
Defined in Format checks, chapter 4. In the book since Edition 2026.11.3. Permanent link to no-citation
O
- ok
-
a reply with no defect got ok.
Defined in Coding the failures, chapter 2. In the book since Edition 2026.11.3. Permanent link to ok
- order-changed pair also order-changed
-
An order-changed pair is one where the verdict followed the position, so it says nothing about which reply is better.
Defined in One answer or two, chapter 5. In the book since Edition 2026.11.3. Permanent link to order-changed pair
- outcome
-
Outcome is whether a reply passed.
- Contrasts with: path
Defined in Outcome and path, chapter 9. In the book since Edition 2026.11.3. Permanent link to outcome
- outcome against path
-
Outcome against path asks whether a pass came with its section in hand, and whether a fail had it.
Defined in Check that answers rest on the cited docs, in the asked version, chapter 9. In the book since Edition 2026.11.3. Permanent link to outcome against path
- over-declining
-
An over-declining reply declines an item the docs answer.
- A kind of: failure mode
Defined in Over-declining, chapter 2. In the book since Edition 2026.11.3. Permanent link to over-declining
P
- paired comparison
-
A paired comparison subtracts, for each item, the verdict of the baseline from the verdict of prompt B, and averages the differences over the set.
- Contrasts with: unpaired comparison
Defined in Paired comparisons, chapter 8. In the book since Edition 2026.11.3. Permanent link to paired comparison
- paired difference
-
The paired difference is rubric B's score minus rubric A's, averaged over the dev rows
Defined in Agreement and kappa, chapter 6. In the book since Edition 2026.11.3. Permanent link to paired difference
- pairwise judge
-
A pairwise judge reads two replies and picks the better one, rather than passing or failing each.
Defined in Pairwise calibration, chapter 6. In the book since Edition 2026.11.3. Permanent link to pairwise judge
- pairwise judgment
-
A pairwise judgment shows the judge two replies to the same request and asks which is better.
Defined in One answer or two, chapter 5. In the book since Edition 2026.11.3. Permanent link to pairwise judgment
- pass or fail rubric
-
A pass or fail rubric gives the judge two answers.
- A kind of: rubric
Defined in Pass, fail, or a scale, chapter 5. In the book since Edition 2026.11.3. Permanent link to pass or fail rubric
- pass rate
-
A pass rate is a count divided by a denominator
pass rate = passed items / itemsDefined in A pass rate and its interval, chapter 1. In the book since Edition 2026.11.3. Permanent link to pass rate
- path
-
Path is what the steps before the reply reached.
Defined in Outcome and path, chapter 9. In the book since Edition 2026.11.3. Permanent link to path
- percentile bootstrap
-
The percentile bootstrap is the simplest interval built on resampling, the middle 95% of the resampled rates.
- A kind of: interval
Defined in A pass rate and its interval, chapter 1. In the book since Edition 2026.11.3. Permanent link to percentile bootstrap
- planted item
-
A planted item was written to catch one particular mistake
Defined in The twenty items, chapter 1. In the book since Edition 2026.11.3. Permanent link to planted item
- position
-
position: whether a pairwise verdict depends on which answer was shown first.
Defined in Find where a judge's verdict stops being reliable, chapter 7. In the book since Edition 2026.11.3. Permanent link to position
- public benchmark
-
A public benchmark is a set of questions and answers someone else published.
Defined in Public benchmarks, chapter 3. In the book since Edition 2026.11.3. Permanent link to public benchmark
R
- ratio()
-
ratio(), which the documentation defines as twice the number of matching elements divided by the total number of elements in both sequences.
Defined in Exact and fuzzy match, chapter 4. In the book since Edition 2026.11.3. Permanent link to ratio()
- recording
-
The recording is a folder of replies, one per item.
Defined in Replay the answers, chapter 1. In the book since Edition 2026.11.3. Permanent link to recording
- reference answer
-
the reference answer, which is the author's statement of what a right reply says
- Part of: item
Defined in What the replies look like, chapter 2. In the book since Edition 2026.11.3. Permanent link to reference answer
- reference-free judge
-
A reference-free judge is given only the request and the reply, and it decides from what it knows whether the reply is right.
Defined in Reference-guided or reference-free, chapter 5. In the book since Edition 2026.11.3. Permanent link to reference-free judge
- reference-guided judge
-
A reference-guided judge is given a written answer for the question, and it checks the reply against that answer.
- Contrasts with: reference-free judge
Defined in Reference-guided or reference-free, chapter 5. In the book since Edition 2026.11.3. Permanent link to reference-guided judge
- retrieval-path count
-
The retrieval-path count asks one question of each item: is a supporting section of the asked version in the top k?
Defined in Retrieval quality belongs to another book, chapter 9. In the book since Edition 2026.11.3. Permanent link to retrieval-path count
- ROUGE-1 F1
-
ROUGE-1 F1 counts single words.
Defined in Where BLEU and ROUGE mislead, chapter 4. In the book since Edition 2026.11.3. Permanent link to ROUGE-1 F1
- ROUGE-N
-
Lin defined ROUGE-N as an n-gram recall: the share of the reference's word runs that the candidate covers
Defined in Where BLEU and ROUGE mislead, chapter 4. In the book since Edition 2026.11.3. Permanent link to ROUGE-N
- rubric
-
A judge works from its instructions, and the instructions are the rubric.
Defined in Write a rubric a judge can apply, chapter 5. In the book since Edition 2026.11.3. Permanent link to rubric
- rubric B
-
Rubric B is a second wording of the calibration task.
- A kind of: rubric
Defined in What a judge is asked, chapter 5. In the book since Edition 2026.11.3. Permanent link to rubric B
S
- scale
-
A scale gives the judge several answers.
- Contrasts with: pass or fail rubric
Defined in Pass, fail, or a scale, chapter 5. In the book since Edition 2026.11.3. Permanent link to scale
- second-model variant also second model
-
The second-model variant keeps the baseline's prompt and hands the answer to a smaller model, Qwen3 with 0.6 billion parameters in an 8-bit build.
Defined in What the replies look like, chapter 2. In the book since Edition 2026.11.3. Permanent link to second-model variant
- self-preference
-
self-preference: whether the judge favours answers written by its own model.
- Contrasts with: length
Defined in Find where a judge's verdict stops being reliable, chapter 7. In the book since Edition 2026.11.3. Permanent link to self-preference
- stale answer also stale-answer
-
stale-answer: the answer sits under a section ID that a version filter removes, so a filter that is too strict misses it.
Defined in Planted items and their traps, chapter 3. In the book since Edition 2026.11.3. Permanent link to stale answer
- stratified
-
The split is stratified, so that each kind appears in both parts in about the same proportion.
Defined in Dev and held-out, chapter 3. In the book since Edition 2026.11.3. Permanent link to stratified
- stray-line
-
stray-line, for a line that ends no sentence once its citations are taken out.
Defined in Format checks, chapter 4. In the book since Edition 2026.11.3. Permanent link to stray-line
- supported
-
supported means the ID is on the item's list for the asked version.
- Measured from: supporting sections
Defined in Citation checks, chapter 4. In the book since Edition 2026.11.3. Permanent link to supported
- supporting sections
-
supporting sections, the parts of the docs that a correct answer rests on.
- Part of: item
Defined in What a test set is for, chapter 3. In the book since Edition 2026.11.3. Permanent link to supporting sections
- system text
-
The system text states the task, the rules and the form the reply must take.
- Part of: rubric
Defined in Write a rubric a judge can apply, chapter 5. In the book since Edition 2026.11.3. Permanent link to system text
T
- task
-
a task is the recipe for an evaluation, made minimally of a dataset, a solver and a scorer
Defined in Running it as a framework task, chapter 11. In the book since Edition 2026.11.3. Permanent link to task
- template
-
The template wraps the request, the reference and the reply in markers, so the judge can tell them apart.
- Part of: rubric
Defined in Write a rubric a judge can apply, chapter 5. In the book since Edition 2026.11.3. Permanent link to template
- test set
-
A test set is a list of questions, each with the answer it should get.
Defined in The twenty items, chapter 1. In the book since Edition 2026.11.3. Permanent link to test set
- transfer assumption
-
The transfer assumption is that the judge's miss rate on human passes, and its pass-through rate on human fails, are the same on sablefile's replies as on the HelpSteer2 rows.
Defined in The transfer assumption, chapter 6. In the book since Edition 2026.11.3. Permanent link to transfer assumption
- true negative rate also TNR
-
The true negative rate, TNR, is the share of human FAIL rows that the judge fails.
Defined in True positive and true negative rates, chapter 6. In the book since Edition 2026.11.3. Permanent link to true negative rate
- true positive rate also TPR
-
The true positive rate, TPR, is the share of human PASS rows that the judge passes.
Defined in True positive and true negative rates, chapter 6. In the book since Edition 2026.11.3. Permanent link to true positive rate
- TruthfulQA
-
TruthfulQA asks questions on which people often answer falsely, such as a common myth.
- A kind of: public benchmark
Defined in TruthfulQA, chapter 3. In the book since Edition 2026.11.3. Permanent link to TruthfulQA
- twin
-
a twin: the same question asked about another version
Defined in The twenty items, chapter 1. In the book since Edition 2026.11.3. Permanent link to twin
U
- unanswerable item
-
Unanswerable items ask about something the docs don't cover, and the right reply is the decline.
- Contrasts with: answerable item
Defined in The twenty items, chapter 1. In the book since Edition 2026.11.3. Permanent link to unanswerable item
- unfiltered variant
-
The unfiltered variant drops the baseline's version filter.
Defined in What the replies look like, chapter 2. In the book since Edition 2026.11.3. Permanent link to unfiltered variant
- unknown-section
-
The check calls a citation to any section that was not in the prompt unknown-section, whichever way it went wrong.
- Part of: format check
Defined in Citations, chapter 1. In the book since Edition 2026.11.3. Permanent link to unknown-section
- unlisted
-
unlisted means a section of the asked version that the item doesn't list.
Defined in Citation checks, chapter 4. In the book since Edition 2026.11.3. Permanent link to unlisted
- unpaired comparison
-
An unpaired comparison takes two separate pass rates and sets them side by side.
Defined in Paired comparisons, chapter 8. In the book since Edition 2026.11.3. Permanent link to unpaired comparison
- unreadable pair
-
An unreadable pair has a missing verdict in either order.
Defined in Position, chapter 7. In the book since Edition 2026.11.3. Permanent link to unreadable pair
- unreadable verdict
-
an unreadable verdict means the judge gave no judgment at all.
- Contrasts with: FAIL
Defined in Verdict first, chapter 5. In the book since Edition 2026.11.3. Permanent link to unreadable verdict
- upper bound
-
The upper bound is the least negative reading the data still allows
Defined in A gate that respects the interval, chapter 11. In the book since Edition 2026.11.3. Permanent link to upper bound
V
- variance set
-
The variance set holds five samples of the assistant's reply to each dev item, drawn at temperature 0.7.
Defined in Variance within one item, chapter 8. In the book since Edition 2026.11.3. Permanent link to variance set
- version swap also version-swap
-
version-swap: the question uses another version's wording, so an assistant that matches wording answers for that version.
Defined in Planted items and their traps, chapter 3. In the book since Edition 2026.11.3. Permanent link to version swap
- version-correctness
-
Version-correctness asks whether the answer fits the asked version.
Defined in Check that answers rest on the cited docs, in the asked version, chapter 9. In the book since Edition 2026.11.3. Permanent link to version-correctness
- version-dependent item
-
Version-dependent items have an answer that changes from one version to the next
Defined in The twenty items, chapter 1. In the book since Edition 2026.11.3. Permanent link to version-dependent item
W
- weak pass
-
A pass with a wide interval is a weak pass
Defined in A gate that respects the interval, chapter 11. In the book since Edition 2026.11.3. Permanent link to weak pass
- wrong-version answer
-
A wrong-version answer gives another version's behaviour for the version the question asks about
- A kind of: failure mode
Defined in Wrong-version answers, chapter 2. In the book since Edition 2026.11.3. Permanent link to wrong-version answer
- wrong-version citation also wrong-version
-
wrong-version means a section that exists in another version only.
- Contrasts with: wrong-version answer
Defined in Citation checks, chapter 4. In the book since Edition 2026.11.3. Permanent link to wrong-version citation
X
- XSTest
-
XSTest is a test suite of safe prompts that look unsafe.
Defined in Outside this book, chapter 10. In the book since Edition 2026.11.3. Permanent link to XSTest